Compare commits

...
Author SHA1 Message Date
Ljy-0827 ec62b8f0f2 fix(bsk): fix ci 2026-08-21 16:34:06 +08:00
Ljy-0827 9393f0e58b feat(bsk): harden browser file transfer transactions 2026-08-21 16:31:47 +08:00
Ljy-0827 f9d395ee41 fix(bsk): fix ci 2026-08-21 14:32:29 +08:00
Ljy-0827 8f81c15656 fix(bsk): upload function safety 2026-08-21 14:11:13 +08:00
Ljy-0827 3cfa9eceef feat(bsk): fix upload function that cannot bypass layout 2026-08-21 11:14:51 +08:00
Ljy-0827 34f47d7387 feat(bsk): fix different type of upload 2026-08-20 18:59:22 +08:00
Ljy-0827 0a184e33d1 feat(bsk): support upload and download 2026-08-20 16:51:50 +08:00
BB-fat 6b78211e79 test(dsh-plugin): drop redundant thumbnail read-back case
Coverage already lives in the attachment-store and ring tests; this
case only re-hit readThumbnail and was flaky on CI.
2026-08-19 03:57:08 +00:00
BB-fat eebf334b40 fix(dsh-plugin): give observation waitFor more room on CI
The thumbnail read-back test timed out at the 2s waitFor budget under
CI load even though capture completed shortly after.
2026-08-19 03:50:14 +00:00
BB-fat a284ffc5ae Merge pull request #112 from Tencent/fix/dsh-plugin-id-and-edge-store
Fix/dsh plugin id and edge store
2026-08-19 11:43:13 +08:00
drakezhangandCursor e720f1ffe5 ci(dsh-plugin): publish from the npm-publish environment
The publish job named its GitHub Environment NPM_TOKEN, which is the
secret name rather than a deployment target, so the repository's
Deployments list showed a bogus NPM_TOKEN entry next to chrome-web-store.
The environment is now npm-publish; the secret it holds keeps its name.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-19 11:20:26 +08:00
drakezhangandCursor 2b7e10ca2b fix(dsh-plugin-browserskill): identify the plugin by its published name
Renaming the package to @wxg-prc-cpg/browser-skill-dsh-plugin left two
identifiers behind on the old directory name, and both are resolved by
name rather than by path.

cordis.patch.yml is the specifier the loader imports, so dsh failed with
ERR_MODULE_NOT_FOUND on startup and took the whole plugin tree down with
it — installing the published package made dsh unusable. The client
bundle registers itself under the same kind of key, which the web shell
looks up by package name, so the screenshot toolview and the observation
overlay were unreachable too.

The bundle id now reads package.json instead of repeating the name, so it
cannot drift again. Quoting is required in the YAML because a leading '@'
is reserved in plain scalars.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-19 11:20:07 +08:00
drakezhangandCursor 1ef93463dd docs: list the Edge Add-ons build alongside the Chrome Web Store
The extension is now published to Edge Add-ons, but every install path
still pointed Edge users at the Chrome Web Store. Both READMEs and
AGENT_INSTALL.md now name a store per browser, and `bsk doctor` offers
both listings when no browser is connected.

Two Edge-specific gaps went with it: Edge blocks content-script injection
on its own storefront the same way Chrome does, so the borrow-confirmation
overlay must treat microsoftedge.microsoft.com/addons as non-injectable;
and the Chinese README still described the pre-install.ps1 manual Windows
setup.

Also give the DeepSeek Harness section its npm package link and the
profile-start step, so plugin users have a complete path.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-19 11:19:47 +08:00
Hannah Liu c7252529f4 Merge pull request #109 from Tencent/fix/oopif-support
fix(extension): support OOPIF and cross-frame geometry
2026-08-18 21:16:39 +08:00
Ljy-0827 0b65b4b295 fix(vom): oopif fix 2026-08-18 19:28:16 +08:00
Ljy-0827 94ec7e73fc fix(vom): iframe oopif geometry 2026-08-18 19:28:16 +08:00
Ljy-0827 42403e55f2 feat(extension): support OOPIF refs with unified frame geometry 2026-08-18 19:28:16 +08:00
Hannah Liu 22019cdf07 Merge pull request #105 from Tencent/fix/vom-iframe
fix(vom): vom support observe iframe
2026-08-18 19:28:13 +08:00
Ljy-0827 8d7c0aa66c fix(extension): support iframe content in VOM observations 2026-08-18 19:18:39 +08:00
BB-fat a9489282fa fix(dsh-plugin-browserskill): keep observation frames in a 2-slot ring
Stop writing overlay thumbnails into the durable attachment store. Each
session now holds only the current and previous PNG in process memory,
reuses the id when the hash is unchanged, and drops the rest on rotate,
remove, or dispose.
2026-08-18 10:15:27 +00:00
Zhang GH d4cddd6224 Merge pull request #99 from Tencent/fix/record-empty-fill
fix(record): preserve committed empty fill steps
2026-08-18 18:12:59 +08:00
Zhang GH 8e4de6cf4b Merge pull request #91 from Tencent/feat/dsh-plugin
feat(packages): add dsh-plugin-browserskill — DeepSeek Harness tools for bsk
2026-08-18 18:12:37 +08:00
BB-fat fea66ce404 fix(dsh-plugin-browserskill): send JSON content-type on overlay interrupt
The observation HTTP fence rejects POSTs without application/json, so the
Interrupt button always got 403.
2026-08-18 09:36:15 +00:00
BB-fat 8bae85127a docs: add DeepSeek Harness plugin section to the README
Document the dsh plugin highlights and the one-line install for the web profile.
2026-08-18 03:13:58 +00:00
BB-fat edb149611b ci: publish the dsh plugin to npm as @wxg-prc-cpg/browser-skill-dsh-plugin
Add a Release dsh plugin workflow (dsh-plugin-v* tags or workflow_dispatch)
that reads NPM_TOKEN from the GitHub Environment of the same name. The
Cordis plugin id stays dsh-plugin-browserskill; only the npm package name
changes.
2026-08-17 10:00:13 +00:00
BB-fat 14c5884cbc style(dsh-plugin-browserskill): fix overlay CSS stylelint errors
Add the missing blank line before the actions comment, and hoist
.tool-button:focus-visible above the wrap-hover rules so specificity
no longer descends.
2026-08-17 09:52:09 +00:00
BB-fat e7aad9bc16 style(dsh-plugin-browserskill): wrap lines to satisfy biome
CI's frontend job failed on formatter-only diffs in the overlay and the
skill-content assertion.
2026-08-17 09:48:27 +00:00
BB-fat 3fb9e9f581 fix(dsh-plugin-browserskill): compact overlay toolbar and four-corner resize
Use icon-only actions with a short hover tip, swap the interrupt glyph to
RiStopCircleLine, and let every corner resize the card without a grip icon.
2026-08-17 07:50:53 +00:00
BB-fat 9854ee2f45 fix(dsh-plugin-browserskill): stop observation overlay from flashing between frames
Hold the last ready thumbnail until the next one decodes, and swap the
img src in place instead of remounting through a placeholder.
2026-08-17 07:27:28 +00:00
BB-fat 51085b960b fix(browser-skill): route DSH usage through injected tools 2026-08-17 07:04:36 +00:00
BB-fat 51df868008 chore(dsh-plugin-browserskill): prepare the package for npm publishing
- @browser-skill/ui and @remixicon/react move to devDependencies (the client
  bundle inlines both; the host face never imports them — verified).
- Drop private, add license: MIT, ship the repo-root MIT LICENSE in files.
- peerDependencies: add the runtime-required @deepseek-ai/schemastery, and
  the previously dangling meta entries @deepseek-ai/dsh-attachment and
  @deepseek-ai/dsh-llm (optional, matching actual type-level use).
- prepack runs the full build; npm pack now yields exactly 7 files
  (lib/index.mjs, lib/index.d.mts, lib/client.cjs, cordis.patch.yml,
  README.md, LICENSE, package.json).
- publint clean: the client face ships as lib/client.cjs (CJS extension in
  a type:module package); add lint:publint script. Built declarations pass a
  NodeNext consumer typecheck (strict, skipLibCheck false).
2026-08-14 17:13:48 +00:00
BB-fat 43ebfab7ba fix(dsh-plugin-browserskill): register the lazy suite from any fiber
Cordis contexts are fiber-owned: assigning onto an Object.create() overlay of
the apply-time context fails with 'cannot set property tools in multiple
fibers' when the reveal fires from the tools/result listener fiber (caught by
the e2e: the skill invocation matched but the suite never registered).
registerTools now tracks disposers through a plain closure over the captured
context — no context mutation anywhere. Also documents lazyTools in the
README and hardens the reveal against a failed registration (log and stay
hidden instead of latching).

Verified end to end against the real stack: first request carries 0 browser_*
schemas (skill catalog entry only); after the scripted skill invocation the
next request carries exactly the 11 browser_* tools and browser_session_start
dispatches for real.
2026-08-14 16:58:30 +00:00
BB-fat ce11325de2 feat(dsh-plugin-browserskill): reveal tool schemas only after skill invocation
Progressive disclosure, final stage: with lazyTools on (default), the eleven
browser_* tool schemas stay out of the system prompt — the browser-skill
catalog entry is the sole advertisement. One successful skill invocation
(model tool call via tools/result, or a /browser-skill user gesture via the
session append feed) registers the suite for the process lifetime, idempotent
on repeats; session resume is covered by scanning durable events for a past
tool/call+tool/result pair (or skill-invocation message) on session entry and
at apply time. lazyTools: false keeps the legacy always-on registration.

Investigation (recorded in the ticket): tools/result carries normalized
name/arguments with an isError discriminant; dsh 0.1 has no official global
visibility switch (ctx.tools.restrict is agent-scoped and throws from a plain
context), making conditional registration the intended pattern; per-step
assembly picks the new suite up on the next step, and visibility changes are
first-class inputs of tool-skill's catalog digest. registerTools now returns
the combined disposer via a prototype-preserving tracking overlay; the skill
prelude tells the model the browser_* tools unlock on skill load.

Tests: +8 (hidden-before/reveal/idempotent/gesture/history hit+miss/dispose/
scanner pairing/two-state apply wiring). 124/124 green.
2026-08-14 16:43:51 +00:00
BB-fat 4d1dcd122f feat(dsh-plugin-browserskill): publish the BSK skill through ctx.skills
The plugin exposed only tools; the BrowserSkill agent skill never reached
the harness's <available_skills> catalog. Register it as an embedded runtime
skill (progressive disclosure: catalog entry resident, body on skill-tool
invocation) via the official ctx.skills seam — silent no-op when the
composition lacks it.

Content is build-time assembled so there is one source of truth:
skill/prelude.md (committed, dsh-specific: tool<->CLI map, owned-session
semantics, plugin-only overrides) + the canonical repo-root skill/SKILL.md
verbatim (the same file crates/bsk-cli/build.rs mirrors; frontmatter
stripped, name/description supplied by the registration). Generated into
src/skill-content.generated.ts (gitignored) — registration and every
pre-step snapshot are pure in-memory reads: no disk, no process, no daemon.

Scripts: build:skill chained into build/test/typecheck (same pattern as
build:css); biome excludes the generated artifacts. Tests: +3 (catalog
content, in-memory weight, silent degradation). 116/116 green.
2026-08-14 16:17:54 +00:00
BB-fat a4e668542d fix(dsh-plugin-browserskill): address review findings (leaks, trust fence, queue race)
Blocking:
- B1: delete screenshot scratch PNGs — observation frames reuse one fixed
  per-session path and unlink after every read (finally); browser_screenshot
  unlinks once the bytes are in the attachment store (kept only when the file
  itself is the model-facing artifact).
- B2: replicate dsh's browser-trust fence on /bsk-observation/*: loopback
  Host only, Origin must match Host, sec-fetch-site: cross-site refused, POST
  requires application/json; README documents the loopback trust premise and
  the 0.0.0.0 warning.

Major:
- M1: KeyedExecutor tail chains previous+task (allSettled) so a task aborted
  while queued cannot release the next one into the running session.
- M2: client tracks the live frame per session — replacing/removing/reset
  revokes the old blob URL at once; loads settling after replacement never
  resurrect it.

Minor: drop replaced thumbRefs (+on remove); beginAction fires inside the
queue (no label overwrite while queued, no idle flash); half-initialized
session cleanup stops through the queue with one retry; SSE (re)open refetches
/state (+ snapshot flag renamed subscribed); install probe uses --version (no
daemon spawn); emulate mobile documents the width+height requirement (the
daemon refuses it alone — verified).

Nit: single BskRunOptions declaration; tails map entry dropped on drain; SSE
cleanup on res close; popOut catches requestWindow rejections; resize handle
gains keyboard control (arrows, 16px steps, aria value attrs).

Tests: +14 (fence rules, scratch lifecycle, queue race, blob revocation,
instrumentation timing, SSE resync). 113/113 green.
2026-08-14 11:56:02 +00:00
BB-fat 52aa356153 test(dsh-plugin-browserskill): build the scoped css before vitest
The client entry imports the generated bsk-ui.nomodule.css; the file is
gitignored, so a fresh CI checkout could not resolve it. Generate it as part
of the test script.
2026-08-14 11:05:05 +00:00
BB-fat d1001e14b2 feat(dsh-plugin-browserskill): reskin the overlay with BrowserSkill's own UI system
The previous pass aligned the observation overlay with the dsh shell; the
product call is the opposite direction — the floating card should read as a
BrowserSkill surface, not a shell-native widget.

- Reuse @browser-skill/ui directly (Button, cn); status dots spec'd after the
  extension popup's ConnectionStatusIndicator; Remix icons for
  interrupt/pin/warn/pop-out.
- Compile the BSK tailwind utility sheet scoped under the .bsk-obs root class
  (scripts/build-client-css.mjs + postcss-prefix-selector) and ship the oklch
  design tokens on the same scope, so nothing leaks into the host shell and
  the shell theme cannot bleed back. tsdown injects .nomodule.css verbatim
  (minified, unhashed) alongside the hashed CSS modules.
- BSK ships no shared tooltip, so the one-time interrupt semantics hint is a
  card-spec bubble (bg-card/border/12px radius), retired after first use.
- Tests: pin react/react-dom to this package's 18.x copies in vitest — the ui
  package sources peer on react ^19 and would otherwise emit react-19
  (transitional) elements that the react-18 renderer rejects. Production is
  unaffected (react stays external, provided by the shell).
- stylelint: ignore the generated bsk-ui.nomodule.css.
2026-08-14 10:00:37 +00:00
BB-fat eb71ddc1da docs(dsh-plugin-browserskill): note the top-right docking and shell-native styling 2026-08-14 08:10:05 +00:00
BB-fat b07b8fec65 fix(dsh-plugin-browserskill): dock below the shell top bar
The plain top:16 default overlapped the shell's Session log action;
dock the card and capsule at the content area's top-right (64px) so
neither collides with the header controls.
2026-08-14 08:06:28 +00:00
BB-fat 47070fdc14 feat(dsh-plugin-browserskill): default the overlay to the top-right + dsh-native styling
Review follow-ups: the card/capsule now default to the top-right (clear
of the composer), and the overlay reads as part of the shell instead of
a foreign widget:

- shared primitives everywhere: Button (outline/ghost sm) for
  Interrupt/Pop out with the shell's danger hover token, Tooltip for
  the one-time interrupt semantics hint (native span anchor — Button
  does not forward refs), StateDot for status, and outline icons
  (Stop/RightUp/ChevronDown/Warning) replacing text glyphs;
- every color/surface/shadow/label now uses --dsw-alias-* semantic
  tokens (bg-overlay, border-l1/l2, state-*-primary, label-*,
  interactive-bg-hover[-danger], brand-primary focus rings,
  dsw-shadow-lv1/lv2) — no invented tokens, no literal colors (dsh 0.1
  ships no spacing/radius scale; those stay px like dsh's own code);
- strip item focus ring uses brand-primary; focus-visible and
  reduced-motion preserved.
2026-08-14 07:57:07 +00:00
BB-fat 916229dc62 docs(dsh-plugin-browserskill): README section for the observation overlay 2026-08-14 07:05:39 +00:00
BB-fat dde707b558 fix(dsh-plugin-browserskill): report interrupt-killed commands as interrupted
A child killed via killFor (overlay interrupt) exits with a null code
and empty output, which used to render as a duplicated 'bsk x failed:
bsk x failed'. Report it as interrupted instead.
2026-08-14 07:01:44 +00:00
BB-fat e2d07574d7 fix(dsh-plugin-browserskill): prefix routes register without a trailing slash
The webserver prefix matcher joins prefix + '/', so a registered
'/bsk-observation/thumbnail/' never matched real paths.
2026-08-14 06:54:01 +00:00
BB-fat 461d204ca3 fix(dsh-plugin-browserskill): serve observation thumbnails over the plugin's own route
The session-authorized client RPC (readAttachment) refuses images that
no session log references — observation frames are plugin-owned runtime
data, so every overlay thumbnail failed with ATTACHMENT_NOT_REFERENCED.
The host now keeps the full attachment ref per frame and serves bytes
through GET /bsk-observation/thumbnail/<id> (verified readImage path);
the client loader is a plain fetch of that route.
2026-08-14 06:49:15 +00:00
BB-fat bc5d9610a3 fix(dsh-plugin-browserskill): serialize per-session commands (observation vs tool calls)
The daemon accepts only one unfinished command per session; the
thumbnail loop racing a model tool call produced 'session already has
an unfinished command' failures on both sides (found by the Phase 1-C
e2e: navigate/snapshot/click all errored while captures kept running).
Add a KeyedExecutor: every command for a session — tool calls and
observation captures alike — runs FIFO; queued tasks reject early on
abort, running tasks keep the runner's signal-driven kill.
2026-08-14 06:41:49 +00:00
BB-fat 2955260d11 fix(dsh-plugin-browserskill): register observation routes once webServer appears
ctx.get('webServer') at apply time raced the web composition's service
registration (the fallback SPA then answered /bsk-observation/* with
index.html). Ride ctx.inject(['webServer'], …) instead: the callback
runs when the service is provided and never runs in headless
compositions, keeping route registration timing-safe and optional.
2026-08-14 06:20:18 +00:00
BB-fat ce89e95211 feat(dsh-plugin-browserskill): multi-session strip, focus pin, and edge states (Phase 1-C)
- Strip (meeting-style multi-session layout): one item per session
  (mini frame + id + status dot), horizontal row with scroll overflow;
  hover reveals a per-item interrupt button that acts without
  refocusing; click pins the focus view (pin badge, click again to
  release); auto-follow picks the most recently active session but
  never steals focus for errored or dead sessions (red edge / greyed
  item instead).
- Edge states: the host flips an availability flag after repeated
  global capture failures (client shows 'browser unavailable', keeps
  the last frame, greys interrupt, and recovers on the next success);
  a session_not_found envelope marks the session dead (no more frame
  requests, grey strip item, removable as usual); lastError now clears
  when the next action starts or succeeds.
- State/SSE wire gains { available } and an 'availability' event;
  SessionObservation gains dead?.

Tests (+8, 97 total): host availability flip, dead marking +
instrumentation drop + clean removal, lastError lifecycle; client
strip render/pin/unpin/auto-follow, error no-steal, unavailable strip,
dead grey, strip-item interrupt without refocus.
2026-08-14 05:58:53 +00:00
BB-fat 24faa10659 feat(dsh-plugin-browserskill): observation overlay — floating card, interrupt, and PiP (Phase 1-B)
Client half of the PiP observation window, wired to the host
ObservationService over the /bsk-observation HTTP+SSE seam:

- ObservationClientStore: initial state fetch + SSE increments +
  on-demand thumbnail blob loading (session-authorized readAttachment,
  same path as the toolview) + interrupt POST. All I/O injectable.
- ObservationOverlay registered into the shell.overlay list seat (the
  sanctioned frame-wide surface; root is off-limits). Hidden with no
  owned sessions; appears on first start; vanishes when all stop.
- Focus view: status row (green/grey/red dot + session + action + mm:ss
  ticking), breathing thumbnail (new frames fade in, failures keep the
  last frame with a warning badge, pre-navigate placeholder), and the
  action area.
- Interrupt: single click, no confirm; greys into 'Interrupting…' and
  back; disabled when nothing is in flight; one-time tooltip explains
  the semantics (stops the current action only; the run continues).
- Card: drag-move via the header, corner-handle resize (min 240x180,
  max 80% viewport), both clamped and kept for the page lifetime;
  collapsible status capsule.
- PiP: Pop out (user gesture) opens Document PiP with the card's
  current size, portals the same content in, and clones the document's
  style nodes; pagehide falls back to the card with state intact;
  unsupported browsers hide the button; nothing auto-pops.

Tests (+17, 89 total): store (fetch/SSE apply/malformed frames/
thumbnail lifecycle/interrupt wire/stop cleanup) and overlay (hidden->
visible lifecycle, status row, thumbnail via loader, interrupt
disabled/active/hint-once, capsule, resize clamps, move clamps, PiP
unsupported, PiP pop/fallback). Styling follows web-styling (dsw-alias
tokens, CSS Modules, focus-visible, reduced-motion).
2026-08-14 05:33:34 +00:00
BB-fat a7bc8bacf6 feat(dsh-plugin-browserskill): host-side ObservationService for the PiP overlay (Phase 1-A)
Per-session live observation for owned sessions only, feeding the
upcoming client overlay:

- State model: SessionObservation (sessionId/url/action/since/
  thumbnailAttachmentId/lastError), one entry per owned session;
  added on start, removed on stop, cleared on dispose.
- Instrumentation: the shared runBsk wrapper begins/ends an action
  (tool name mapped to a verb) around every model-facing call and tags
  the child with its session id; action end triggers an immediate
  thumbnail refresh.
- Thumbnail loop: 1.5s cadence while active, 8s idle downclock, x3
  failure backoff, silent frame retention on errors; captures run
  OUTSIDE the instrumentation (no action events, no current-pointer
  movement) and land in the attachment store by reference.
- Interrupt: runner.killFor(tag) kills exactly the tagged in-flight bsk
  children — chat-Stop-equivalent semantics for the current or a named
  owned session; foreign ids and the no-in-flight case return false.
- Remote seam: dsh 0.1's Typert Remote pipeline and forwarded-event
  allowlist are closed to out-of-tree packages, so state/events/
  interrupt are served over the documented webServer route seam:
  GET /bsk-observation/state, GET /bsk-observation/events (SSE),
  POST /bsk-observation/interrupt; routes mount only when a webServer
  service exists (headless compositions skip them).
- Config: observationEnabled (true), thumbnailIntervalMs (1500),
  idleIntervalMs (8000).

Tests (+13, 72 total): state machine + event stream, cadence fast/idle/
backoff, headless no-store silence, observation-traffic isolation,
interrupt routing (default/specified/none/foreign), HTTP route
handlers, disabled-webServer no-op.
2026-08-14 04:59:26 +00:00
BB-fat 3d050e0f26 feat(dsh-plugin-browserskill): strict ownership boundary — foreign sessions are invisible
Tighten the shared-daemon boundary from 'cleanup only touches owned
sessions' to 'the plugin only ever sees and operates on sessions it
created':

- browser_session_list no longer queries the daemon at all; it returns
  the plugin's own session table (with the current marker), so foreign
  sessions cannot even be enumerated through the plugin.
- Every tool's optional session argument must name a plugin-created
  session; foreign/unknown ids are rejected with a clear error before
  any command reaches the daemon (the reference-adoption channel is
  gone).
- Unchanged: dispose stops exactly the owned set, stop requires
  ownership, the start reservation protocol, and handle-precise child
  kills.

Tests updated (59): explicit-foreign rejection without daemon contact,
registry-only listing, reservation race, stale-handle-tolerant dispose.
e2e: a manually started foreign session was rejected on reference,
absent from session_list, refused on stop, and survived dispose while
both plugin sessions were cleaned up.
2026-08-14 04:12:38 +00:00
BB-fat afdc398730 fix(dsh-plugin-browserskill): strict session ownership on a shared daemon
Review hardening for daemon-sharing deployments (other agents,
terminals, or dsh instances on the same bsk daemon):

- Ownership split: only sessions created by browser_session_start are
  'owned'. Explicitly referenced foreign sessions are still tracked for
  the current-session pointer, but browser_session_stop refuses them
  and unload cleanup stops exactly the owned set — no path can stop a
  session this plugin did not create.
- Start race: the capacity check is now a synchronous
  reserveStart/completeStart/abandonStart protocol, so two concurrent
  starts can never both pass the cap (check-and-reserve is atomic on
  the event loop); a rejected or failed start never leaks a session.
- Stale handles: dispose already swallowed per-stop errors; the owned
  set is computed at dispose time so an externally stopped session is a
  no-op, not a failure.

Tests (+7): reservation race (two concurrent starts, cap 1 -> exactly
one spawn), foreign-stop refusal (no stop command reaches the daemon),
reference adoption never becoming owned, resolveForStop pointer
stability, and an apply-level dispose test proving only owned sessions
are stopped while stale stops are tolerated. e2e: a manually started
foreign session survived the plugin's full lifecycle while both plugin
sessions were cleaned up.
2026-08-14 03:45:52 +00:00
BB-fat 6b96fcca32 feat(dsh-plugin-browserskill): add the web client half with a browser_screenshot toolview
Make the package dual-face: a dsh.client declaration (platform 'web')
plus an exports["./client"] bundle built in dsh's closure-factory shape
(window.__ModuleLoader__.load handoff, platform modules external,
everything else inlined, CSS Modules via lightningcss).

The client registers a keyed 'tool.call.toolview' view for
browser_screenshot that keeps the terminal block (command + output) and,
when the settled result carries an image block, resolves the durable
attachment through the client session's authorized readAttachment RPC
and renders it with the shared MessageImage thumbnail/lightbox atoms.
Path-only results render unchanged; every other browser_* tool keeps
the stock terminal card.

Tests: view-model derivation (running/ok/error, image vs path-only),
component rendering through @testing-library/react + happy-dom, and the
registration key. biome.json now excludes the package's lib/ build
output (same treatment as dist/).
2026-08-14 02:54:33 +00:00
BB-fat 0771292336 fix(dsh-plugin-browserskill): keep the terminal result card for multi-block results
browser_screenshot on an image-capable route returns text + image
blocks; the shared result presenter required exactly one text block and
bailed to the generic card (tool name + raw args). Project the first
text block instead so the terminal card shows in both modes.
2026-08-13 18:06:57 +00:00
BB-fat 54869cabd0 fix(dsh-plugin-browserskill): include the bsk error hint in the thrown message
The daemon's JSON error envelope carries an actionable hint (e.g.
'choose an input, textarea, or contenteditable element from the latest
snapshot'), but the thrown BskError only embedded the message field, so
the model-facing error text never showed the hint. Append it.
2026-08-13 17:56:16 +00:00
BB-fat 2de7c5c6ff fix(dsh-plugin-browserskill): enforce the session cap before spawning
The concurrency cap was checked in registry.add() AFTER
`bsk session start` had already created the session, so a rejected
start leaked a live session outside the plugin's tracking (found by the
real-browser e2e: the rejected third session stayed in
`bsk session list` after dispose). Check capacity before spawning;
registry.add() keeps the same check as a backstop.
2026-08-13 15:27:29 +00:00
BB-fat 703fc64399 feat(packages): add dsh-plugin-browserskill — DeepSeek Harness tools for bsk
Add a new monorepo package that registers BrowserSkill (bsk) browser
automation as model-visible tools in DeepSeek Harness (dsh):

- 11 tools mapping to bsk CLI --json commands: session start/stop/list,
  navigate, snapshot, observe, click, fill, press, screenshot, emulate
- multi-session support: optional session arg with a current-session
  pointer, session id echoed in every result, configurable concurrency
  cap (default 5), and dispose-time cleanup of plugin-started sessions
- cancellation: exec.signal aborts kill the underlying bsk child process
- terminal-style UI cards (command line as the call card, output as the
  result card); screenshots inline the PNG through the host attachment
  store when the model route accepts image input, else return the path
- config: bskPath / defaultTimeoutMs / maxSessions via Schemastery schema
- install guidance when the bsk binary is missing; bsk JSON error
  envelopes (code/message/hint) surfaced to the model
- unit tests mock the bsk runner (no real browser required); the root
  lint script now also runs the package's typecheck + tests so the
  existing CI frontend job covers the new package (a dedicated CI job
  diff is provided in the PR; the pushing token lacks workflow scope)
2026-08-13 15:04:52 +00:00
130 changed files with 19170 additions and 523 deletions
+105
View File
@@ -0,0 +1,105 @@
name: Release dsh plugin
on:
push:
tags:
- dsh-plugin-v*
workflow_dispatch:
permissions:
contents: read
concurrency:
group: release-dsh-plugin-${{ github.ref }}
cancel-in-progress: false
env:
PACKAGE_NAME: "@wxg-prc-cpg/browser-skill-dsh-plugin"
PACKAGE_DIR: packages/dsh-plugin-browserskill
jobs:
resolve:
runs-on: ubuntu-latest
outputs:
version: ${{ steps.version.outputs.version }}
tag: ${{ steps.version.outputs.tag }}
steps:
- uses: actions/checkout@v6
- name: Resolve plugin version
id: version
shell: bash
run: |
set -euo pipefail
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
VERSION="$(node -p "require('./${{ env.PACKAGE_DIR }}/package.json').version")"
else
VERSION="${GITHUB_REF_NAME#dsh-plugin-v}"
fi
VERSION="${VERSION#v}"
echo "version=${VERSION}" >> "$GITHUB_OUTPUT"
echo "tag=dsh-plugin-v${VERSION}" >> "$GITHUB_OUTPUT"
guard:
needs: resolve
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Verify tag matches package.json name and version
shell: bash
run: |
set -euo pipefail
PKG_NAME="$(node -p "require('./${{ env.PACKAGE_DIR }}/package.json').name")"
PKG_VERSION="$(node -p "require('./${{ env.PACKAGE_DIR }}/package.json').version")"
RELEASE_VERSION="${{ needs.resolve.outputs.version }}"
if [ "$PKG_NAME" != "$PACKAGE_NAME" ]; then
echo "package.json name (${PKG_NAME}) does not match expected ${PACKAGE_NAME}"
exit 1
fi
if [ "$PKG_VERSION" != "$RELEASE_VERSION" ]; then
echo "${PACKAGE_DIR}/package.json version (${PKG_VERSION}) does not match release version (${RELEASE_VERSION})"
exit 1
fi
echo "Version guard passed: ${PKG_NAME}@${RELEASE_VERSION}"
publish:
needs: [resolve, guard]
runs-on: ubuntu-latest
# Secret NPM_TOKEN is stored on the "npm-publish" GitHub Environment.
environment: npm-publish
permissions:
contents: read
steps:
- uses: actions/checkout@v6
- uses: pnpm/action-setup@v6
- uses: actions/setup-node@v6
with:
node-version: 22
cache: pnpm
registry-url: https://registry.npmjs.org
scope: "@wxg-prc-cpg"
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Typecheck plugin
run: pnpm --filter "${{ env.PACKAGE_NAME }}" typecheck
- name: Test plugin
run: pnpm --filter "${{ env.PACKAGE_NAME }}" test
- name: Publish to npm
working-directory: ${{ env.PACKAGE_DIR }}
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
run: |
set -euo pipefail
if [ -z "${NODE_AUTH_TOKEN:-}" ]; then
echo "::error::Environment secret NPM_TOKEN is not configured"
exit 1
fi
pnpm publish --access public --no-git-checks
echo "Published ${PACKAGE_NAME}@${{ needs.resolve.outputs.version }}"
+3
View File
@@ -3,3 +3,6 @@
**/.output/**
**/target/**
**/node_modules/**
# Generated client CSS (tailwind compile output); built by
# packages/dsh-plugin-browserskill/scripts/build-client-css.mjs.
**/dsh-plugin-browserskill/src/client/bsk-ui.nomodule.css
+9 -5
View File
@@ -34,13 +34,17 @@ where only `extension connected` fails is expected; go to Step 3.
## 3. Open the extension install page
If `extension connected` is `FAIL` (`0 browsers connected`), the user likely
has not installed or enabled the browser extension yet. Open the
[Chrome Web Store install page](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi)
in the user's browser, then tell the user:
has not installed or enabled the browser extension yet. Open the install page
matching the user's browser —
[Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi)
for Chrome and other Chromium browsers,
[Edge Add-ons](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg)
for Microsoft Edge — then tell the user:
> Install **BrowserSkill** from the
> [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi),
> open the popup, and wait until it turns green. Reply when done.
> [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi)
> (or [Edge Add-ons](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg)
> on Edge), open the popup, and wait until it turns green. Reply when done.
If opening the page fails, give the user the same link. Then run `bsk doctor`
once more. All `ok`/`na` → tell the user it's ready.
Generated
+1
View File
@@ -137,6 +137,7 @@ name = "bsk"
version = "0.1.10"
dependencies = [
"anyhow",
"base64",
"bsk-protocol",
"clap",
"console",
+1
View File
@@ -36,6 +36,7 @@ flate2 = "1.1.9"
tar = "0.4.46"
zip = { version = "7.2.0", default-features = false, features = ["deflate"] }
sha2 = "0.11.0"
base64 = "0.22"
tempfile = "3"
nix = { version = "0.29", features = ["signal", "process"] }
rand = "0.8"
+37 -5
View File
@@ -13,7 +13,7 @@
</p>
**BrowserSkill** connects Cursor, Claude Code, Codex, OpenClaw, CodeBuddy,
WorkBuddy, Pi, Hermes Agent, and other shell-capable AI agents to your already logged-in
WorkBuddy, Pi, Hermes Agent, DeepSeek Harness, and other AI agents to your already logged-in
browser.
Need the agent to touch a tab you already have open? It must borrow that tab
@@ -67,7 +67,8 @@ Set up browser-skill on this machine by following https://raw.githubusercontent.
<br>
Install the CLI, then install the extension from the [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi).
Install the CLI, then install the extension from the [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi)
or [Edge Add-ons](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg).
#### 1. Install the `bsk` CLI
@@ -91,7 +92,14 @@ bsk --version
#### 2. Install the browser extension
Install BrowserSkill from the [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi).
Install BrowserSkill from your browser's store:
| Browser | Store listing |
| --- | --- |
| Chrome | [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi) |
| Microsoft Edge | [Edge Add-ons](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg) |
On other Chromium-based browsers, install the Chrome Web Store build.
#### 3. Install the skill
@@ -123,7 +131,8 @@ internal variants and install paths.
Other shell-capable agent harnesses are supported too. Copy
[`skill/SKILL.md`](skill/SKILL.md) into your harness's skills directory as
`browser-skill/SKILL.md` to install the skill manually.
`browser-skill/SKILL.md` to install the skill manually. DeepSeek Harness uses a
dedicated plugin instead — see [DeepSeek Harness plugin](#deepseek-harness-plugin).
</details>
@@ -133,6 +142,26 @@ Start a new Agent session and write a prompt that needs the browser, for example
/browser-skill open example.com and summarize what is on the page.
```
## DeepSeek Harness plugin
Using [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (`dsh`)?
BrowserSkill ships a first-class dsh plugin on npm as
[`@wxg-prc-cpg/browser-skill-dsh-plugin`](https://www.npmjs.com/package/@wxg-prc-cpg/browser-skill-dsh-plugin).
It injects native `browser_*` tools (no shelling out to `bsk`) and a live Web UI
overlay of each Agent Window.
Add it to a dsh profile, then start that profile:
```sh
dsh plugin --profile web add @wxg-prc-cpg/browser-skill-dsh-plugin
dsh --profile web
```
The plugin carries its own copy of the skill, so `bsk install-skill` is not needed
for dsh — but the `bsk` CLI and the browser extension are still prerequisites. See
the [plugin README](packages/dsh-plugin-browserskill/README.md) for the tool list,
configuration, and the observation overlay.
## How It Works
BrowserSkill is a local bridge between your agent harness and your browser.
@@ -166,7 +195,9 @@ flowchart TB
The agent never talks to the browser directly. It asks the `bsk` CLI to perform a
browser task; the local daemon routes that request to the extension; the
extension runs it in an Agent Window.
extension runs it in an Agent Window. DeepSeek Harness takes the same path
through the [plugin](#deepseek-harness-plugin): the agent calls injected
`browser_*` tools, and the plugin invokes `bsk` on its behalf.
## For Developers
@@ -176,6 +207,7 @@ The repository is a Cargo + pnpm workspace:
- `crates/bsk-protocol` — shared wire types and JSON schemas
- `apps/extension` — browser extension
- `packages/ui` and `packages/i18n` — shared extension UI support
- `packages/dsh-plugin-browserskill` — DeepSeek Harness plugin (`@wxg-prc-cpg/browser-skill-dsh-plugin`)
## License
+32 -7
View File
@@ -12,7 +12,7 @@
<a href="README.md">English</a> · 中文
</p>
**BrowserSkill** 把 Cursor、Claude Code、Codex、OpenClaw、CodeBuddy、WorkBuddy、Pi、Hermes Agent 等支持 Shell 的 AI Agent 连接到你已登录的浏览器。
**BrowserSkill** 把 Cursor、Claude Code、Codex、OpenClaw、CodeBuddy、WorkBuddy、Pi、Hermes Agent、DeepSeek Harness 等 AI Agent 连接到你已登录的浏览器。
需要 Agent 操作你已打开的标签页?必须显式借用该标签,任务结束后归还,其余浏览器窗口不受影响。
@@ -54,7 +54,8 @@ BrowserSkill 由两个本地运行组件组成:`bsk` CLI/daemon 和浏览器
<br>
先安装 CLI,再从 [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi) 安装浏览器扩展。
先安装 CLI,再从 [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi)
或 [Edge 加载项商店](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg) 安装浏览器扩展。
#### 1. 安装 `bsk` CLI
@@ -64,8 +65,11 @@ BrowserSkill 由两个本地运行组件组成:`bsk` CLI/daemon 和浏览器
curl -fsSL https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.sh | sh
```
**Windows**:从 [最新 CLI release](https://github.com/Tencent/BrowserSkill/releases/latest)
下载 `bsk-v<version>-x86_64-pc-windows-msvc.zip`,解压后将 `bsk.exe` 加入 `PATH`。
**Windows**(PowerShell,安装到 `~/.local/bin`):
```powershell
irm https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.ps1 | iex
```
验证二进制:
@@ -75,7 +79,14 @@ bsk --version
#### 2. 安装浏览器扩展
从 [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi) 安装 BrowserSkill。
在对应浏览器的商店安装 BrowserSkill:
| 浏览器 | 商店页面 |
| --- | --- |
| Chrome | [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi) |
| Microsoft Edge | [Edge 加载项商店](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg) |
其他基于 Chromium 的浏览器,安装 Chrome Web Store 版本即可。
#### 3. 安装 skill
@@ -102,7 +113,7 @@ bsk install-skill
用 <kbd>Space</kbd> 选择需要安装的 Agent harness,然后按 <kbd>Enter</kbd> 安装 skill。运行 `bsk install-skill --list` 可查看 internal 变体及安装路径。
其他支持 Shell 的 Agent harness 也可使用 BrowserSkill,但需手动将 [`skill/SKILL.md`](skill/SKILL.md) 复制到对应 skills 目录下的 `browser-skill/SKILL.md`。
其他支持 Shell 的 Agent harness 也可使用 BrowserSkill,但需手动将 [`skill/SKILL.md`](skill/SKILL.md) 复制到对应 skills 目录下的 `browser-skill/SKILL.md`。DeepSeek Harness 走独立插件,见 [DeepSeek Harness 插件](#deepseek-harness-插件)。
</details>
@@ -112,6 +123,19 @@ bsk install-skill
/browser-skill open example.com and summarize what is on the page.
```
## DeepSeek Harness 插件
在用 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness)(`dsh`)?BrowserSkill 提供了官方 dsh 插件,已发布到 npm:[`@wxg-prc-cpg/browser-skill-dsh-plugin`](https://www.npmjs.com/package/@wxg-prc-cpg/browser-skill-dsh-plugin)。它会注入原生 `browser_*` 工具(无需再通过 Shell 调用 `bsk`),并在 Web UI 中实时观察每个 Agent Window。
把它装进某个 dsh profile,然后启动该 profile:
```sh
dsh plugin --profile web add @wxg-prc-cpg/browser-skill-dsh-plugin
dsh --profile web
```
插件自带 skill,所以在 dsh 下无需执行 `bsk install-skill`;但 `bsk` CLI 和浏览器扩展仍是前置条件。工具清单、配置项与观察浮层见[插件 README](packages/dsh-plugin-browserskill/README.md)。
## 工作原理
BrowserSkill 是 Agent 运行时与浏览器之间的本地桥接层。
@@ -143,7 +167,7 @@ flowchart TB
style UserWindows fill:#f8fafc,stroke:#cbd5e1,color:#334155
```
Agent 不直接与浏览器通信。它通过 `bsk` CLI 下发浏览器任务;本地 daemon 把请求路由到扩展;扩展在 Agent Window 中执行。
Agent 不直接与浏览器通信。它通过 `bsk` CLI 下发浏览器任务;本地 daemon 把请求路由到扩展;扩展在 Agent Window 中执行。DeepSeek Harness 走同一条链路,只是经由 [插件](#deepseek-harness-插件):Agent 调用注入的 `browser_*` 工具,由插件代为执行 `bsk`。
## 面向开发者
@@ -153,6 +177,7 @@ Agent 不直接与浏览器通信。它通过 `bsk` CLI 下发浏览器任务;
- `crates/bsk-protocol` — 共享协议类型与 JSON Schema
- `apps/extension` — 浏览器扩展
- `packages/ui` 和 `packages/i18n` — 扩展 UI 共享支持
- `packages/dsh-plugin-browserskill` — DeepSeek Harness 插件(`@wxg-prc-cpg/browser-skill-dsh-plugin`)
## 许可证
+6 -3
View File
@@ -1,6 +1,6 @@
# BrowserSkill — Privacy Policy
**Last updated:** May 25, 2026
**Last updated:** August 20, 2026
This Privacy Policy describes how the **BrowserSkill** browser extension (the "Extension") handles information when you install and use it. BrowserSkill is published as part of the open-source [BrowserSkill](https://github.com/Tencent/BrowserSkill) project. The source code is publicly auditable.
@@ -14,7 +14,7 @@ BrowserSkill is a local automation bridge that lets AI coding agents (such as Cu
## 2. Single Purpose
The Extension's single purpose is to expose browser automation primitives (navigation, DOM observation, screenshots, clicks, form filling, tab management) to a locally running BrowserSkill daemon over a WebSocket connection on `127.0.0.1`, so that an AI agent invoked by the user can interact with web pages on the user's behalf.
The Extension's single purpose is to expose browser automation primitives (navigation, DOM observation, screenshots, clicks, form filling, task-scoped file transfer, and tab management) to a locally running BrowserSkill daemon over a WebSocket connection on `127.0.0.1`, so that an AI agent invoked by the user can interact with web pages on the user's behalf.
## 3. Data the Extension Accesses
@@ -26,6 +26,7 @@ Depending on the commands the user (via their AI agent) sends to the local daemo
| **User input simulated by the agent** | Mouse clicks, keystrokes, and form values that the AI agent dispatches through the Chrome DevTools Protocol (CDP). | Required to perform automation actions the user has asked the agent to do. |
| **Tab and window metadata** | Tab IDs, URLs, titles, window IDs of the Agent Window and any tabs the user explicitly authorizes. | Required to target automation commands at the correct tab/window. |
| **Local extension storage** | A randomly generated 8-character instance ID and an optional user-supplied label. | Used so the local daemon can recognize this browser instance across reconnects. No personal data is stored. |
| **File transfers requested by the agent** | Local files explicitly supplied to `bsk upload`, and the file created by a single `bsk download` action. | Required to attach a task file to a web page or return a browser-generated download to the invoking local agent. |
| **OS notifications** | Permission to display a system notification when the agent requests to "borrow" one of the user's existing tabs. | Required to obtain explicit, per-tab user consent before the agent touches any pre-existing tab. |
## 4. Data the Extension Does **Not** Collect
@@ -34,7 +35,7 @@ BrowserSkill does **not**:
- Send any data to remote servers, the Extension's authors, or any third party.
- Call any LLM, AI, or cloud API. The Extension contains no API keys, model identifiers, or remote endpoints.
- Read or transmit cookies, browsing history, bookmarks, downloads, saved passwords, or autofill data.
- Read or transmit cookies, browsing history, bookmarks, saved passwords, or autofill data. It observes only the download initiated by an active `bsk download` call, not download history generally.
- Use webcam, microphone, geolocation, or any device sensor.
- Include analytics, telemetry, crash reporting, advertising SDKs, or fingerprinting code.
- Track users across websites or across sessions.
@@ -50,6 +51,7 @@ The Extension requests the following Chrome permissions. Each is used solely for
- **`alarms`** — Periodically wake the service worker to keep the local WebSocket connection alive.
- **`idle`** — Detect when the device returns from idle/locked so the Extension can promptly re-establish the local WebSocket connection after the machine wakes. No idle data is stored or transmitted.
- **`notifications`** — Show a system notification to obtain user approval before the agent borrows a user-owned tab.
- **`downloads`** — Correlate and route the one browser download initiated by an active `bsk download` command. If that claimed transaction fails, BrowserSkill cancels an in-progress file or removes its completed temporary browser file. It is not used to enumerate download history or alter unclaimed downloads.
- **`storage`** — Persist a random instance ID and optional label in `chrome.storage.local`.
- **Host permission `<all_urls>`** — Inject a small status overlay (showing "Agent Active") on pages controlled by the agent, and enable automation across whatever sites the user directs the agent to. The Extension does **not** read or transmit page content from sites the agent is not actively driving.
@@ -61,6 +63,7 @@ All Extension activity stays on the user's local device. The only network traffi
- The instance ID and optional label persist in `chrome.storage.local` until the user uninstalls the Extension or clears extension storage.
- Page content, screenshots, DOM snapshots, and other observed data are returned to the local daemon in response to commands and are **not retained by the Extension**. They live only as long as the agent's tool call.
- Upload and download bytes are staged by the local daemon in a private, session-scoped directory. Download staging is removed after it is copied to the requested destination. Upload staging is retained until the session ends so a later form submission can still read the attached file. Remaining staging is removed when the session ends or disconnects, or when the daemon next starts after a crash.
## 8. User Control
@@ -1,5 +1,5 @@
import { describe, expect, it, vi } from "vitest";
import { type CdpDebuggerApi, ChromiumCdp } from "../chromium-cdp";
import { type CdpDebuggee, type CdpDebuggerApi, ChromiumCdp } from "../chromium-cdp";
function fakeChromeEvent<TArgs extends unknown[]>() {
const listeners = new Set<(...args: TArgs) => void>();
@@ -14,7 +14,7 @@ function fakeChromeEvent<TArgs extends unknown[]>() {
}
function fakeApi() {
const onEvent = fakeChromeEvent<[chrome.debugger.Debuggee, string, unknown]>();
const onEvent = fakeChromeEvent<[CdpDebuggee, string, unknown]>();
const onDetach = fakeChromeEvent<[chrome.debugger.Debuggee, string]>();
const api: CdpDebuggerApi = {
attach: vi.fn(async () => {}),
@@ -29,6 +29,135 @@ function fakeApi() {
}
describe("ChromiumCdp", () => {
it("discovers multiple iframe targets and recursively routes nested OOPIF commands", async () => {
const { api, onEvent } = fakeApi();
(api.sendCommand as ReturnType<typeof vi.fn>).mockImplementation(
async (
target: chrome.debugger.Debuggee & { sessionId?: string },
method: string,
params?: { frameId?: string },
) => {
if (method === "DOM.getFrameOwner") {
return { backendNodeId: params?.frameId === "nested" ? 300 : 200 };
}
if (method !== "Page.getFrameTree") return {};
if (target.sessionId === "right-session") {
return {
frameTree: {
frame: { id: "right", url: "https://right.test" },
childFrames: [
{ frame: { id: "nested", parentId: "right", url: "https://nested.test" } },
],
},
};
}
if (target.sessionId === "nested-session") {
return { frameTree: { frame: { id: "nested", url: "https://nested.test" } } };
}
return {
frameTree: {
frame: { id: "main", url: "https://app.test" },
childFrames: [
{ frame: { id: "left", parentId: "main", url: "https://left.test" } },
{
frame: { id: "right", parentId: "main", url: "https://right.test" },
childFrames: [
{ frame: { id: "nested", parentId: "right", url: "https://nested.test" } },
],
},
],
},
};
},
);
const cdp = new ChromiumCdp(api);
await cdp.ensureAttached(4);
onEvent.fire({ tabId: 4 }, "Target.attachedToTarget", { sessionId: "right-session" });
onEvent.fire({ tabId: 4 } as chrome.debugger.Debuggee, "Target.attachedToTarget", {
sessionId: "nested-session",
});
const graph = await cdp.getFrameGraph(4);
expect(graph.frames).toHaveLength(4);
expect(graph.frames.find((frame) => frame.frameId === "left")?.target).toEqual({ tabId: 4 });
expect(graph.frames.find((frame) => frame.frameId === "right")?.target.sessionId).toBe(
"right-session",
);
expect(graph.frames.find((frame) => frame.frameId === "nested")?.target.sessionId).toBe(
"nested-session",
);
expect(graph.frames.find((frame) => frame.frameId === "right")?.ownerBackendNodeId).toBe(200);
expect(graph.frames.find((frame) => frame.frameId === "nested")?.ownerBackendNodeId).toBe(300);
expect(api.sendCommand).toHaveBeenCalledWith(
{ tabId: 4, sessionId: "right-session" },
"DOM.getFrameOwner",
{ frameId: "nested" },
);
expect(api.sendCommand).toHaveBeenCalledWith(
{ tabId: 4, sessionId: "nested-session" },
"Target.setAutoAttach",
expect.objectContaining({ flatten: true }),
);
});
it("waits for recursively attached iframe targets without a depth limit", async () => {
const { api, onEvent } = fakeApi();
const depth = 6;
const scheduled = new Set<string>();
const nestedTree = (index: number): Record<string, unknown> => ({
frame: {
id: index === 0 ? "main" : `frame-${index}`,
...(index > 0 ? { parentId: index === 1 ? "main" : `frame-${index - 1}` } : {}),
},
...(index < depth ? { childFrames: [nestedTree(index + 1)] } : {}),
});
(api.sendCommand as ReturnType<typeof vi.fn>).mockImplementation(
async (target: CdpDebuggee, method: string, params?: { frameId?: string }) => {
if (method === "Target.setAutoAttach") {
const parentIndex = target.sessionId
? Number(target.sessionId.replace("session-", ""))
: 0;
const nextIndex = parentIndex + 1;
const key = `${target.sessionId ?? "root"}:${nextIndex}`;
if (nextIndex <= depth && !scheduled.has(key)) {
scheduled.add(key);
setTimeout(() => {
onEvent.fire(
{ tabId: 4, ...(target.sessionId ? { sessionId: target.sessionId } : {}) },
"Target.attachedToTarget",
{
sessionId: `session-${nextIndex}`,
targetInfo: { type: "iframe" },
},
);
}, 5);
}
return {};
}
if (method === "Page.getFrameTree") {
if (!target.sessionId) return { frameTree: nestedTree(0) };
const index = Number(target.sessionId.replace("session-", ""));
return { frameTree: nestedTree(index) };
}
if (method === "DOM.getFrameOwner") {
return { backendNodeId: Number(params?.frameId?.replace("frame-", "")) + 100 };
}
return {};
},
);
const cdp = new ChromiumCdp(api);
const graph = await cdp.getFrameGraph(4);
expect(graph.frames).toHaveLength(depth + 1);
for (let index = 1; index <= depth; index += 1) {
expect(
graph.frames.find((frame) => frame.frameId === `frame-${index}`)?.target.sessionId,
).toBe(`session-${index}`);
}
});
it("coalesces concurrent attach calls for the same tab", async () => {
const { api } = fakeApi();
let releaseAttach!: () => void;
@@ -0,0 +1,136 @@
import { describe, expect, it } from "vitest";
import { buildFrameGraph, type CdpFrameTreeNode, type CdpFrameTreeSource } from "../frame-graph";
describe("buildFrameGraph", () => {
it("keeps sibling frames distinct and routes nested OOPIFs to their child sessions", () => {
const graph = buildFrameGraph([
{
target: { tabId: 4 },
tree: {
frame: { id: "main", url: "https://app.test" },
childFrames: [
{ frame: { id: "left", parentId: "main", url: "https://left.test" } },
{
frame: { id: "right", parentId: "main", url: "https://right.test" },
childFrames: [
{ frame: { id: "nested", parentId: "right", url: "https://nested.test" } },
],
},
],
},
},
{
target: { tabId: 4, sessionId: "right-session" },
tree: {
frame: { id: "right", url: "https://right.test" },
childFrames: [{ frame: { id: "nested", parentId: "right", url: "https://nested.test" } }],
},
},
{
target: { tabId: 4, sessionId: "nested-session" },
tree: { frame: { id: "nested", url: "https://nested.test" } },
},
]);
expect(graph?.frames).toHaveLength(4);
expect(graph?.frames.find((frame) => frame.frameId === "left")?.target).toEqual({ tabId: 4 });
expect(graph?.frames.find((frame) => frame.frameId === "right")).toMatchObject({
parentFrameId: "main",
target: { tabId: 4, sessionId: "right-session" },
});
expect(graph?.frames.find((frame) => frame.frameId === "nested")).toMatchObject({
parentFrameId: "right",
target: { tabId: 4, sessionId: "nested-session" },
});
});
it("propagates each target boundary to its same-process descendants", () => {
const sources: CdpFrameTreeSource[] = [
{
target: { tabId: 4 },
tree: {
frame: { id: "main" },
childFrames: [
{
frame: { id: "oopif-a", parentId: "main" },
childFrames: [
{
frame: { id: "same-process-b", parentId: "oopif-a" },
childFrames: [
{
frame: { id: "oopif-c", parentId: "same-process-b" },
childFrames: [{ frame: { id: "same-process-d", parentId: "oopif-c" } }],
},
],
},
],
},
],
},
},
{
target: { tabId: 4, sessionId: "session-a" },
tree: {
frame: { id: "oopif-a" },
childFrames: [
{
frame: { id: "same-process-b", parentId: "oopif-a" },
childFrames: [{ frame: { id: "oopif-c", parentId: "same-process-b" } }],
},
],
},
},
{
target: { tabId: 4, sessionId: "session-c" },
tree: {
frame: { id: "oopif-c" },
childFrames: [{ frame: { id: "same-process-d", parentId: "oopif-c" } }],
},
},
];
const graph = buildFrameGraph(sources);
expect(graph?.frames.map((frame) => [frame.frameId, frame.target.sessionId])).toEqual([
["main", undefined],
["oopif-a", "session-a"],
["same-process-b", "session-a"],
["oopif-c", "session-c"],
["same-process-d", "session-c"],
]);
expect(graph?.frames.find((frame) => frame.frameId === "oopif-a")?.parentFrameId).toBe("main");
expect(graph?.frames.find((frame) => frame.frameId === "oopif-c")?.parentFrameId).toBe(
"same-process-b",
);
const reordered = buildFrameGraph([sources[2], sources[1], sources[0]]);
expect(reordered).toEqual(graph);
});
it("walks deeply nested frame trees without consuming the JavaScript call stack", () => {
const depth = 5_000;
const root: CdpFrameTreeNode = { frame: { id: "frame-0" } };
let parent = root;
for (let index = 1; index <= depth; index += 1) {
const child: CdpFrameTreeNode = {
frame: { id: `frame-${index}`, parentId: `frame-${index - 1}` },
};
parent.childFrames = [child];
parent = child;
}
const graph = buildFrameGraph([{ target: { tabId: 4 }, tree: root }]);
expect(graph?.frames).toHaveLength(depth + 1);
expect(graph?.frames.at(-1)?.frameId).toBe(`frame-${depth}`);
});
it("does not loop when an in-memory frame tree contains an object cycle", () => {
const root: CdpFrameTreeNode = { frame: { id: "main" } };
root.childFrames = [root];
expect(buildFrameGraph([{ target: { tabId: 4 }, tree: root }])?.frames).toEqual([
{ frameId: "main", target: { tabId: 4 } },
]);
});
});
+211 -12
View File
@@ -31,6 +31,15 @@ import type {
NetworkEntryKind,
NetworkResult,
} from "@/transport/types";
import {
buildFrameGraph,
type CdpFrameGraph,
type CdpFrameTreeNode,
type CdpFrameTreeSource,
type CdpTarget,
} from "./frame-graph";
export type CdpDebuggee = chrome.debugger.Debuggee & { sessionId?: string };
/**
* Minimal slice of `chrome.debugger` the rest of the extension
@@ -38,21 +47,15 @@ import type {
* fake without monkey-patching the real `chrome` global.
*/
export interface CdpDebuggerApi {
attach(target: chrome.debugger.Debuggee, requiredVersion: string): Promise<void>;
detach(target: chrome.debugger.Debuggee): Promise<void>;
sendCommand(
target: chrome.debugger.Debuggee,
method: string,
commandParams?: object,
): Promise<unknown>;
attach(target: CdpDebuggee, requiredVersion: string): Promise<void>;
detach(target: CdpDebuggee): Promise<void>;
sendCommand(target: CdpDebuggee, method: string, commandParams?: object): Promise<unknown>;
/**
* Fires for every CDP event (`Page.lifecycleEvent`, `DOM.documentUpdated`,
* …). The first callback argument is the source debuggee; the second
* is the CDP method name; the third is the payload.
*/
onEvent: chrome.events.Event<
(source: chrome.debugger.Debuggee, method: string, params: unknown) => void
>;
onEvent: chrome.events.Event<(source: CdpDebuggee, method: string, params: unknown) => void>;
/**
* Fires when Chrome unilaterally detaches us — most commonly because
* the tab navigated to a chrome:// URL or the user clicked
@@ -110,6 +113,8 @@ const MAX_CONSOLE_STACK_FRAMES = 20;
const MAX_NETWORK_BUFFER = 200;
const MAX_NETWORK_FIELD_LENGTH = 4096;
const MAX_NETWORK_REQUEST_META = 1024;
const FRAME_DISCOVERY_TIMEOUT_MS = 1000;
const FRAME_DISCOVERY_QUIET_MS = 20;
interface ParsedDialogOpening {
type: JavaScriptDialogType;
@@ -131,6 +136,27 @@ interface NetworkRequestMeta {
truncated: boolean;
}
interface FrameDiscoveryState {
sessions: Set<string>;
pending: Set<Promise<void>>;
generation: number;
}
async function settleBeforeDeadline(promises: Promise<void>[], deadline: number): Promise<boolean> {
if (promises.length === 0) return true;
const remaining = deadline - Date.now();
if (remaining <= 0) return false;
let timer: ReturnType<typeof setTimeout> | undefined;
const timedOut = await Promise.race([
Promise.allSettled(promises).then(() => false),
new Promise<boolean>((resolve) => {
timer = setTimeout(() => resolve(true), remaining);
}),
]);
if (timer) clearTimeout(timer);
return !timedOut;
}
/**
* Wrapper around `chrome.debugger` that owns the "attach once per
* tabId" cache and exposes typed `send<T>()`.
@@ -149,10 +175,12 @@ export class ChromiumCdp {
private readonly networkSequences = new Map<number, number>();
private readonly networkDomainsEnabledTabs = new Set<number>();
private readonly networkRequestMeta = new Map<number, Map<string, NetworkRequestMeta>>();
private readonly frameDiscovery = new Map<number, FrameDiscoveryState>();
private detachSubscription: { dispose(): void } | null = null;
private dialogSubscription: { dispose(): void } | null = null;
private consoleSubscription: { dispose(): void } | null = null;
private networkSubscription: { dispose(): void } | null = null;
private frameTargetSubscription: { dispose(): void } | null = null;
constructor(api: CdpDebuggerApi = chromeDebuggerApi) {
this.api = api;
@@ -160,6 +188,7 @@ export class ChromiumCdp {
this.bindDialogHandler();
this.bindConsoleHandler();
this.bindNetworkHandler();
this.bindFrameTargetHandler();
}
/** Attach to `tabId` if we haven't already in this driver. */
@@ -177,6 +206,9 @@ export class ChromiumCdp {
await this.enableConsoleDomains(tabId);
await this.enableNetworkDomainBestEffort(tabId);
this.attachedTabs.add(tabId);
await this.enableFrameDiscovery({ tabId }).catch((err) => {
console.debug("[bsk cdp] frame discovery unavailable", { tabId, err });
});
} catch (err) {
// A CDP domain enable failed after the raw attach succeeded
// (e.g. `Page.enable` rejects because the tab just navigated to
@@ -222,6 +254,73 @@ export class ChromiumCdp {
}
}
async sendToTarget<T = unknown>(target: CdpTarget, method: string, params?: object): Promise<T> {
if (!this.attachedTabs.has(target.tabId)) {
await this.ensureAttached(target.tabId);
}
try {
return (await this.api.sendCommand(target, method, params ?? {})) as T;
} catch (err) {
throw normalizeError(err);
}
}
async getFrameGraph(tabId: number): Promise<CdpFrameGraph> {
await this.ensureAttached(tabId);
await this.enableFrameDiscovery({ tabId }).catch(() => {});
await this.drainFrameAttachTasks(tabId);
const sources: CdpFrameTreeSource[] = [];
const root = await this.sendToTarget<{ frameTree?: CdpFrameTreeNode }>(
{ tabId },
"Page.getFrameTree",
{},
);
if (root.frameTree) sources.push({ target: { tabId }, tree: root.frameTree });
const sessions = [...(this.frameDiscovery.get(tabId)?.sessions ?? [])];
const childTrees = await Promise.all(
sessions.map(async (sessionId): Promise<CdpFrameTreeSource | null> => {
const target = { tabId, sessionId };
try {
const reply = await this.sendToTarget<{ frameTree?: CdpFrameTreeNode }>(
target,
"Page.getFrameTree",
{},
);
return reply.frameTree ? { target, tree: reply.frameTree } : null;
} catch {
return null;
}
}),
);
for (const source of childTrees) {
if (source) sources.push(source);
}
const graph = buildFrameGraph(sources);
if (!graph) throw new Error("Page.getFrameTree returned no root frame");
const frameById = new Map(graph.frames.map((frame) => [frame.frameId, frame]));
await Promise.all(
graph.frames.map(async (frame) => {
if (!frame.parentFrameId) return;
const parent = frameById.get(frame.parentFrameId);
if (!parent) return;
try {
const owner = await this.sendToTarget<{ backendNodeId?: number }>(
parent.target,
"DOM.getFrameOwner",
{ frameId: frame.frameId },
);
if (owner.backendNodeId !== undefined) frame.ownerBackendNodeId = owner.backendNodeId;
} catch {
// The frame may have navigated between tree capture and owner lookup.
}
}),
);
return graph;
}
/** Return a cursor marking the current dialog sequence for `tabId`. */
dialogCursor(tabId: number): DialogCursor {
return this.dialogSequences.get(tabId) ?? 0;
@@ -332,6 +431,7 @@ export class ChromiumCdp {
this.clearDialogState(tabId);
this.clearConsoleState(tabId);
this.clearNetworkState(tabId);
this.clearFrameState(tabId);
try {
await this.api.detach({ tabId });
} catch (err) {
@@ -354,7 +454,7 @@ export class ChromiumCdp {
}
/** Subscribe to all CDP events. Returned disposable removes the listener. */
onEvent(handler: (source: chrome.debugger.Debuggee, method: string, params: unknown) => void): {
onEvent(handler: (source: CdpDebuggee, method: string, params: unknown) => void): {
dispose(): void;
} {
this.api.onEvent.addListener(handler);
@@ -378,6 +478,7 @@ export class ChromiumCdp {
this.networkSequences.clear();
this.networkDomainsEnabledTabs.clear();
this.networkRequestMeta.clear();
this.frameDiscovery.clear();
await Promise.all(
tabs.map(async (tabId) => {
try {
@@ -428,9 +529,104 @@ export class ChromiumCdp {
}
}
private async enableFrameDiscovery(target: CdpTarget): Promise<void> {
await this.api.sendCommand(target, "Target.setAutoAttach", {
autoAttach: true,
waitForDebuggerOnStart: false,
flatten: true,
filter: [{ type: "iframe", exclude: false }],
});
}
private bindFrameTargetHandler(): void {
if (this.frameTargetSubscription) return;
const listener = (source: CdpDebuggee, method: string, params: unknown) => {
const tabId = source.tabId;
if (typeof tabId !== "number") return;
const raw = (params ?? {}) as Record<string, unknown>;
const sessionId = typeof raw.sessionId === "string" ? raw.sessionId : undefined;
if (!sessionId) return;
if (method === "Target.detachedFromTarget") {
const state = this.frameDiscovery.get(tabId);
if (state?.sessions.delete(sessionId)) state.generation += 1;
return;
}
if (method !== "Target.attachedToTarget") return;
const targetInfo = raw.targetInfo as { type?: string } | undefined;
if (targetInfo?.type && targetInfo.type !== "iframe") return;
const state = this.frameDiscoveryState(tabId);
if (state.sessions.has(sessionId)) return;
state.sessions.add(sessionId);
state.generation += 1;
const task = this.initializeFrameTarget({ tabId, sessionId });
state.pending.add(task);
void task.finally(() => {
state.pending.delete(task);
});
};
this.api.onEvent.addListener(listener);
this.frameTargetSubscription = {
dispose: () => this.api.onEvent.removeListener(listener),
};
}
private async initializeFrameTarget(target: CdpTarget): Promise<void> {
try {
await this.enableFrameDiscovery(target);
} catch (err) {
const state = this.frameDiscovery.get(target.tabId);
if (state?.sessions.delete(target.sessionId as string)) state.generation += 1;
console.debug("[bsk cdp] child frame target initialization failed", { target, err });
}
}
private async drainFrameAttachTasks(tabId: number): Promise<void> {
const deadline = Date.now() + FRAME_DISCOVERY_TIMEOUT_MS;
while (Date.now() < deadline) {
const state = this.frameDiscovery.get(tabId);
const generation = state?.generation ?? 0;
if (state?.pending.size && !(await settleBeforeDeadline([...state.pending], deadline))) break;
// Target.setAutoAttach may enqueue the next attachedToTarget event after
// its command promise settles. Wait for a short quiet window, then finish
// only if neither the state object nor its generation changed.
const quietTime = Math.min(FRAME_DISCOVERY_QUIET_MS, deadline - Date.now());
if (quietTime <= 0) break;
await new Promise<void>((resolve) => setTimeout(resolve, quietTime));
const current = this.frameDiscovery.get(tabId);
if (
current === state &&
(current?.generation ?? 0) === generation &&
!current?.pending.size
) {
return;
}
}
console.debug("[bsk cdp] frame discovery did not reach quiescence before timeout", { tabId });
}
private frameDiscoveryState(tabId: number): FrameDiscoveryState {
const existing = this.frameDiscovery.get(tabId);
if (existing) return existing;
const created: FrameDiscoveryState = {
sessions: new Set(),
pending: new Set(),
generation: 0,
};
this.frameDiscovery.set(tabId, created);
return created;
}
private clearFrameState(tabId: number): void {
this.frameDiscovery.delete(tabId);
}
private bindDialogHandler(): void {
if (this.dialogSubscription) return;
const listener = (source: chrome.debugger.Debuggee, method: string, params: unknown) => {
const listener = (source: CdpDebuggee, method: string, params: unknown) => {
if (method !== "Page.javascriptDialogOpening") return;
const tabId = source.tabId;
if (typeof tabId !== "number") return;
@@ -614,6 +810,7 @@ export class ChromiumCdp {
this.clearDialogState(source.tabId);
this.clearConsoleState(source.tabId);
this.clearNetworkState(source.tabId);
this.clearFrameState(source.tabId);
}
};
this.api.onDetach.addListener(listener);
@@ -632,6 +829,8 @@ export class ChromiumCdp {
this.consoleSubscription = null;
this.networkSubscription?.dispose();
this.networkSubscription = null;
this.frameTargetSubscription?.dispose();
this.frameTargetSubscription = null;
}
/** Detach tabs only when no other live session has claimed them. */
@@ -0,0 +1,107 @@
export interface CdpTarget {
tabId: number;
sessionId?: string;
}
export interface CdpFrame {
frameId: string;
parentFrameId?: string;
ownerBackendNodeId?: number;
url?: string;
target: CdpTarget;
}
export interface CdpFrameGraph {
rootFrameId: string;
frames: CdpFrame[];
}
export interface CdpFrameTreeNode {
frame: {
id: string;
parentId?: string;
url?: string;
};
childFrames?: CdpFrameTreeNode[];
}
export interface CdpFrameTreeSource {
target: CdpTarget;
tree: CdpFrameTreeNode;
}
export function cdpTargetKey(target: CdpTarget): string {
return `${target.tabId}:${target.sessionId ?? "root"}`;
}
function mergeFrame(
frames: Map<string, CdpFrame>,
order: string[],
node: CdpFrameTreeNode,
target: CdpTarget,
): void {
const existing = frames.get(node.frame.id);
if (!existing) order.push(node.frame.id);
const parentFrameId = node.frame.parentId || existing?.parentFrameId;
const url = node.frame.url || existing?.url;
frames.set(node.frame.id, {
frameId: node.frame.id,
...(parentFrameId ? { parentFrameId } : {}),
...(url ? { url } : {}),
target,
});
}
function walkFrameTree(
source: CdpFrameTreeSource,
targetByRootFrameId: ReadonlyMap<string, CdpTarget>,
frames: Map<string, CdpFrame>,
order: string[],
): void {
const stack: Array<{ node: CdpFrameTreeNode; inheritedTarget: CdpTarget }> = [
{ node: source.tree, inheritedTarget: source.target },
];
const expanded = new WeakSet<CdpFrameTreeNode>();
while (stack.length > 0) {
const current = stack.pop();
if (!current || expanded.has(current.node)) continue;
expanded.add(current.node);
const target = targetByRootFrameId.get(current.node.frame.id) ?? current.inheritedTarget;
mergeFrame(frames, order, current.node, target);
const children = current.node.childFrames ?? [];
for (let index = children.length - 1; index >= 0; index -= 1) {
stack.push({ node: children[index], inheritedTarget: target });
}
}
}
export function buildFrameGraph(sources: CdpFrameTreeSource[]): CdpFrameGraph | null {
const rootSource = sources.find((source) => source.target.sessionId === undefined) ?? sources[0];
if (!rootSource) return null;
const frames = new Map<string, CdpFrame>();
const order: string[] = [];
const targetByRootFrameId = new Map(
sources.map((source) => [source.tree.frame.id, source.target] as const),
);
// The top-level source is authoritative if malformed input reports the same
// frame as both a root target and a child target.
targetByRootFrameId.set(rootSource.tree.frame.id, rootSource.target);
walkFrameTree(rootSource, targetByRootFrameId, frames, order);
for (const source of sources) {
if (source === rootSource) continue;
walkFrameTree(source, targetByRootFrameId, frames, order);
}
return {
rootFrameId: rootSource.tree.frame.id,
frames: order.flatMap((frameId) => {
const frame = frames.get(frameId);
return frame ? [frame] : [];
}),
};
}
@@ -8,6 +8,7 @@ import {
useRef,
useState,
} from "react";
import type { HelpHighlightRect } from "@/lib/help-bridge";
import logoUrl from "../../assets/logo.png";
export interface HelpRequestData {
@@ -19,6 +20,10 @@ export interface HelpRequestData {
displayMode?: "full" | "compact";
/** CSS selectors to scroll to + flash-highlight. */
selectors: string[];
/** Pre-resolved top-viewport rectangles, used for cross-frame targets. */
rects?: HelpHighlightRect[];
/** Re-resolve cross-frame rectangles after scroll, resize, or layout changes. */
refreshRects?: () => Promise<HelpHighlightRect[] | undefined>;
onContinue: (note: string) => void;
onCancel: () => void;
}
@@ -49,6 +54,7 @@ const PANEL_WIDTH = 420;
const FALLBACK_PANEL_H = 180;
const FALLBACK_PANEL_H_COLLAPSED = 44;
const EMPTY_SELECTORS: string[] = [];
const EMPTY_RECTS: HelpHighlightRect[] = [];
function clamp(value: number, min: number, max: number): number {
return Math.max(min, Math.min(max, value));
@@ -179,6 +185,8 @@ export function HelpRequestOverlay({ request }: Props) {
const { t } = useTranslation("extension");
const isCompact = request?.displayMode === "compact";
const effectiveSelectors = isCompact ? EMPTY_SELECTORS : (request?.selectors ?? EMPTY_SELECTORS);
const [liveRects, setLiveRects] = useState<HelpHighlightRect[] | null>(null);
const effectiveRects = isCompact ? EMPTY_RECTS : (liveRects ?? request?.rects ?? EMPTY_RECTS);
const [note, setNote] = useState("");
const [collapsed, setCollapsed] = useState(false);
const [boxes, setBoxes] = useState<Box[]>([]);
@@ -201,6 +209,7 @@ export function HelpRequestOverlay({ request }: Props) {
// Reset the note and collapse state whenever a new request appears.
useEffect(() => {
setNote("");
setLiveRects(null);
setCollapsed(false);
placementRef.current = null;
setDragPos(null);
@@ -209,6 +218,32 @@ export function HelpRequestOverlay({ request }: Props) {
dragStartRef.current = null;
}, [request?.id]);
useEffect(() => {
if (!request || isCompact || !request.refreshRects) return;
let disposed = false;
let resolving = false;
const refresh = async () => {
if (resolving) return;
resolving = true;
try {
const rects = await request.refreshRects?.();
if (!disposed && rects) setLiveRects(rects);
} finally {
resolving = false;
}
};
const onViewportChange = () => void refresh();
window.addEventListener("scroll", onViewportChange, true);
window.addEventListener("resize", onViewportChange);
const interval = window.setInterval(onViewportChange, 500);
return () => {
disposed = true;
window.removeEventListener("scroll", onViewportChange, true);
window.removeEventListener("resize", onViewportChange);
window.clearInterval(interval);
};
}, [request, isCompact]);
// Scroll the first matched target into view once per distinct request id.
useLayoutEffect(() => {
if (!request) return;
@@ -231,7 +266,7 @@ export function HelpRequestOverlay({ request }: Props) {
return;
}
const update = () => {
const nextBoxes = measure(effectiveSelectors);
const nextBoxes = [...measure(effectiveSelectors), ...effectiveRects];
setBoxes((prev) => (boxesEqual(prev, nextBoxes) ? prev : nextBoxes));
};
update();
@@ -243,7 +278,7 @@ export function HelpRequestOverlay({ request }: Props) {
window.removeEventListener("resize", update);
window.clearInterval(interval);
};
}, [request, effectiveSelectors]);
}, [request, effectiveSelectors, effectiveRects]);
const onHeaderPointerDown = useCallback(
(e: ReactPointerEvent<HTMLDivElement>) => {
@@ -305,7 +340,7 @@ export function HelpRequestOverlay({ request }: Props) {
return;
}
if (userMovedRef.current) return;
const measured = boxes.length > 0 ? boxes : measure(effectiveSelectors);
const measured = boxes.length > 0 ? boxes : [...measure(effectiveSelectors), ...effectiveRects];
const anchor = unionBox(measured);
const banner = bannerRef.current;
if (!anchor || !banner) {
@@ -329,7 +364,7 @@ export function HelpRequestOverlay({ request }: Props) {
}
const nextPos = placePanel(anchor, panelW, panelH, placementRef.current.placement);
setPanelPos((prev) => (panelPosEqual(prev, nextPos) ? prev : nextPos));
}, [request, boxes, collapsed, isCompact, effectiveSelectors]);
}, [request, boxes, collapsed, isCompact, effectiveSelectors, effectiveRects]);
if (!request) return null;
@@ -1,6 +1,6 @@
import { i18n } from "@browser-skill/i18n";
import { I18nextProvider } from "@browser-skill/i18n/react";
import { cleanup, fireEvent, render, screen } from "@testing-library/react";
import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
import { createElement } from "react";
import { afterEach, describe, expect, it, vi } from "vitest";
import { type HelpRequestData, HelpRequestOverlay } from "../HelpRequestOverlay";
@@ -38,6 +38,36 @@ describe("HelpRequestOverlay", () => {
expect(screen.getByText("Please complete the captcha")).toBeTruthy();
});
it("renders explicit viewport rectangles for cross-frame targets", () => {
const { container } = renderOverlay(
baseRequest({ rects: [{ top: 30, left: 40, width: 120, height: 50 }] }),
);
const highlight = container.querySelector<HTMLElement>("[data-slot='help-highlight']");
expect(highlight?.style.top).toBe("30px");
expect(highlight?.style.left).toBe("40px");
expect(highlight?.style.width).toBe("120px");
expect(highlight?.style.height).toBe("50px");
});
it("re-resolves cross-frame rectangles after the viewport changes", async () => {
const refreshRects = vi.fn(async () => [{ top: 80, left: 90, width: 140, height: 60 }]);
const { container } = renderOverlay(
baseRequest({
rects: [{ top: 30, left: 40, width: 120, height: 50 }],
refreshRects,
}),
);
window.dispatchEvent(new Event("scroll"));
await waitFor(() => {
const highlight = container.querySelector<HTMLElement>("[data-slot='help-highlight']");
expect(highlight?.style.top).toBe("80px");
expect(highlight?.style.left).toBe("90px");
});
expect(refreshRects).toHaveBeenCalledTimes(1);
});
it("keeps the inactive render stable", () => {
const { container, rerender } = renderOverlay(null);
@@ -0,0 +1,64 @@
import { describe, expect, it, vi } from "vitest";
import { createHelpRequestData } from "../help-request";
describe("createHelpRequestData", () => {
it("gives live delivery and recovery the same rectangle refresh mechanism", async () => {
const finish = vi.fn();
const query = vi.fn(async () => ({
active: true,
request: {
requestId: "help-1",
prompt: "complete the challenge",
selectors: [],
rects: [{ top: 80, left: 90, width: 140, height: 60 }],
timeoutMs: 1_000,
},
}));
const request = createHelpRequestData(
{
requestId: "help-1",
prompt: "complete the challenge",
selectors: [],
rects: [{ top: 10, left: 20, width: 30, height: 40 }],
timeoutMs: 1_000,
},
{ finish, query },
);
await expect(request.refreshRects?.()).resolves.toEqual([
{ top: 80, left: 90, width: 140, height: 60 },
]);
request.onContinue(" done ");
request.onCancel();
expect(query).toHaveBeenCalledOnce();
expect(finish).toHaveBeenNthCalledWith(1, "help-1", "continued", " done ");
expect(finish).toHaveBeenNthCalledWith(2, "help-1", "cancelled");
});
it("ignores rectangle replies for a replaced request", async () => {
const request = createHelpRequestData(
{
requestId: "help-1",
prompt: "complete the challenge",
selectors: [],
timeoutMs: 1_000,
},
{
finish: vi.fn(),
query: vi.fn(async () => ({
active: true,
request: {
requestId: "help-2",
prompt: "new challenge",
selectors: [],
rects: [{ top: 1, left: 2, width: 3, height: 4 }],
timeoutMs: 1_000,
},
})),
},
);
await expect(request.refreshRects?.()).resolves.toBeUndefined();
});
});
@@ -0,0 +1,31 @@
import type { HelpQueryResponse, HelpRequestMessage } from "@/lib/help-bridge";
import type { HelpRequestData } from "./HelpRequestOverlay";
export interface HelpRequestDataDeps {
finish(requestId: string, outcome: "continued" | "cancelled", note?: string): void;
query(): Promise<HelpQueryResponse | undefined>;
}
/** Build the content-side representation shared by live delivery and recovery. */
export function createHelpRequestData(
message: Omit<HelpRequestMessage, "type">,
deps: HelpRequestDataDeps,
): HelpRequestData {
return {
id: message.requestId,
prompt: message.prompt,
...(message.title ? { title: message.title } : {}),
...(message.displayMode ? { displayMode: message.displayMode } : {}),
selectors: message.selectors,
rects: message.rects,
refreshRects: async () => {
const response = await deps.query();
return response?.active && response.request?.requestId === message.requestId
? response.request.rects
: undefined;
},
onContinue: (note: string) =>
deps.finish(message.requestId, "continued", note.trim() ? note : undefined),
onCancel: () => deps.finish(message.requestId, "cancelled"),
};
}
+14 -21
View File
@@ -7,6 +7,7 @@ import { BorrowConfirmationOverlay } from "@/content/BorrowConfirmationOverlay";
import { ControlOverlay } from "@/content/ControlOverlay";
import { createCaptureSuppressController } from "@/content/capture-suppress";
import { HelpRequestOverlay } from "@/content/HelpRequestOverlay";
import { createHelpRequestData } from "@/content/help-request";
import overlayCss from "@/content/overlay.css?inline";
import { OverlayController, shouldShowAgentControlOverlay } from "@/content/overlay-controller";
import { RecordOverlay } from "@/content/RecordOverlay";
@@ -344,16 +345,7 @@ export default defineContentScript({
if (isHelpRequestMessage(message)) {
const helpMsg = message as HelpRequestMessage;
const previousHelp = overlays.setAgentHelpRequest({
id: helpMsg.requestId,
prompt: helpMsg.prompt,
...(helpMsg.title ? { title: helpMsg.title } : {}),
...(helpMsg.displayMode ? { displayMode: helpMsg.displayMode } : {}),
selectors: helpMsg.selectors,
onContinue: (note: string) =>
void sendHelpFinish(helpMsg.requestId, "continued", note.trim() ? note : undefined),
onCancel: () => void sendHelpFinish(helpMsg.requestId, "cancelled"),
});
const previousHelp = mountHelpRequest(helpMsg);
if (previousHelp && previousHelp.id !== helpMsg.requestId) {
void sendHelpFinish(previousHelp.id, "cancelled");
}
@@ -406,17 +398,18 @@ export default defineContentScript({
});
}
function mountHelpRequest(helpMsg: Omit<HelpRequestMessage, "type">): void {
overlays.setAgentHelpRequest({
id: helpMsg.requestId,
prompt: helpMsg.prompt,
...(helpMsg.title ? { title: helpMsg.title } : {}),
...(helpMsg.displayMode ? { displayMode: helpMsg.displayMode } : {}),
selectors: helpMsg.selectors,
onContinue: (note: string) =>
void sendHelpFinish(helpMsg.requestId, "continued", note.trim() ? note : undefined),
onCancel: () => void sendHelpFinish(helpMsg.requestId, "cancelled"),
});
function mountHelpRequest(helpMsg: Omit<HelpRequestMessage, "type">) {
return overlays.setAgentHelpRequest(
createHelpRequestData(helpMsg, {
finish: (requestId, outcome, note) => {
void sendHelpFinish(requestId, outcome, note);
},
query: () =>
chrome.runtime.sendMessage({ type: HELP_QUERY }) as Promise<
HelpQueryResponse | undefined
>,
}),
);
}
async function queryActiveHelpWithRetry(): Promise<void> {
@@ -32,6 +32,7 @@ describe("help-bridge", () => {
prompt: "log in",
displayMode: "compact",
selectors: ["#login"],
rects: [{ top: 10, left: 20, width: 100, height: 40 }],
timeoutMs: 1000,
}),
).toBe(true);
+19
View File
@@ -16,6 +16,13 @@ export const HELP_ACK = "bsk-help-ack";
export const HELP_QUERY = "bsk-help-query";
export const HELP_FINISH = "bsk-help-finish";
export interface HelpHighlightRect {
top: number;
left: number;
width: number;
height: number;
}
export interface HelpRequestMessage {
type: typeof HELP_REQUEST;
requestId: string;
@@ -26,6 +33,8 @@ export interface HelpRequestMessage {
displayMode?: "full" | "compact";
/** CSS selectors to scroll to + flash-highlight (may be empty). */
selectors: string[];
/** Top-viewport rectangles for targets that cannot be represented by a root selector. */
rects?: HelpHighlightRect[];
timeoutMs: number;
}
@@ -74,6 +83,16 @@ export function isHelpRequestMessage(msg: unknown): msg is HelpRequestMessage {
(m.displayMode === undefined || m.displayMode === "full" || m.displayMode === "compact") &&
Array.isArray(m.selectors) &&
m.selectors.every((selector) => typeof selector === "string") &&
(m.rects === undefined ||
(Array.isArray(m.rects) &&
m.rects.every(
(rect) =>
typeof rect === "object" &&
rect !== null &&
["top", "left", "width", "height"].every((key) =>
Number.isFinite((rect as Record<string, unknown>)[key]),
),
))) &&
typeof m.timeoutMs === "number"
);
}
@@ -33,4 +33,19 @@ describe("RefStore", () => {
expect(s.resolve("@e11")).toBe(11);
expect(s.size()).toBe(2);
});
it("preserves the child CDP session as part of ref identity", () => {
const s = new RefStore();
s.set("e1", 42, {
tabId: 7,
frameId: "child-frame",
cdpSessionId: "child-session",
});
expect(s.resolveEntry("e1")).toMatchObject({
backendNodeId: 42,
tabId: 7,
frameId: "child-frame",
cdpSessionId: "child-session",
});
});
});
@@ -1,9 +1,10 @@
/**
* Per-session map from `@e<N>` snapshot refs to CDP `backendNodeId`.
* Per-session map from `@e<N>` snapshot refs to a CDP node address.
*
* Each fresh `tool.snapshot` resets the store: M6 will call
* `replace(...)` with the new ref → backendNodeId pairs. Tools like
* `tool.click` consume the store via `resolve("@e1")`.
* `replace(...)` with the new ref → node address pairs. A node address
* includes the owning flat CDP session/frame when the element lives in
* an OOPIF; tools resolve live geometry from that identity at call time.
*
* Refs are session-scoped (§7): looking up a ref in the wrong session
* returns `null`, never silently leaks. Storing values in different
@@ -15,10 +16,19 @@ export type BackendNodeId = number;
export interface RefEntry {
backendNodeId: BackendNodeId;
tabId: number | null;
frameId?: string;
cdpSessionId?: string;
generation: number;
}
type RefInput = BackendNodeId | { backendNodeId: BackendNodeId; tabId: number };
export type RefInput =
| BackendNodeId
| {
backendNodeId: BackendNodeId;
tabId: number;
frameId?: string;
cdpSessionId?: string;
};
export class RefStore {
private readonly map = new Map<string, RefEntry>();
@@ -44,7 +54,7 @@ export class RefStore {
}
/**
* Replace the entire store with a new ref → backendNodeId mapping.
* Replace the entire store with a new ref → CDP node identity mapping.
* Used after every fresh `tool.snapshot`.
*/
replace(entries: Iterable<readonly [string, RefInput]>): void {
@@ -53,10 +63,20 @@ export class RefStore {
for (const [ref, input] of entries) this.map.set(normaliseRef(ref), this.entry(input));
}
set(ref: string, id: BackendNodeId, opts: { tabId?: number } = {}): void {
set(
ref: string,
id: BackendNodeId,
opts: {
tabId?: number;
frameId?: string;
cdpSessionId?: string;
} = {},
): void {
this.map.set(normaliseRef(ref), {
backendNodeId: id,
tabId: opts.tabId ?? null,
...(opts.frameId ? { frameId: opts.frameId } : {}),
...(opts.cdpSessionId ? { cdpSessionId: opts.cdpSessionId } : {}),
generation: this.generation,
});
}
@@ -80,6 +100,8 @@ export class RefStore {
return {
backendNodeId: input.backendNodeId,
tabId: input.tabId,
...(input.frameId ? { frameId: input.frameId } : {}),
...(input.cdpSessionId ? { cdpSessionId: input.cdpSessionId } : {}),
generation: this.generation,
};
}
@@ -612,4 +612,14 @@ describe("isInjectableContentScriptUrl", () => {
).toBe(false);
expect(isInjectableContentScriptUrl("https://chromewebstore.google.com/")).toBe(false);
});
it("rejects the Edge Add-ons store", () => {
expect(
isInjectableContentScriptUrl(
"https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg",
),
).toBe(false);
// Non-store paths on the same host stay injectable.
expect(isInjectableContentScriptUrl("https://microsoftedge.microsoft.com/")).toBe(true);
});
});
@@ -168,6 +168,107 @@ describe("ToolDispatcher", () => {
});
});
it("bypasses and restores the control overlay for an upload trigger click", async () => {
const sendMessage = vi.fn(async () => undefined);
vi.stubGlobal("chrome", {
tabs: {
get: vi.fn(async () => ({ id: 7, windowId: 4242, active: true })),
query: vi.fn(async () => [{ id: 7, windowId: 4242, active: true }]),
sendMessage,
},
});
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
},
});
const ctx = await sessions.start("aa11");
ctx.refStore.set("e1", 123, { tabId: 7 });
const send = vi.fn(async <T>(_tabId: number, method: string, params?: object) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } } as T;
}
if (method === "DOM.getContentQuads") {
return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] } as T;
}
if (method === "DOM.resolveNode") {
return { object: { objectId: "trigger-object" } } as T;
}
if (method === "DOM.describeNode") return { node: { backendNodeId: 456 } } as T;
if (method === "Runtime.callFunctionOn") {
const declaration = (params as { functionDeclaration?: string }).functionDeclaration ?? "";
if (declaration.includes("count: state.inputs.length")) {
return { result: { value: { count: 1, multiple: false } } } as T;
}
if (declaration.includes("inputs[0]")) {
return { result: { objectId: "input-object" } } as T;
}
return { result: { value: true } } as T;
}
if (method === "Runtime.evaluate") {
const expression = (params as { expression?: string }).expression ?? "";
if (expression.includes("overlayDetails")) {
return { result: { value: { hitIndex: 0 } } } as T;
}
if (expression.includes("overlayHostPresent")) {
return {
result: {
value: { overlayHostPresent: true, overlayHostConnected: true },
},
} as T;
}
if (expression.includes("count:")) {
return { result: { value: { count: 1, multiple: false } } } as T;
}
if (expression.includes("?.inputs[0]")) {
return { result: { objectId: "input-object" } } as T;
}
return { result: { value: true } } as T;
}
return {} as T;
});
const cdp = {
send,
detachSession: vi.fn(async () => {}),
ensureNetworkCapture: vi.fn(async () => {}),
networkEntriesSince: vi.fn(() => ({
tab_id: 7,
entries: [],
next_since: 0,
truncated: false,
})),
setDeviceMetricsOverride: vi.fn(async () => {}),
clearDeviceMetricsOverride: vi.fn(async () => {}),
setUserAgentOverride: vi.fn(async () => {}),
setTouchEmulationEnabled: vi.fn(async () => {}),
};
const dispatcher = new ToolDispatcher({ transport, sessions, cdp: cdp as TestDispatcherCdp });
dispatcher.start();
deliver(
makeRequest("tool.upload", {
session_id: "aa11",
ref: "@e1",
files: [{ transfer_id: "tr_1", name: "test.png", staged_path: "/stage/test.png" }],
}),
);
await flushMicrotasks();
await vi.waitFor(() => expect(sent).toHaveLength(1));
expect(sent[0]).toMatchObject({ result: { tab_id: 7, file_names: ["test.png"] } });
expect(sendMessage).toHaveBeenNthCalledWith(1, 7, {
type: "bh-automation-bypass",
enabled: true,
});
expect(sendMessage).toHaveBeenNthCalledWith(2, 7, {
type: "bh-automation-bypass",
enabled: false,
});
});
it("detaches CDP state before stopping a session", async () => {
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
@@ -1,24 +1,54 @@
import { describe, expect, it } from "vitest";
import { nodeBoundingRect, quadBoundingRect } from "../element-geometry";
import { nodeContentRegion } from "../element-geometry";
import type { CdpRunner } from "../shared";
describe("quadBoundingRect", () => {
it("computes axis-aligned bounds from an 8-double quad", () => {
expect(quadBoundingRect([10, 20, 110, 20, 110, 60, 10, 60])).toEqual({
x: 10,
y: 20,
width: 100,
height: 40,
describe("nodeContentRegion", () => {
it("rejects invalid content quads before using the box model fallback", async () => {
const send = async (_tabId: number, method: string) => {
if (method === "DOM.getContentQuads") return { quads: [[0, 0, 0, 0, 0, 0, 0, 0]] };
if (method === "DOM.getBoxModel") {
return { model: { content: [5, 10, 25, 10, 25, 30, 5, 30] } };
}
throw new Error(`unexpected CDP call ${method}`);
};
await expect(nodeContentRegion({ send: send as CdpRunner["send"] }, 7, 555)).resolves.toEqual([
[
{ x: 5, y: 10 },
{ x: 25, y: 10 },
{ x: 25, y: 30 },
{ x: 5, y: 30 },
],
]);
});
it("preserves every visible content fragment", async () => {
const send = async () => ({
quads: [
[10, 20, 60, 20, 60, 40, 10, 40],
[15, 50, 90, 50, 90, 80, 15, 80],
],
});
const cdp: CdpRunner = {
send: send as CdpRunner["send"],
};
await expect(nodeContentRegion(cdp, 7, 555)).resolves.toEqual([
[
{ x: 10, y: 20 },
{ x: 60, y: 20 },
{ x: 60, y: 40 },
{ x: 10, y: 40 },
],
[
{ x: 15, y: 50 },
{ x: 90, y: 50 },
{ x: 90, y: 80 },
{ x: 15, y: 80 },
],
]);
});
it("returns null for degenerate quads", () => {
expect(quadBoundingRect([0, 0, 0, 0, 0, 0, 0, 0])).toBeNull();
expect(quadBoundingRect([1, 2, 3])).toBeNull();
});
});
describe("nodeBoundingRect", () => {
it("falls back to visible descendant bounds for zero-size containers", async () => {
const calls: Array<{ method: string; params?: object }> = [];
const send = async (_tabId: number, method: string, params?: object) => {
@@ -38,12 +68,14 @@ describe("nodeBoundingRect", () => {
};
const cdp = { send: send as CdpRunner["send"] };
await expect(nodeBoundingRect(cdp, 7, 555)).resolves.toEqual({
x: 242,
y: 468,
width: 769,
height: 180,
});
await expect(nodeContentRegion(cdp, 7, 555)).resolves.toEqual([
[
{ x: 242, y: 468 },
{ x: 1011, y: 468 },
{ x: 1011, y: 648 },
{ x: 242, y: 648 },
],
]);
expect(calls.map((c) => c.method)).toEqual([
"DOM.getContentQuads",
"DOM.getBoxModel",
@@ -0,0 +1,559 @@
import { describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import { type DownloadsApi, handleDownload } from "../download";
import { captureBrowserDownload } from "../download-capture";
import { uploadThroughActivatedFileInput } from "../file-input-transaction";
import type { ResolvedActionTarget } from "../interaction";
import type { CdpRunner } from "../shared";
import { handleUpload } from "../upload";
function sessions() {
return new SessionManager({
agentWindow: {
create: vi.fn(async () => 100),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
},
});
}
function tabsApi() {
return {
get: vi.fn(
async (tabId: number) => ({ id: tabId, windowId: 100, active: true }) as chrome.tabs.Tab,
),
query: vi.fn(async () => [{ id: 4, windowId: 100, active: true } as chrome.tabs.Tab]),
};
}
function fakeEvent<T extends (...args: never[]) => unknown>() {
const listeners = new Set<T>();
return {
addListener: (listener: T) => listeners.add(listener),
removeListener: (listener: T) => listeners.delete(listener),
emit: (...args: Parameters<T>) => {
for (const listener of listeners) listener(...args);
},
};
}
function actionTarget(frameId?: string): ResolvedActionTarget {
return {
tab: { tabId: 4, windowId: 100, active: true },
backendNodeId: 123,
cdpTarget: { tabId: 4 },
...(frameId ? { frameId } : {}),
usedRef: "e3",
};
}
function uploadCdp(
options: {
inputCount?: number;
multiple?: boolean;
chooser?: { frameId?: string; backendNodeId?: number; mode?: string };
pendingResolve?: boolean;
} = {},
) {
const calls: Array<{ method: string; params?: object }> = [];
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
calls.push({ method, params });
if (method === "Page.setInterceptFileChooserDialog") return {};
if (method === "Page.getLayoutMetrics")
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
if (method === "DOM.getContentQuads") return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] };
if (method === "DOM.resolveNode") {
if (options.pendingResolve) return new Promise<never>(() => {});
return { object: { objectId: "trigger-object" } };
}
if (method === "Runtime.callFunctionOn") {
const declaration = (params as { functionDeclaration?: string }).functionDeclaration ?? "";
if (declaration.includes("Object.defineProperty")) return { result: { value: true } };
if (declaration.includes("count: state.inputs.length")) {
return {
result: {
value: {
count: options.inputCount ?? 1,
multiple: options.multiple ?? true,
},
},
};
}
if (declaration.includes("inputs[0]")) {
return { result: { objectId: "input-object" } };
}
return { result: { value: true } };
}
if (method === "DOM.describeNode") return { node: { backendNodeId: 456 } };
if (
method === "Input.dispatchMouseEvent" &&
(params as { type?: string }).type === "mousePressed" &&
options.chooser
) {
cdpEvent?.({ tabId: 4 }, "Page.fileChooserOpened", options.chooser);
}
return {};
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
return {
calls,
emitChooser: (event: { frameId?: string; backendNodeId?: number; mode?: string }) =>
cdpEvent?.({ tabId: 4 }, "Page.fileChooserOpened", event),
cdp,
};
}
describe("file transfer tools", () => {
it("captures the file input activated by the requested click and injects only staged paths", async () => {
const manager = sessions();
const ctx = await manager.start("s1");
ctx.refStore.set("e3", 123, { tabId: 4 });
const { cdp, calls } = uploadCdp();
const result = await handleUpload(
manager,
{
session_id: "s1",
ref: "@e3",
files: [
{ transfer_id: "tr_1", name: "one.png", staged_path: "/private/stage/one" },
{ transfer_id: "tr_2", name: "two.png", staged_path: "/private/stage/two" },
],
},
{ cdp, tabsApi: tabsApi() },
);
expect(result).toMatchObject({ tab_id: 4, file_names: ["one.png", "two.png"] });
expect(calls[0]).toEqual({
method: "Page.setInterceptFileChooserDialog",
params: { enabled: true },
});
expect(calls).toContainEqual({
method: "Page.setInterceptFileChooserDialog",
params: { enabled: false, cancel: true },
});
expect(calls).toContainEqual({
method: "DOM.setFileInputFiles",
params: { files: ["/private/stage/one", "/private/stage/two"], backendNodeId: 456 },
});
});
it("fails immediately when the trigger does not activate a file input", async () => {
const { cdp } = uploadCdp({ inputCount: 0 });
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger: async () => ({ tab_id: 4, x: 10, y: 10 }),
});
expect(result).toMatchObject({
code: "unsupported",
data: { reason: "file_input_not_activated", phase: "resolve_input" },
});
});
it("fails before clicking when chooser interception is unavailable", async () => {
const trigger = vi.fn(async () => ({ tab_id: 4, x: 10, y: 10 }));
const cdp: CdpRunner = {
send: vi.fn(async (_tabId: number, method: string) => {
if (method === "Page.setInterceptFileChooserDialog") {
throw new Error("method unavailable");
}
return {};
}) as CdpRunner["send"],
};
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger,
});
expect(result).toMatchObject({
code: "cdp_failed",
data: { effect_state: "none", phase: "arm_interception" },
});
expect(trigger).not.toHaveBeenCalled();
});
it("uses an exact chooser event as an independent input-location signal", async () => {
const { cdp, calls, emitChooser } = uploadCdp({ inputCount: 0 });
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget("f1"),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger: async () => {
emitChooser({ frameId: "f1", backendNodeId: 789, mode: "selectSingle" });
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({ multiple: false });
expect(calls).toContainEqual({
method: "DOM.setFileInputFiles",
params: { files: ["/private/stage/one"], backendNodeId: 789 },
});
});
it("reports a File System Access picker without waiting for a timeout", async () => {
const { cdp, emitChooser } = uploadCdp({
inputCount: 0,
});
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget("f1"),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger: async () => {
emitChooser({ frameId: "f1", mode: "selectSingle" });
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({
code: "unsupported",
message: "upload trigger invoked a non-input file picker",
data: { reason: "file_input_not_activated", phase: "resolve_input" },
});
});
it("bounds a stuck file-input probe", async () => {
const { cdp } = uploadCdp({ pendingResolve: true });
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 5,
trigger: vi.fn(),
});
expect(result).toMatchObject({
code: "timeout",
data: { reason: "file_input_probe_failed", phase: "arm_input_probe" },
});
});
it("marks a timed-out file assignment unknown and detaches browser state", async () => {
const fixture = uploadCdp();
const originalSend = fixture.cdp.send;
const detach = vi.fn(async () => {});
fixture.cdp.detach = detach;
fixture.cdp.send = vi.fn((tabId: number, method: string, params?: object) => {
if (method === "DOM.setFileInputFiles") return new Promise<never>(() => {});
return originalSend(tabId, method, params);
}) as CdpRunner["send"];
const result = await uploadThroughActivatedFileInput({
cdp: fixture.cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 10,
trigger: async () => ({ tab_id: 4, x: 10, y: 10 }),
});
expect(result).toMatchObject({
code: "timeout",
data: { effect_state: "unknown", phase: "set_files" },
});
expect(detach).toHaveBeenCalledWith(4);
});
it("routes one exact-target download through a browser-relative capability", async () => {
const manager = sessions();
const ctx = await manager.start("s1");
ctx.refStore.set("e3", 123, { tabId: 4 });
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 9,
url: "https://example.test/result.zip",
finalUrl: "https://example.test/result.zip",
filename: "result.zip",
state: "in_progress",
fileSize: -1,
totalBytes: 12,
mime: "application/zip",
danger: "safe",
} as chrome.downloads.DownloadItem;
const completed = {
...initial,
filename: "/profile/Downloads/BrowserSkill/tr_1/result.zip",
state: "complete",
fileSize: 12,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => [completed]),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
let suggested: chrome.downloads.DownloadFilenameSuggestion | undefined;
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
if (method === "Page.getLayoutMetrics")
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
if (method === "DOM.getContentQuads") return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] };
if (
method === "Input.dispatchMouseEvent" &&
(params as { type?: string }).type === "mousePressed"
) {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: "result.zip",
});
await new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, (value) => {
suggested = value;
resolve();
});
});
onCreated.emit(completed);
}
return {};
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const result = await handleDownload(
manager,
{ session_id: "s1", ref: "@e3", browser_relative_dir: "BrowserSkill/tr_1" },
{ cdp, tabsApi: tabsApi(), downloads },
);
expect(suggested).toEqual({
filename: "BrowserSkill/tr_1/result.zip",
conflictAction: "overwrite",
});
expect(result).toMatchObject({
tab_id: 4,
suggested_filename: "result.zip",
byte_size: 12,
browser_path: completed.filename,
});
});
it("does not claim a download without an intent from the exact target", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => []),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as unknown as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const unrelated = {
id: 17,
url: "https://example.test/unrelated.zip",
finalUrl: "https://example.test/unrelated.zip",
filename: "unrelated.zip",
state: "in_progress",
} as chrome.downloads.DownloadItem;
let defaultSuggestionCalled = false;
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4, sessionId: "expected-child" },
downloads,
browserRelativeDir: "BrowserSkill/tr_1",
timeoutMs: 5,
trigger: async () => {
cdpEvent?.({ tabId: 4, sessionId: "other-child" }, "Page.downloadWillBegin", {
url: unrelated.url,
suggestedFilename: unrelated.filename,
});
onDeterminingFilename.emit(unrelated, (suggestion) => {
defaultSuggestionCalled = suggestion === undefined;
});
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(defaultSuggestionCalled).toBe(true);
expect(downloads.cancel).not.toHaveBeenCalled();
expect(result).toMatchObject({
code: "cdp_failed",
data: { reason: "download_capture_failed" },
});
});
it("correlates a filename candidate that arrives before the CDP intent", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 21,
url: "https://example.test/candidate-first.bin",
finalUrl: "https://example.test/candidate-first.bin",
filename: "candidate-first.bin",
state: "in_progress",
fileSize: -1,
totalBytes: 4,
bytesReceived: 0,
} as chrome.downloads.DownloadItem;
const complete = {
...initial,
state: "complete",
fileSize: 4,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => [complete]),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
let suggestion: chrome.downloads.DownloadFilenameSuggestion | undefined;
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_21",
timeoutMs: 1_000,
trigger: async () => {
const suggested = new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, (value) => {
suggestion = value;
resolve();
});
});
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: initial.filename,
});
await suggested;
onCreated.emit(complete);
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(suggestion).toEqual({
filename: "BrowserSkill/tr_21/candidate-first.bin",
conflictAction: "overwrite",
});
expect(result).toMatchObject({ item: { id: 21, state: "complete" } });
});
it("rejects ambiguous attribution without cancelling either unclaimed download", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => []),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const defaults: number[] = [];
const candidate = (id: number) =>
({
id,
url: "https://example.test/same.bin",
finalUrl: "https://example.test/same.bin",
filename: "same.bin",
state: "in_progress",
}) as chrome.downloads.DownloadItem;
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_ambiguous",
timeoutMs: 100,
trigger: async () => {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: "https://example.test/same.bin",
suggestedFilename: "same.bin",
});
for (const id of [31, 32]) {
onDeterminingFilename.emit(candidate(id), (value) => {
if (value === undefined) defaults.push(id);
});
}
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(defaults.sort()).toEqual([31, 32]);
expect(downloads.cancel).not.toHaveBeenCalled();
expect(result).toMatchObject({
code: "cdp_failed",
data: { effect_state: "unknown", phase: "attribution" },
});
});
});
@@ -0,0 +1,286 @@
import { describe, expect, it, vi } from "vitest";
import { resolveFrameProjection, resolveNodeGeometry } from "../frame-geometry";
import {
clipPolygon,
polygonArea,
polygonBounds,
projectAndClipRegion,
projectUnitPoint,
type Quad,
rectPolygon,
regionBounds,
} from "../geometry";
import type { CdpRunner } from "../shared";
describe("frame geometry projection", () => {
it("keeps region bounds separate from polygon area", () => {
const region = [
rectPolygon({ x: 0, y: 0, w: 10, h: 10 }),
rectPolygon({ x: 20, y: 5, w: 5, h: 5 }),
];
expect(regionBounds(region)).toEqual({ x: 0, y: 0, width: 25, height: 10 });
expect(polygonArea(region[0])).toBe(100);
});
it("maps through the iframe content box instead of its border box", () => {
const contentQuad: Quad = [
{ x: 204, y: 306 },
{ x: 604, y: 306 },
{ x: 604, y: 506 },
{ x: 204, y: 506 },
];
const projected = projectAndClipRegion(
[rectPolygon({ x: 10, y: 20, w: 100, h: 40 })],
[{ sourceViewport: { width: 200, height: 100 }, destinationQuad: contentQuad }],
{ width: 1000, height: 800 },
);
expect(polygonBounds(projected.flat())).toEqual({ x: 224, y: 346, width: 200, height: 80 });
});
it("uses a projective mapping for perspective-transformed iframe quads", () => {
const quad: Quad = [
{ x: 0, y: 0 },
{ x: 200, y: 20 },
{ x: 180, y: 140 },
{ x: 20, y: 100 },
];
expect(projectUnitPoint({ x: 0, y: 0 }, quad)).toEqual(quad[0]);
expect(projectUnitPoint({ x: 1, y: 0 }, quad)).toEqual(quad[1]);
expect(projectUnitPoint({ x: 1, y: 1 }, quad)).toEqual(quad[2]);
expect(projectUnitPoint({ x: 0, y: 1 }, quad)).toEqual(quad[3]);
});
it("clips against the iframe polygon rather than its bounding box", () => {
const frameQuad: Quad = [
{ x: 50, y: 0 },
{ x: 100, y: 50 },
{ x: 50, y: 100 },
{ x: 0, y: 50 },
];
expect(clipPolygon(rectPolygon({ x: 0, y: 0, w: 10, h: 10 }), frameQuad)).toEqual([]);
});
it("does not reapply same-target frame transforms around an OOPIF", async () => {
const send = vi.fn(async (_tabId, method, params) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 800, clientHeight: 600 } };
}
if (method === "DOM.getBoxModel") {
const backendNodeId = (params as { backendNodeId?: number }).backendNodeId;
return backendNodeId === 20
? { model: { content: [110, 70, 310, 70, 310, 170, 110, 170] } }
: { model: { content: [100, 50, 400, 50, 400, 250, 100, 250] } };
}
throw new Error(`unexpected root command ${method}`);
});
const cdp: CdpRunner = {
send: send as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
}
throw new Error(`unexpected child command ${method}`);
}) as CdpRunner["sendToTarget"],
};
const projection = await resolveFrameProjection(
cdp,
{
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "same-process-parent",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4 },
},
{
frameId: "oopif",
parentFrameId: "same-process-parent",
ownerBackendNodeId: 20,
target: { tabId: 4, sessionId: "oopif-session" },
},
],
},
"oopif",
);
expect(projection?.edges).toEqual([
{
sourceViewport: { width: 200, height: 100 },
destinationQuad: [
{ x: 110, y: 70 },
{ x: 310, y: 70 },
{ x: 310, y: 170 },
{ x: 110, y: 170 },
],
destinationClips: [
[
{ x: 100, y: 50 },
{ x: 400, y: 50 },
{ x: 400, y: 250 },
{ x: 100, y: 250 },
],
],
},
]);
});
it("clips live OOPIF geometry at every frame boundary", async () => {
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 800, clientHeight: 600 } };
}
if (method === "DOM.getBoxModel") {
return { model: { content: [100, 100, 300, 100, 300, 200, 100, 200] } };
}
throw new Error(`unexpected root command ${method}`);
}) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
}
if (method === "DOM.getContentQuads") {
return { quads: [[-50, -20, 250, -20, 250, 120, -50, 120]] };
}
throw new Error(`unexpected child command ${method}`);
}) as CdpRunner["sendToTarget"],
getFrameGraph: vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4, sessionId: "child-session" },
},
],
})),
};
const geometry = await resolveNodeGeometry(cdp, 4, {
target: { tabId: 4, sessionId: "child-session" },
frameId: "child",
backendNodeId: 101,
});
expect(geometry).toMatchObject({
topBounds: { x: 100, y: 100, width: 200, height: 100 },
actionPoint: { x: 200, y: 150 },
});
});
it("reuses one frame graph while scrolling and projecting live geometry", async () => {
const getFrameGraph = vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4, sessionId: "child-session" },
},
],
}));
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method) => {
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 800, clientHeight: 600 } };
}
if (method === "DOM.getBoxModel") {
return { model: { content: [100, 100, 300, 100, 300, 200, 100, 200] } };
}
throw new Error(`unexpected root command ${method}`);
}) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => {
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
}
if (method === "DOM.getContentQuads") {
return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] };
}
throw new Error(`unexpected child command ${method}`);
}) as CdpRunner["sendToTarget"],
getFrameGraph,
};
await expect(
resolveNodeGeometry(
cdp,
4,
{
target: { tabId: 4, sessionId: "child-session" },
frameId: "child",
backendNodeId: 101,
},
{ scrollIntoView: true },
),
).resolves.not.toMatchObject({ code: expect.any(String) });
expect(getFrameGraph).toHaveBeenCalledOnce();
});
it("clips live geometry to same-target iframe ancestors", async () => {
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 800, clientHeight: 600 } };
}
if (method === "DOM.getBoxModel") {
return { model: { content: [100, 100, 300, 100, 300, 200, 100, 200] } };
}
if (method === "DOM.getContentQuads") {
return { quads: [[50, 50, 350, 50, 350, 250, 50, 250]] };
}
throw new Error(`unexpected ${method}`);
}) as CdpRunner["send"],
getFrameGraph: vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4 },
},
],
})),
};
const geometry = await resolveNodeGeometry(cdp, 4, {
target: { tabId: 4 },
frameId: "child",
backendNodeId: 101,
});
expect(geometry).toMatchObject({
topBounds: { x: 100, y: 100, width: 200, height: 100 },
actionPoint: { x: 200, y: 150 },
});
});
it("fails closed when an OOPIF frame graph is unavailable", async () => {
const cdp: CdpRunner = {
send: vi.fn(),
sendToTarget: vi.fn(async (_target, method) => {
if (method === "DOM.getContentQuads") {
return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] };
}
throw new Error(`unexpected ${method}`);
}) as CdpRunner["sendToTarget"],
};
await expect(
resolveNodeGeometry(cdp, 4, {
target: { tabId: 4, sessionId: "child-session" },
frameId: "child",
backendNodeId: 101,
}),
).resolves.toMatchObject({ code: "cdp_failed" });
});
});
@@ -1,5 +1,6 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import { RefStore } from "@/session-manager/ref-store";
import type { RequestHelpParams } from "@/transport/types";
import { handleRequestHelp, type RequestHelpDeps, resetHelpLifecycleForTests } from "../human-loop";
@@ -50,11 +51,10 @@ function installHelpLifecycleChrome() {
}
function fakeManager(sessionId: string, agentWindowId: number, tabId: number) {
const refStore = new RefStore();
const mgr = {
get: (id: string) =>
id === sessionId
? { sessionId, agentWindowId, refStore: { resolve: () => null }, borrowedTabs: new Map() }
: null,
id === sessionId ? { sessionId, agentWindowId, refStore, borrowedTabs: new Map() } : null,
findByWindowId: (wid: number) => (wid === agentWindowId ? { sessionId } : null),
} as unknown as SessionManager;
return mgr;
@@ -339,26 +339,23 @@ describe("handleRequestHelp", () => {
ac.abort();
await expect(pending).resolves.toMatchObject({ code: "cancelled" });
expect(cdpSend).toHaveBeenCalledWith(
5,
"DOM.querySelectorAll",
expect.objectContaining({ selector: "[data-bsk-help]" }),
);
expect(cdpSend).toHaveBeenCalledWith(
6,
"DOM.querySelectorAll",
expect.objectContaining({ selector: "[data-bsk-help]" }),
expect(cdpSend).not.toHaveBeenCalledWith(
expect.anything(),
"DOM.removeAttribute",
expect.anything(),
);
});
it("tags ref targets via CDP and reports them matched", async () => {
it("resolves ref targets to explicit viewport rectangles without mutating the page", async () => {
const refStore = new RefStore();
refStore.set("e1", 42, { tabId: 5 });
const mgr = {
get: (id: string) =>
id === "abcd"
? {
sessionId: "abcd",
agentWindowId: 99,
refStore: { resolve: () => 42 },
refStore,
borrowedTabs: new Map(),
}
: null,
@@ -366,7 +363,17 @@ describe("handleRequestHelp", () => {
} as unknown as SessionManager;
const deps = baseDeps({
cdp: {
send: vi.fn(async () => ({ object: { objectId: "obj-1" } })),
send: vi.fn(async (_tabId, method) => {
if (method === "DOM.getDocument") return { root: { nodeId: 1 } };
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "DOM.getContentQuads") {
return { quads: [[10, 20, 110, 20, 110, 60, 10, 60]] };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
}
throw new Error(`unexpected ${method}`);
}),
} as unknown as RequestHelpDeps["cdp"],
});
const res = await handleRequestHelp(
@@ -375,14 +382,97 @@ describe("handleRequestHelp", () => {
deps,
);
const sentMsg = (deps.sendToTab as ReturnType<typeof vi.fn>).mock.calls[0][1];
expect(sentMsg.selectors).toContain('[data-bsk-help="0"]');
expect(sentMsg.selectors).toEqual([]);
expect(sentMsg.rects).toEqual([{ top: 20, left: 10, width: 100, height: 40 }]);
expect(res).toMatchObject({
outcome: "continued",
tab_id: 5,
resolved_targets: [{ matched: true, ref: "@e1" }],
resolved_targets: [
{
matched: true,
ref: "@e1",
},
],
});
});
it("projects an OOPIF ref through its live content quad for the help overlay", async () => {
const mgr = {
get: (id: string) =>
id === "abcd"
? {
sessionId: "abcd",
agentWindowId: 99,
refStore: {
resolveEntry: () => ({
backendNodeId: 42,
tabId: 5,
frameId: "child",
cdpSessionId: "child-session",
generation: 1,
}),
},
borrowedTabs: new Map(),
}
: null,
findByWindowId: (wid: number) => (wid === 99 ? { sessionId: "abcd" } : null),
} as unknown as SessionManager;
const send = vi.fn(async (_tabId, method) => {
if (method === "DOM.getDocument") return { root: { nodeId: 1 } };
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "DOM.getBoxModel") {
return { model: { content: [204, 306, 604, 306, 604, 506, 204, 506] } };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
}
throw new Error(`unexpected root ${method}`);
});
const sendToTarget = vi.fn(async (_target, method) => {
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "DOM.getContentQuads") {
return { quads: [[10, 20, 110, 20, 110, 60, 10, 60]] };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
}
throw new Error(`unexpected child ${method}`);
});
const deps = baseDeps({
cdp: {
send,
sendToTarget,
getFrameGraph: vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 5 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 99,
target: { tabId: 5, sessionId: "child-session" },
},
],
})),
} as unknown as NonNullable<RequestHelpDeps["cdp"]>,
});
const res = await handleRequestHelp(
mgr,
baseParams({ tab_id: 5, targets: [{ ref: "@e1" }] }),
deps,
);
const sentMsg = (deps.sendToTab as ReturnType<typeof vi.fn>).mock.calls[0][1];
expect(sentMsg.rects).toEqual([{ top: 346, left: 224, width: 200, height: 80 }]);
expect(res).toMatchObject({ resolved_targets: [{ matched: true, ref: "@e1" }] });
expect(sendToTarget).toHaveBeenCalledWith(
{ tabId: 5, sessionId: "child-session" },
"DOM.getContentQuads",
{ backendNodeId: 42 },
);
});
it("reports ref target unmatched when ref does not resolve", async () => {
const deps = baseDeps();
const res = await handleRequestHelp(
@@ -398,16 +488,15 @@ describe("handleRequestHelp", () => {
});
it("reports ref target unmatched when ref is for another tab", async () => {
const refStore = new RefStore();
refStore.set("e1", 42, { tabId: 4 });
const mgr = {
get: (id: string) =>
id === "abcd"
? {
sessionId: "abcd",
agentWindowId: 99,
refStore: {
resolve: (ref: string, opts: { tabId?: number }) =>
ref === "e1" && opts.tabId === 4 ? 42 : null,
},
refStore,
borrowedTabs: new Map(),
}
: null,
@@ -30,6 +30,9 @@ function makeFakeCdp(handlers: Record<string, (params: unknown) => unknown>) {
const sendImpl = async (tabId: number, method: string, params?: object) => {
sent.push({ tabId, method, params });
const h = handlers[method];
if (!h && method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
}
if (!h) throw new Error(`unexpected CDP call ${method}`);
return h(params);
};
@@ -164,6 +167,63 @@ describe("handleClick", () => {
});
});
it("resolves frame refs in their CDP session and dispatches input in top coordinates", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
ctx.refStore.set("e3", 1234, {
tabId: 4,
frameId: "child-frame",
cdpSessionId: "child-session",
});
const fake = makeFakeCdp({
"Input.dispatchMouseEvent": () => ({}),
"DOM.scrollIntoViewIfNeeded": () => ({}),
"DOM.getBoxModel": () => ({
model: { content: [204, 306, 604, 306, 604, 506, 204, 506] },
}),
});
fake.cdp.getFrameGraph = vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child-frame",
parentFrameId: "main",
ownerBackendNodeId: 99,
target: { tabId: 4, sessionId: "child-session" },
},
],
}));
const targetCalls: Array<{ sessionId?: string; method: string }> = [];
fake.cdp.sendToTarget = vi.fn(async (target, method) => {
targetCalls.push({ sessionId: target.sessionId, method });
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "DOM.getContentQuads") {
return { quads: [[10, 20, 110, 20, 110, 60, 10, 60]] };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
}
throw new Error(`unexpected child CDP call ${method}`);
}) as CdpRunner["sendToTarget"];
const res = await handleClick(
sm,
{ session_id: "aa11", ref: "@e3" },
{ cdp: fake.cdp, tabsApi: fake.tabsApi },
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
// The iframe content box starts at (204, 306) after its border and is scaled 2x.
expect(res).toMatchObject({ x: 324, y: 386 });
expect(targetCalls).toEqual([
{ sessionId: "child-session", method: "DOM.scrollIntoViewIfNeeded" },
{ sessionId: "child-session", method: "DOM.getContentQuads" },
{ sessionId: "child-session", method: "Page.getLayoutMetrics" },
]);
expect(fake.sent.filter((call) => call.method === "Input.dispatchMouseEvent")).toHaveLength(3);
});
it("enables overlay bypass before mouse events when overlay blocks the click point", async () => {
const order: string[] = [];
const bypassOverlay = vi.fn(async (_tabId: number, enabled: boolean) => {
@@ -5,6 +5,7 @@ import { OVERLAY_HOST_MARKER_ATTR, OVERLAY_HOST_NAME } from "@/lib/overlay-bridg
import { SessionManager } from "@/session-manager/manager";
import type { CdpRunner } from "@/tools/shared";
import {
buildFrameVomScene,
buildVomScene,
type CdpAxNode,
handleGetHtml,
@@ -62,6 +63,9 @@ function makeFakeCdp(handlers: Record<string, (params?: object) => unknown>) {
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
sent.push({ method, params });
const handler = handlers[method];
if (!handler && method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
}
if (!handler) throw new Error(`unexpected CDP call ${method}`);
return handler(params);
});
@@ -508,6 +512,292 @@ describe("handleScreenshot overlay suppression", () => {
// ---------------------------------------------------------------------------
describe("buildVomScene", () => {
it("merges multiple sibling and nested frames without mixing their targets", () => {
const node = (
backendNodeId: number,
parentBackendNodeId: number | null,
frameId: string,
tag: string,
attrs: Record<string, string> = {},
): CapturedNode => ({
backendNodeId,
parentBackendNodeId,
frameId,
tag,
attrs,
rect: { x: 0, y: 0, w: 100, h: 30 },
localRect: { x: 0, y: 0, w: 100, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
});
const mainNodes = [
node(1, null, "main", "body"),
node(10, 1, "main", "iframe", { title: "First" }),
node(20, 1, "main", "iframe", { title: "Second" }),
];
const firstNodes = [
node(100, null, "first", "body"),
node(101, 100, "first", "button", { "aria-label": "First action" }),
];
const secondNodes = [
node(200, null, "second", "body"),
node(201, 200, "second", "h2"),
node(210, 200, "second", "iframe", { title: "Nested" }),
];
const nestedNodes = [
node(300, null, "nested", "body"),
node(301, 300, "nested", "input", { placeholder: "Nested field" }),
];
const captured: CapturedViewModel = {
viewport: { width: 1200, height: 800 },
nodes: mainNodes,
iframeNodes: new Map([
[10, firstNodes],
[20, secondNodes],
[210, nestedNodes],
]),
frameNodes: new Map([
["main", mainNodes],
["first", firstNodes],
["second", secondNodes],
["nested", nestedNodes],
]),
frameOwnerBackendNodeIds: new Map([
["first", 10],
["second", 20],
["nested", 210],
]),
rootFrameId: "main",
excludedBackendNodeIds: new Set(),
};
const axNode = (
nodeId: string,
backendDOMNodeId: number,
role: string,
parentId?: string,
name?: string,
): CdpAxNode => ({
nodeId,
backendDOMNodeId,
role: { type: "role", value: role },
...(parentId ? { parentId } : {}),
...(name ? { name: { type: "computedString", value: name } } : {}),
});
const result = buildFrameVomScene(
[
{
frameId: "main",
contextScopeId: "main",
target: { tabId: 7 },
domNodes: mainNodes,
axNodes: [
axNode("root", 1, "RootWebArea"),
axNode("frame-1", 10, "Iframe", "root", "First"),
axNode("frame-2", 20, "Iframe", "root", "Second"),
],
},
{
frameId: "first",
contextScopeId: "first",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 7, sessionId: "session-first" },
domNodes: firstNodes,
axNodes: [
axNode("root", 100, "RootWebArea"),
axNode("button", 101, "button", "root", "First action"),
],
},
{
frameId: "second",
contextScopeId: "second",
parentFrameId: "main",
ownerBackendNodeId: 20,
target: { tabId: 7, sessionId: "session-second" },
domNodes: secondNodes,
axNodes: [
axNode("root", 200, "RootWebArea"),
axNode("heading", 201, "heading", "root", "Second heading"),
axNode("nested-frame", 210, "Iframe", "root", "Nested"),
],
},
{
frameId: "nested",
contextScopeId: "nested",
parentFrameId: "second",
ownerBackendNodeId: 210,
target: { tabId: 7, sessionId: "session-nested" },
domNodes: nestedNodes,
axNodes: [
axNode("root", 300, "RootWebArea"),
axNode("input", 301, "textbox", "root", "Nested field"),
],
},
],
captured,
);
const rendered = renderVom(result);
expect(rendered.text).toContain('Iframe "First"');
expect(rendered.text).toContain('@e1 button "First action"');
expect(rendered.text).toContain('heading "Second heading"');
expect(rendered.text).toContain('Iframe "Nested"');
expect(rendered.text).toContain('textbox "Nested field"');
expect(rendered.refs.find((ref) => ref.backendNodeId === 101)?.frameId).toBe("first");
expect(rendered.refs.find((ref) => ref.backendNodeId === 301)?.frameId).toBe("nested");
const firstAction = result.nodes.find((item) => item.backendNodeId === 101);
const nestedInput = result.nodes.find((item) => item.backendNodeId === 301);
expect(firstAction).toMatchObject({ frameId: "first", contextScopeId: "first" });
expect(nestedInput).toMatchObject({ frameId: "nested", contextScopeId: "nested" });
expect(nestedInput?.parentId).toBe(result.nodes.find((item) => item.backendNodeId === 210)?.id);
});
it("does not place a child document at the page root when its frame boundary is unresolved", () => {
const childNode: CapturedNode = {
backendNodeId: 101,
parentBackendNodeId: null,
frameId: "child",
tag: "button",
attrs: {},
rect: { x: 0, y: 0, w: 100, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
};
const captured: CapturedViewModel = {
viewport: { width: 800, height: 600 },
nodes: [],
iframeNodes: new Map(),
frameNodes: new Map([["child", [childNode]]]),
rootFrameId: "main",
excludedBackendNodeIds: new Set(),
};
const result = buildFrameVomScene(
[
{
frameId: "main",
contextScopeId: "main",
target: { tabId: 7 },
domNodes: [],
axNodes: [],
},
{
frameId: "child",
contextScopeId: "child",
parentFrameId: "main",
target: { tabId: 7 },
domNodes: [childNode],
axNodes: [
{
nodeId: "button",
frameId: "child",
backendDOMNodeId: 101,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Child action" },
},
],
},
],
captured,
);
expect(result.nodes).toEqual([]);
});
it("keeps AX-only frame content attached to its iframe boundary", () => {
const mainNodes: CapturedNode[] = [
{
backendNodeId: 1,
parentBackendNodeId: null,
frameId: "main",
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 800, h: 600 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 10,
parentBackendNodeId: 1,
frameId: "main",
tag: "iframe",
attrs: { title: "Remote" },
rect: { x: 100, y: 100, w: 400, h: 300 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
},
];
const captured: CapturedViewModel = {
viewport: { width: 800, height: 600 },
nodes: mainNodes,
iframeNodes: new Map(),
frameNodes: new Map([
["main", mainNodes],
["child", []],
]),
rootFrameId: "main",
excludedBackendNodeIds: new Set(),
};
const result = buildFrameVomScene(
[
{
frameId: "main",
contextScopeId: "main",
target: { tabId: 7 },
domNodes: mainNodes,
axNodes: [
{
nodeId: "main-root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
},
{
nodeId: "frame-owner",
parentId: "main-root",
backendDOMNodeId: 10,
role: { type: "role", value: "Iframe" },
name: { type: "computedString", value: "Remote" },
},
],
},
{
frameId: "child",
contextScopeId: "child",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 7, sessionId: "child-session" },
domNodes: [],
axNodes: [
{
nodeId: "child-root",
backendDOMNodeId: 100,
role: { type: "role", value: "RootWebArea" },
},
{
nodeId: "child-button",
parentId: "child-root",
backendDOMNodeId: 101,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "AX fallback action" },
},
],
},
],
captured,
);
const childButton = result.nodes.find((node) => node.backendNodeId === 101);
expect(childButton).toMatchObject({ frameId: "child", contextScopeId: "child" });
expect(childButton?.parentId).toBe(result.nodes.find((node) => node.backendNodeId === 10)?.id);
expect(renderVom(result).text).toContain('@e1 button "AX fallback action"');
});
it("joins AX semantics with captured geometry by backendDOMNodeId", () => {
const axNodes: CdpAxNode[] = [
{
@@ -816,6 +1106,7 @@ describe("buildVomScene", () => {
parentId: "2",
role: { type: "role", value: "textbox" },
name: { type: "x", value: "输入密码" },
value: { value: "iframe-secret" },
backendDOMNodeId: 202,
properties: [{ name: "inputType", value: { value: "password" } }],
},
@@ -870,6 +1161,10 @@ describe("buildVomScene", () => {
expect(passwordNodes).toHaveLength(1);
expect(passwordNodes[0]).toEqual(expect.objectContaining({ id: 202, sensitive: true }));
expect(passwordNodes[0].value).toBeUndefined();
const rendered = renderVom(buildVomScene(axNodes, captured)).text;
expect(rendered).toContain('textbox "输入密码" [filled] ="•••"');
expect(rendered).not.toContain("iframe-secret");
});
it("keeps unnamed iframe controls but skips unnamed iframe links", () => {
@@ -3123,6 +3418,32 @@ describe("handleGetHtml", () => {
expect(deps.send).toHaveBeenCalledTimes(1);
});
it("reads a frame ref through its child CDP session", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
ctx.refStore.set("e7", 4242, {
tabId: 4,
frameId: "child-frame",
cdpSessionId: "child-session",
});
const deps = makeDeps({});
const sendToTarget = vi.fn(async (target, method, params) => {
expect(target).toEqual({ tabId: 4, sessionId: "child-session" });
expect(method).toBe("DOM.getOuterHTML");
expect(params).toEqual({ backendNodeId: 4242 });
return { outerHTML: "<button>frame</button>" };
});
(deps.cdp as CdpRunner).sendToTarget = sendToTarget as unknown as NonNullable<
CdpRunner["sendToTarget"]
>;
const res = await handleGetHtml(sm, { session_id: "aa11", ref: "@e7" }, deps);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.html).toBe("<button>frame</button>");
expect(deps.send).not.toHaveBeenCalled();
});
it("returns not_found when a ref belongs to another tab", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
@@ -46,6 +46,25 @@ describe("lookupSnapshotRef", () => {
});
describe("resolveSnapshotRef", () => {
it("preserves the frame and child session needed to route iframe refs", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
ctx.refStore.set("e3", 1234, {
tabId: 4,
frameId: "child-frame",
cdpSessionId: "child-session",
});
const expected = {
backendNodeId: 1234,
refKey: "e3",
frameId: "child-frame",
cdpSessionId: "child-session",
};
expect(lookupSnapshotRef(ctx, "@e3", 4)).toEqual(expected);
expect(resolveSnapshotRef(ctx, "@e3", 4)).toEqual(expected);
});
it("returns not_found for unknown ref", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
@@ -250,9 +250,10 @@ const DEFAULT_NOTIFICATION_COPY: BorrowNotificationCopy = {
// URL injection-eligibility test
// ---------------------------------------------------------------------------
const CHROME_WEB_STORE_RES = [
const EXTENSION_STORE_RES = [
/^https:\/\/chrome\.google\.com\/webstore/i,
/^https:\/\/chromewebstore\.google\.com/i,
/^https:\/\/microsoftedge\.microsoft\.com\/addons/i,
];
/**
@@ -264,14 +265,15 @@ const CHROME_WEB_STORE_RES = [
* - `ftp://` no longer hosts content scripts reliably in modern Chrome.
* - `about:` / `chrome:` / `chrome-extension:` / `edge:` / `devtools:` /
* `view-source:` / `data:` / `blob:` are blocked by the platform.
* - The Chrome Web Store (both legacy and new domains) is also blocked.
* - Extension storefronts are blocked by their own browser: the Chrome Web
* Store (legacy and new domains) under Chrome, and Edge Add-ons under Edge.
*
* Mirrors the implicit scheme list documented in
* https://developer.chrome.com/docs/extensions/develop/concepts/match-patterns.
*/
export function isInjectableContentScriptUrl(url: string | undefined): boolean {
if (!url) return false;
for (const re of CHROME_WEB_STORE_RES) {
for (const re of EXTENSION_STORE_RES) {
if (re.test(url)) return false;
}
return /^https?:\/\//i.test(url);
+40
View File
@@ -4,6 +4,7 @@ import type { Transport } from "@/transport/transport";
import type {
ClickParams,
ConsoleParams,
DownloadParams,
EmulateParams,
EvaluateParams,
FillParams,
@@ -28,10 +29,12 @@ import type {
ScreenshotParams,
SelectParams,
SnapshotParams,
UploadParams,
WaitForNavigationParams,
} from "@/transport/types";
import { isRequestFrame } from "@/transport/types";
import { handleConsole } from "./console";
import { handleDownload } from "./download";
import { type EmulateCdpRunner, handleEmulate } from "./emulate";
import { handleEvaluate } from "./evaluate";
import { handleRequestHelp } from "./human-loop";
@@ -74,6 +77,7 @@ import {
type TabReturnParams,
type TabSelectParams,
} from "./tabs";
import { handleUpload } from "./upload";
import { handleWaitForNavigation } from "./waits";
import { handleWindowResize, type WindowResizeParams } from "./window";
@@ -504,6 +508,40 @@ export class ToolDispatcher {
),
signal,
);
case "tool.upload":
return this.withHoverReleaseForRequest(
req.params as UploadParams,
() =>
this.cdp
? handleUpload(this.sessions, req.params as UploadParams, {
cdp: this.cdp,
tabsApi: chromeTabsApi,
signal,
bypassOverlay,
})
: Promise.resolve({
code: "unsupported",
message: "upload requires CDP",
} satisfies RpcError),
signal,
);
case "tool.download":
return this.withHoverReleaseForRequest(
req.params as DownloadParams,
() =>
this.cdp
? handleDownload(this.sessions, req.params as DownloadParams, {
cdp: this.cdp,
tabsApi: chromeTabsApi,
signal,
bypassOverlay,
})
: Promise.resolve({
code: "unsupported",
message: "download requires CDP",
} satisfies RpcError),
signal,
);
case "tool.evaluate":
return handleEvaluate(
this.sessions,
@@ -721,6 +759,8 @@ function sessionIdForBrowserControlMethod(req: RequestFrame): string | null {
case "tool.fill":
case "tool.press":
case "tool.select":
case "tool.upload":
case "tool.download":
case "tool.evaluate":
case "tool.observe":
case "tool.request_help":
@@ -0,0 +1,349 @@
// Order-independent coordinator for one browser download. CDP supplies the
// exact target/frame intent while chrome.downloads supplies the download id
// and filename routing hook; neither event is assumed to arrive first.
import type { CdpTarget } from "@/browser-driver/frame-graph";
import type { ClickResult, RpcError, TransferEffectState } from "@/transport/types";
import { transferError } from "./errors";
import { type CdpRunner, isRpcError } from "./shared";
const CORRELATION_GRACE_MS = 750;
const UNIQUE_SETTLE_MS = 50;
const SIZE_POLL_MS = 250;
type DeterminingFilenameListener = (
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true;
interface ListenerEvent<T> {
addListener(listener: T): void;
removeListener(listener: T): void;
}
export interface DownloadsApi {
onCreated: ListenerEvent<(item: chrome.downloads.DownloadItem) => void>;
onChanged: ListenerEvent<(delta: chrome.downloads.DownloadDelta) => void>;
onDeterminingFilename: ListenerEvent<DeterminingFilenameListener>;
search(query: chrome.downloads.DownloadQuery): Promise<chrome.downloads.DownloadItem[]>;
cancel(downloadId: number): Promise<void>;
removeFile(downloadId: number): Promise<void>;
}
export const chromeDownloadsApi: DownloadsApi = {
get onCreated() {
return chrome.downloads.onCreated;
},
get onChanged() {
return chrome.downloads.onChanged;
},
get onDeterminingFilename() {
return chrome.downloads.onDeterminingFilename;
},
search: (query) => chrome.downloads.search(query),
cancel: (id) => chrome.downloads.cancel(id),
removeFile: (id) => chrome.downloads.removeFile(id),
};
export interface DownloadCaptureOptions {
cdp: CdpRunner;
target: CdpTarget;
expectedFrameId?: string;
downloads: DownloadsApi;
browserRelativeDir: string;
maxByteSize?: number;
timeoutMs: number;
signal?: AbortSignal;
trigger(): Promise<ClickResult | RpcError>;
}
export interface DownloadCaptureResult {
click: ClickResult;
item: chrome.downloads.DownloadItem;
}
interface DownloadIntent {
url: string;
suggestedFilename: string;
frameId?: string;
}
interface DownloadCandidate {
item: chrome.downloads.DownloadItem;
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void;
suggested: boolean;
graceTimer: ReturnType<typeof setTimeout>;
}
function safeBasename(filename: string): string {
const basename = filename.split(/[\\/]/).pop()?.trim();
return basename && basename !== "." && basename !== ".." ? basename : "download";
}
function sameTarget(source: { tabId?: number; sessionId?: string }, target: CdpTarget): boolean {
return source.tabId === target.tabId && source.sessionId === target.sessionId;
}
function matchesIntent(item: chrome.downloads.DownloadItem, intent: DownloadIntent): boolean {
const urlMatches = item.url === intent.url || item.finalUrl === intent.url;
return urlMatches && safeBasename(item.filename) === safeBasename(intent.suggestedFilename);
}
function knownSize(item: chrome.downloads.DownloadItem): number | undefined {
if (item.fileSize >= 0) return item.fileSize;
if (item.totalBytes >= 0) return item.totalBytes;
return undefined;
}
function captureError(
message: string,
effectState: TransferEffectState,
phase: string,
cleanupFailed = false,
): RpcError {
return transferError("cdp_failed", "download_capture_failed", message, {
effectState,
phase,
...(cleanupFailed ? { cleanupState: "failed" } : {}),
});
}
export async function captureBrowserDownload(
options: DownloadCaptureOptions,
): Promise<DownloadCaptureResult | RpcError> {
let click: ClickResult | undefined;
let intent: DownloadIntent | undefined;
let capturedId: number | undefined;
let capturedItem: chrome.downloads.DownloadItem | undefined;
let settled = false;
let succeeded = false;
let failureResult: RpcError | undefined;
let uniquenessTimer: ReturnType<typeof setTimeout> | undefined;
let operationTimer: ReturnType<typeof setTimeout> | undefined;
let sizePoll: ReturnType<typeof setInterval> | undefined;
const candidates = new Map<number, DownloadCandidate>();
const createdItems = new Map<number, chrome.downloads.DownloadItem>();
let resolveCompletion!: (item: chrome.downloads.DownloadItem) => void;
let rejectCompletion!: (error: Error) => void;
const completion = new Promise<chrome.downloads.DownloadItem>((resolve, reject) => {
resolveCompletion = resolve;
rejectCompletion = reject;
});
const fail = (error: Error) => {
if (settled) return;
settled = true;
rejectCompletion(error);
};
const complete = (item: chrome.downloads.DownloadItem) => {
if (settled) return;
const size = knownSize(item);
if (size !== undefined && options.maxByteSize !== undefined && size > options.maxByteSize) {
fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`));
return;
}
settled = true;
capturedItem = item;
resolveCompletion(item);
};
const suggestDefault = (candidate: DownloadCandidate) => {
if (candidate.suggested) return;
candidate.suggested = true;
candidate.suggest();
};
const matchingCandidates = (): DownloadCandidate[] => {
const currentIntent = intent;
return currentIntent
? [...candidates.values()].filter(
(candidate) => !candidate.suggested && matchesIntent(candidate.item, currentIntent),
)
: [];
};
const claimUnique = () => {
uniquenessTimer = undefined;
if (settled || capturedId !== undefined || !intent) return;
const matches = matchingCandidates();
if (matches.length !== 1) {
if (matches.length > 1) {
for (const candidate of matches) suggestDefault(candidate);
fail(new Error("download attribution is ambiguous"));
}
return;
}
const candidate = matches[0];
candidate.suggested = true;
clearTimeout(candidate.graceTimer);
capturedId = candidate.item.id;
capturedItem = candidate.item;
candidate.suggest({
filename: `${options.browserRelativeDir}/${safeBasename(intent.suggestedFilename)}`,
conflictAction: "overwrite",
});
const size = knownSize(candidate.item);
if (size !== undefined && options.maxByteSize !== undefined && size > options.maxByteSize) {
fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`));
return;
}
const created = createdItems.get(candidate.item.id);
if (created?.state === "interrupted") {
fail(new Error(created.error ?? "download interrupted"));
} else if (created?.state === "complete") {
complete(created);
}
};
const reconcile = () => {
if (settled || capturedId !== undefined || !intent) return;
const matches = matchingCandidates();
if (matches.length > 1) {
for (const candidate of matches) suggestDefault(candidate);
fail(new Error("download attribution is ambiguous"));
return;
}
if (matches.length === 1 && !uniquenessTimer) {
uniquenessTimer = setTimeout(claimUnique, UNIQUE_SETTLE_MS);
}
};
const determiningListener: DeterminingFilenameListener = (item, suggest) => {
const candidate: DownloadCandidate = {
item,
suggest,
suggested: false,
graceTimer: setTimeout(() => {
suggestDefault(candidate);
candidates.delete(item.id);
if (intent && matchesIntent(item, intent) && capturedId === undefined) {
fail(new Error("download correlation grace elapsed before unique attribution"));
}
}, CORRELATION_GRACE_MS),
};
candidates.set(item.id, candidate);
reconcile();
return true;
};
const createdListener = (item: chrome.downloads.DownloadItem) => {
createdItems.set(item.id, item);
if (capturedId !== item.id) return;
capturedItem = item;
if (item.state === "interrupted") {
fail(new Error(item.error ?? "download interrupted"));
} else if (item.state === "complete") {
complete(item);
}
};
const changedListener = async (delta: chrome.downloads.DownloadDelta) => {
if (capturedId === undefined || delta.id !== capturedId || settled) return;
if (delta.state?.current === "interrupted" || delta.error?.current) {
fail(new Error(delta.error?.current ?? "download interrupted"));
return;
}
if (delta.state?.current === "complete") {
try {
const [item] = await options.downloads.search({ id: delta.id });
if (item) complete(item);
else fail(new Error("completed download disappeared"));
} catch (err) {
fail(err instanceof Error ? err : new Error(String(err)));
}
}
};
const onAbort = () => fail(new DOMException("aborted", "AbortError"));
const cdpSubscription = options.cdp.onEvent?.((source, method, raw) => {
if (method !== "Page.downloadWillBegin" || !sameTarget(source, options.target)) return;
const event = raw as { url?: unknown; suggestedFilename?: unknown; frameId?: unknown };
if (typeof event.url !== "string" || typeof event.suggestedFilename !== "string") return;
if (options.expectedFrameId && event.frameId !== options.expectedFrameId) {
fail(new Error("download originated from a different frame"));
return;
}
if (intent) {
fail(new Error("download trigger produced more than one browser download intent"));
return;
}
intent = {
url: event.url,
suggestedFilename: event.suggestedFilename,
...(typeof event.frameId === "string" ? { frameId: event.frameId } : {}),
};
reconcile();
});
if (!cdpSubscription) {
return captureError("CDP download intent subscription unavailable", "none", "arm");
}
options.downloads.onDeterminingFilename.addListener(determiningListener);
options.downloads.onCreated.addListener(createdListener);
options.downloads.onChanged.addListener(changedListener);
options.signal?.addEventListener("abort", onAbort, { once: true });
operationTimer = setTimeout(
() => fail(new Error("download did not complete before timeout")),
options.timeoutMs,
);
sizePoll = setInterval(() => {
if (capturedId === undefined || settled || options.maxByteSize === undefined) return;
void options.downloads
.search({ id: capturedId })
.then(([item]) => {
if (!item || settled) return;
capturedItem = item;
if (item.bytesReceived > (options.maxByteSize as number)) {
fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`));
}
})
.catch((err) => fail(err instanceof Error ? err : new Error(String(err))));
}, SIZE_POLL_MS);
try {
const triggered = await options.trigger();
if (isRpcError(triggered)) {
void completion.catch(() => undefined);
const effect: TransferEffectState =
capturedId !== undefined ? "committed" : intent ? "unknown" : "none";
failureResult = {
...triggered,
data: { ...triggered.data, effect_state: effect, phase: "trigger" },
};
return failureResult;
}
click = triggered;
const item = await completion;
succeeded = true;
return { click, item };
} catch (err) {
const effect: TransferEffectState =
capturedId !== undefined ? "committed" : click ? "unknown" : "none";
failureResult = captureError(
err instanceof Error ? err.message : String(err),
effect,
capturedId !== undefined ? "download" : "attribution",
);
return failureResult;
} finally {
settled = true;
if (operationTimer) clearTimeout(operationTimer);
if (uniquenessTimer) clearTimeout(uniquenessTimer);
if (sizePoll) clearInterval(sizePoll);
options.signal?.removeEventListener("abort", onAbort);
options.downloads.onDeterminingFilename.removeListener(determiningListener);
options.downloads.onCreated.removeListener(createdListener);
options.downloads.onChanged.removeListener(changedListener);
cdpSubscription.dispose();
for (const candidate of candidates.values()) {
clearTimeout(candidate.graceTimer);
if (candidate.item.id !== capturedId) suggestDefault(candidate);
}
if (!succeeded && capturedId !== undefined) {
const item = capturedItem ?? createdItems.get(capturedId);
try {
if (item?.state === "complete") {
await options.downloads.removeFile(capturedId);
} else {
await options.downloads.cancel(capturedId);
}
} catch {
if (failureResult?.data) failureResult.data.cleanup_state = "failed";
}
}
}
}
+64
View File
@@ -0,0 +1,64 @@
// Download orchestration: validate session/tab ownership, then delegate one
// browser-global chrome.downloads transaction to download-capture.ts.
import type { SessionManager } from "@/session-manager/manager";
import type { DownloadParams, DownloadResult, RpcError } from "@/transport/types";
import { captureBrowserDownload, chromeDownloadsApi, type DownloadsApi } from "./download-capture";
import { clickResolvedTarget, type InteractionDeps, resolveActionTarget } from "./interaction";
import { enforceAgentWindow, isRpcError, lookupSession, resolveTargetTab } from "./shared";
let downloadActive = false;
export type { DownloadsApi } from "./download-capture";
export interface DownloadDeps extends InteractionDeps {
downloads?: DownloadsApi;
}
export async function handleDownload(
manager: SessionManager,
params: DownloadParams,
deps: DownloadDeps,
): Promise<DownloadResult | RpcError> {
if (downloadActive) return { code: "invalid_params", message: "another bsk download is active" };
downloadActive = true;
try {
const ctx = lookupSession(manager, params, "download");
if (isRpcError(ctx)) return ctx;
const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "download");
if (denied) return denied;
if (!params.browser_relative_dir) {
return { code: "invalid_params", message: "download requires a daemon capability directory" };
}
const address = await resolveActionTarget(deps.cdp, ctx, target, params, "download");
if (isRpcError(address)) return address;
const capture = await captureBrowserDownload({
cdp: deps.cdp,
target: address.cdpTarget,
downloads: deps.downloads ?? chromeDownloadsApi,
browserRelativeDir: params.browser_relative_dir,
maxByteSize: params.max_byte_size,
timeoutMs: params.timeout_ms ?? 120_000,
signal: deps.signal,
expectedFrameId: address.frameId,
trigger: () => clickResolvedTarget(ctx, address, {}, deps),
});
if (isRpcError(capture)) return capture;
const { click, item } = capture;
return {
tab_id: target.tabId,
used_ref: click.used_ref,
used_selector: click.used_selector,
suggested_filename: item.filename.split(/[\\/]/).pop() ?? "download",
byte_size: item.fileSize >= 0 ? item.fileSize : item.totalBytes,
mime: item.mime || undefined,
danger: item.danger,
browser_path: item.filename,
};
} finally {
downloadActive = false;
}
}
+41 -96
View File
@@ -1,9 +1,15 @@
// Shared CDP helpers for scrolling nodes into view and computing
// viewport-space bounding rectangles. Used by interaction tools (click
// centre) and observation tools (element screenshot clip).
// Resolve and scroll nodes inside one CDP target. Cross-frame projection
// belongs to frame-geometry.ts.
import type { RpcError } from "@/transport/types";
import { rpcError } from "./errors";
import {
parseCdpQuad,
polygonBounds,
type Region,
rectPolygon,
type ViewportRect,
} from "./geometry";
import { type CdpRunner, isRpcError } from "./shared";
const ELEMENT_NOT_VISIBLE_MESSAGE =
@@ -13,45 +19,6 @@ function elementNotVisibleError(): RpcError {
return rpcError("permission_denied", "element_not_visible", ELEMENT_NOT_VISIBLE_MESSAGE);
}
/** Viewport-space axis-aligned bounding box for CDP `Page.captureScreenshot` clip. */
export interface ViewportRect {
x: number;
y: number;
width: number;
height: number;
}
/**
* Compute the axis-aligned bounding box of an 8-double CDP content
* quad / box-model polygon. Exported for unit tests.
*/
export function quadBoundingRect(quad: number[]): ViewportRect | null {
if (quad.length !== 8) return null;
const xs = [quad[0], quad[2], quad[4], quad[6]];
const ys = [quad[1], quad[3], quad[5], quad[7]];
const minX = Math.min(...xs);
const maxX = Math.max(...xs);
const minY = Math.min(...ys);
const maxY = Math.max(...ys);
const width = maxX - minX;
const height = maxY - minY;
if (width <= 0 || height <= 0) return null;
return { x: minX, y: minY, width, height };
}
function rectToQuad(rect: ViewportRect): number[] {
return [
rect.x,
rect.y,
rect.x + rect.width,
rect.y,
rect.x + rect.width,
rect.y + rect.height,
rect.x,
rect.y + rect.height,
];
}
/**
* Resolve `backendNodeId` → CDP `objectId` so we can invoke
* `Runtime.callFunctionOn` against the live JS object.
@@ -112,48 +79,24 @@ export async function scrollNodeIntoView(
}
}
/**
* Viewport-space bounding box for `backendNodeId`. Prefers
* `DOM.getContentQuads`, then `DOM.getBoxModel`, then the union of
* visible descendant client rects for zero-size overflow containers.
*/
export async function nodeBoundingRect(
function validCdpRegion(quads: number[][] | undefined): Region {
return (quads ?? []).flatMap((raw) => {
const polygon = parseCdpQuad(raw);
return polygon && polygonBounds(polygon) ? [polygon] : [];
});
}
async function resolveVisibleContentRegion(
cdp: CdpRunner,
tabId: number,
backendNodeId: number,
): Promise<ViewportRect | RpcError> {
const polygon = await visibleContentPolygon(cdp, tabId, backendNodeId);
if (isRpcError(polygon)) return polygon;
// visibleContentPolygon only returns quads with positive area.
return quadBoundingRect(polygon)!;
}
/**
* Compute the centroid of a CDP content quad. Quads are reported as
* 8 doubles in clockwise order: (x1, y1, x2, y2, x3, y3, x4, y4).
*/
export function quadCentre(quad: number[]): { x: number; y: number } {
const x = (quad[0] + quad[4]) / 2;
const y = (quad[1] + quad[5]) / 2;
return { x, y };
}
/** Centre of a `DOM.getBoxModel` `content` polygon (also 8 doubles). */
export function boxCentre(box: number[]): { x: number; y: number } {
return quadCentre(box);
}
async function visibleContentPolygon(
cdp: CdpRunner,
tabId: number,
backendNodeId: number,
): Promise<number[] | RpcError> {
): Promise<Region | RpcError> {
try {
const quads = await cdp.send<{ quads?: number[][] }>(tabId, "DOM.getContentQuads", {
backendNodeId,
});
const quad = quads.quads?.find((q) => q.length === 8 && quadBoundingRect(q) !== null);
if (quad) return quad;
const visible = validCdpRegion(quads.quads);
if (visible.length > 0) return visible;
} catch (err) {
console.debug("[bsk element-geometry] getContentQuads failed", err);
}
@@ -162,15 +105,34 @@ async function visibleContentPolygon(
backendNodeId,
});
const content = box.model?.content;
if (content && content.length === 8 && quadBoundingRect(content)) return content;
const fallback = validCdpRegion(content ? [content] : undefined);
if (fallback.length > 0) return fallback;
} catch (err) {
console.debug("[bsk element-geometry] getBoxModel failed", err);
}
const descendantRect = await descendantBoundingRect(cdp, tabId, backendNodeId);
if (descendantRect) return rectToQuad(descendantRect);
if (descendantRect) {
return [
rectPolygon({
x: descendantRect.x,
y: descendantRect.y,
w: descendantRect.width,
h: descendantRect.height,
}),
];
}
return elementNotVisibleError();
}
/** Resolve every visible content region in the node's target-local viewport. */
export async function nodeContentRegion(
cdp: CdpRunner,
tabId: number,
backendNodeId: number,
): Promise<Region | RpcError> {
return resolveVisibleContentRegion(cdp, tabId, backendNodeId);
}
async function descendantBoundingRect(
cdp: CdpRunner,
tabId: number,
@@ -241,20 +203,3 @@ function parseViewportRect(value: unknown): ViewportRect | null {
if (![x, y, width, height].every(Number.isFinite) || width <= 0 || height <= 0) return null;
return { x, y, width, height };
}
/**
* Get the click point for `backendNodeId`. Prefers
* `DOM.getContentQuads` (richer for rotated / transformed elements);
* falls back to `DOM.getBoxModel`, then visible descendant bounds.
* Returns `permission_denied` with "element not visible" when every
* geometry path comes back empty / fails.
*/
export async function nodeCentre(
cdp: CdpRunner,
tabId: number,
backendNodeId: number,
): Promise<{ x: number; y: number } | RpcError> {
const polygon = await visibleContentPolygon(cdp, tabId, backendNodeId);
if (isRpcError(polygon)) return polygon;
return quadCentre(polygon);
}
+27 -1
View File
@@ -2,10 +2,23 @@
// rendering. Extension handlers attach reasons here; human-facing copy
// lives in bsk-cli `render_error.rs`.
import type { ErrorCode, RpcError, RpcErrorData, RpcErrorReason } from "@/transport/types";
import type {
ErrorCode,
RpcError,
RpcErrorData,
RpcErrorReason,
TransferCleanupState,
TransferEffectState,
} from "@/transport/types";
export type { RpcErrorData, RpcErrorReason };
export interface TransferErrorOptions {
effectState: TransferEffectState;
phase: string;
cleanupState?: TransferCleanupState;
}
export function rpcError(
code: ErrorCode,
reason: RpcErrorReason,
@@ -15,3 +28,16 @@ export function rpcError(
const data: RpcErrorData = { reason, ...extra };
return { code, message, data };
}
export function transferError(
code: ErrorCode,
reason: RpcErrorReason,
message: string,
options: TransferErrorOptions,
): RpcError {
return rpcError(code, reason, message, {
effect_state: options.effectState,
phase: options.phase,
...(options.cleanupState ? { cleanup_state: options.cleanupState } : {}),
});
}
@@ -0,0 +1,484 @@
// One upload transaction with an explicit browser-side commit boundary.
// Chrome interception prevents a native chooser from escaping automation;
// chooser events and a frame-scoped DOM probe are independent input-location
// signals, so event delivery is not required for standard file inputs.
import type { ClickResult, RpcError, TransferEffectState } from "@/transport/types";
import { transferError } from "./errors";
import type { ResolvedActionTarget } from "./interaction";
import { type CdpRunner, isRpcError, sendToCdpTarget } from "./shared";
const CLEANUP_TIMEOUT_MS = 1_000;
const ACTIVATION_GRACE_MS = 1_000;
const PROBE_INTERVAL_MS = 20;
type UploadPhase =
| "arm_interception"
| "arm_input_probe"
| "trigger"
| "resolve_input"
| "set_files"
| "cleanup";
interface RuntimeReply {
result?: { value?: unknown; objectId?: string };
exceptionDetails?: { text?: string; exception?: { description?: string } };
}
interface ChooserEvent {
frameId?: string;
backendNodeId?: number;
mode?: "selectSingle" | "selectMultiple";
}
export interface FileInputTransactionOptions {
cdp: CdpRunner;
actionTarget: ResolvedActionTarget;
files: string[];
timeoutMs: number;
signal?: AbortSignal;
trigger(): Promise<ClickResult | RpcError>;
}
export interface FileInputTransactionResult {
click: ClickResult;
multiple: boolean;
}
class BoundedWaitError extends Error {
constructor(
readonly kind: "timeout" | "aborted",
message: string,
) {
super(message);
}
}
function remainingMs(deadline: number): number {
return Math.max(0, deadline - Date.now());
}
async function waitBounded<T>(
promise: Promise<T>,
deadline: number,
signal: AbortSignal | undefined,
timeoutMessage: string,
): Promise<T> {
const remaining = remainingMs(deadline);
if (signal?.aborted) throw new BoundedWaitError("aborted", "upload transaction aborted");
if (remaining === 0) throw new BoundedWaitError("timeout", timeoutMessage);
let timer: ReturnType<typeof setTimeout> | undefined;
let onAbort: (() => void) | undefined;
const boundary = new Promise<never>((_resolve, reject) => {
timer = setTimeout(() => reject(new BoundedWaitError("timeout", timeoutMessage)), remaining);
if (signal) {
onAbort = () => reject(new BoundedWaitError("aborted", "upload transaction aborted"));
signal.addEventListener("abort", onAbort, { once: true });
}
});
try {
return await Promise.race([promise, boundary]);
} finally {
if (timer) clearTimeout(timer);
if (signal && onAbort) signal.removeEventListener("abort", onAbort);
}
}
function runtimeError(reply: RuntimeReply, fallback: string): Error | null {
if (!reply.exceptionDetails) return null;
return new Error(
reply.exceptionDetails.exception?.description ?? reply.exceptionDetails.text ?? fallback,
);
}
function transferFailure(
code: RpcError["code"],
reason: "file_input_probe_failed" | "file_input_not_activated" | "set_file_input_failed",
message: string,
effectState: TransferEffectState,
phase: UploadPhase,
): RpcError {
return transferError(code, reason, message, { effectState, phase });
}
function enrichFailure(
error: RpcError,
effectState: TransferEffectState,
phase: UploadPhase,
): RpcError {
return {
...error,
data: { ...error.data, effect_state: effectState, phase },
};
}
function sameTarget(
source: { tabId?: number; sessionId?: string },
target: { tabId: number; sessionId?: string },
): boolean {
return source.tabId === target.tabId && source.sessionId === target.sessionId;
}
async function delay(ms: number, signal?: AbortSignal): Promise<void> {
await waitBounded(
new Promise((resolve) => setTimeout(resolve, ms)),
Date.now() + ms + 1,
signal,
"upload activation probe timed out",
);
}
async function callOnTrigger<T = RuntimeReply>(
options: FileInputTransactionOptions,
objectId: string,
functionDeclaration: string,
args: unknown[],
returnByValue: boolean,
): Promise<T> {
return sendToCdpTarget<T>(options.cdp, options.actionTarget.cdpTarget, "Runtime.callFunctionOn", {
objectId,
functionDeclaration,
arguments: args.map((value) => ({ value })),
returnByValue,
awaitPromise: false,
});
}
export async function uploadThroughActivatedFileInput(
options: FileInputTransactionOptions,
): Promise<FileInputTransactionResult | RpcError> {
const deadline = Date.now() + options.timeoutMs;
const target = options.actionTarget.cdpTarget;
const objectGroup = `bsk-upload-${crypto.randomUUID()}`;
const stateKey = `__bskUpload_${crypto.randomUUID().replaceAll("-", "")}`;
const chooserEvents: ChooserEvent[] = [];
let interceptionArmed = false;
let probeArmed = false;
let triggerObjectId: string | undefined;
let outcome: FileInputTransactionResult | RpcError = transferFailure(
"protocol_error",
"file_input_probe_failed",
"upload transaction ended without an outcome",
"none",
"cleanup",
);
const chooserSubscription = options.cdp.onEvent?.((source, method, raw) => {
if (method !== "Page.fileChooserOpened" || !sameTarget(source, target)) return;
const event = raw as ChooserEvent;
chooserEvents.push(event);
});
try {
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", {
enabled: true,
}),
deadline,
options.signal,
"arming native file chooser interception timed out",
);
interceptionArmed = true;
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"arm_interception",
);
return outcome;
}
try {
const resolved = await waitBounded(
sendToCdpTarget<{ object?: { objectId?: string } }>(
options.cdp,
target,
"DOM.resolveNode",
{
backendNodeId: options.actionTarget.backendNodeId,
objectGroup,
},
),
deadline,
options.signal,
"resolving upload trigger timed out",
);
triggerObjectId = resolved.object?.objectId;
if (!triggerObjectId) throw new Error("DOM.resolveNode returned no trigger objectId");
const armed = await waitBounded(
callOnTrigger<RuntimeReply>(
options,
triggerObjectId,
`function(key) {
const doc = this.ownerDocument;
const owner = doc.defaultView;
if (!owner) return false;
const state = { inputs: [], listener: null };
Object.defineProperty(owner, key, { value: state, configurable: true });
state.listener = event => {
const path = typeof event.composedPath === "function" ? event.composedPath() : [];
const candidate = path[0] || event.target;
if (candidate && candidate.nodeType === 1 &&
candidate.localName === "input" && candidate.type === "file") {
if (!state.inputs.includes(candidate)) state.inputs.push(candidate);
event.preventDefault();
}
};
doc.addEventListener("click", state.listener, true);
return true;
}`,
[stateKey],
true,
),
deadline,
options.signal,
"arming frame-scoped file input probe timed out",
);
const armError = runtimeError(armed, "failed to arm file input probe");
if (armError || armed.result?.value !== true) {
throw armError ?? new Error("file input probe did not arm");
}
probeArmed = true;
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"arm_input_probe",
);
return outcome;
}
let click: ClickResult | RpcError;
try {
click = await waitBounded(
options.trigger(),
deadline,
options.signal,
"upload trigger timed out",
);
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"trigger",
);
return outcome;
}
if (isRpcError(click)) {
outcome = enrichFailure(click, "none", "trigger");
return outcome;
}
try {
const activationDeadline = Math.min(deadline, Date.now() + ACTIVATION_GRACE_MS);
let summary: { count: number; multiple: boolean } = { count: 0, multiple: false };
while (Date.now() < activationDeadline) {
if (chooserEvents.length > 0) break;
const reply = await callOnTrigger<RuntimeReply>(
options,
triggerObjectId,
`function(key) {
const state = this.ownerDocument.defaultView?.[key];
return state
? { count: state.inputs.length, multiple: state.inputs[0]?.multiple === true }
: { count: 0, multiple: false };
}`,
[stateKey],
true,
);
const summaryError = runtimeError(reply, "failed to inspect activated file input");
if (summaryError) throw summaryError;
const value = reply.result?.value as { count?: unknown; multiple?: unknown } | undefined;
summary = {
count: typeof value?.count === "number" ? value.count : 0,
multiple: value?.multiple === true,
};
if (summary.count > 0) break;
await delay(Math.min(PROBE_INTERVAL_MS, remainingMs(activationDeadline)), options.signal);
}
if (chooserEvents.length > 1) {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
"upload trigger activated more than one file chooser",
"none",
"resolve_input",
);
return outcome;
}
const chooser = chooserEvents[0];
let backendNodeId: number | undefined;
let multiple = summary.multiple;
if (chooser) {
if (options.actionTarget.frameId && chooser.frameId !== options.actionTarget.frameId) {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
"upload trigger activated a file chooser in a different frame",
"none",
"resolve_input",
);
return outcome;
}
if (typeof chooser.backendNodeId !== "number") {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
"upload trigger invoked a non-input file picker",
"none",
"resolve_input",
);
return outcome;
}
backendNodeId = chooser.backendNodeId;
multiple = chooser.mode === "selectMultiple";
} else {
if (summary.count !== 1) {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
summary.count === 0
? "upload trigger did not activate an input[type=file]"
: "upload trigger activated more than one input[type=file]",
"none",
"resolve_input",
);
return outcome;
}
const input = await callOnTrigger<RuntimeReply>(
options,
triggerObjectId,
`function(key) { return this.ownerDocument.defaultView?.[key]?.inputs[0]; }`,
[stateKey],
false,
);
const inputError = runtimeError(input, "failed to resolve activated file input object");
if (inputError) throw inputError;
if (!input.result?.objectId) throw new Error("activated file input returned no objectId");
const described = await sendToCdpTarget<{ node?: { backendNodeId?: number } }>(
options.cdp,
target,
"DOM.describeNode",
{ objectId: input.result.objectId },
);
backendNodeId = described.node?.backendNodeId;
if (typeof backendNodeId !== "number") {
throw new Error("DOM.describeNode returned no file input backendNodeId");
}
}
if (!multiple && options.files.length !== 1) {
outcome = enrichFailure(
{ code: "invalid_params", message: "file input accepts exactly one file" },
"none",
"resolve_input",
);
return outcome;
}
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "DOM.setFileInputFiles", {
files: options.files,
backendNodeId,
}),
deadline,
options.signal,
"setting file input files timed out",
);
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"set_file_input_failed",
err instanceof Error ? err.message : String(err),
"unknown",
"set_files",
);
return outcome;
}
outcome = { click, multiple };
return outcome;
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"resolve_input",
);
return outcome;
}
} finally {
chooserSubscription?.dispose();
let cleanupFailed = false;
if (probeArmed && triggerObjectId) {
try {
await waitBounded(
callOnTrigger(
options,
triggerObjectId,
`function(key) {
const owner = this.ownerDocument.defaultView;
const state = owner?.[key];
if (state?.listener) this.ownerDocument.removeEventListener("click", state.listener, true);
if (owner) delete owner[key];
}`,
[stateKey],
true,
),
Date.now() + CLEANUP_TIMEOUT_MS,
undefined,
"cleaning file input probe timed out",
);
} catch {
cleanupFailed = true;
}
}
if (interceptionArmed) {
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", {
enabled: false,
cancel: true,
}),
Date.now() + CLEANUP_TIMEOUT_MS,
undefined,
"disabling file chooser interception timed out",
);
} catch {
cleanupFailed = true;
}
}
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "Runtime.releaseObjectGroup", { objectGroup }),
Date.now() + CLEANUP_TIMEOUT_MS,
undefined,
"releasing upload object group timed out",
);
} catch {
cleanupFailed = true;
}
const effect = isRpcError(outcome)
? (outcome.data?.effect_state as TransferEffectState | undefined)
: "committed";
if (effect === "unknown" || cleanupFailed) {
await options.cdp.detach?.(target.tabId);
}
if (cleanupFailed && isRpcError(outcome)) {
outcome.data = { ...outcome.data, cleanup_state: "failed" };
}
}
}
+290
View File
@@ -0,0 +1,290 @@
import { type CdpFrame, type CdpFrameGraph, type CdpTarget } from "@/browser-driver/frame-graph";
import type { RpcError } from "@/transport/types";
import { nodeContentRegion, scrollNodeIntoView } from "./element-geometry";
import {
clipPolygon,
type GeometryProjection,
type Point,
type Polygon,
type ProjectiveEdge,
parseCdpQuad,
polygonArea,
polygonCentroid,
projectRegionToViewport,
type Quad,
type Region,
rectPolygon,
regionBounds,
type Size,
type ViewportRect,
} from "./geometry";
import { type CdpRunner, cdpRunnerForTarget, isRpcError, sendToCdpTarget } from "./shared";
export interface NodeAddress {
target: CdpTarget;
backendNodeId: number;
frameId?: string;
}
export interface ResolvedNodeGeometry {
topVisibleRegions: Region;
topBounds: ViewportRect;
actionPoint: Point;
}
function geometryError(message: string): RpcError {
return { code: "cdp_failed", message };
}
function frameMap(graph: CdpFrameGraph): Map<string, CdpFrame> {
return new Map(graph.frames.map((frame) => [frame.frameId, frame]));
}
function targetRootFrame(byId: Map<string, CdpFrame>, frame: CdpFrame): CdpFrame | null {
const seen = new Set<string>();
let current = frame;
while (current.parentFrameId) {
if (seen.has(current.frameId)) return null;
seen.add(current.frameId);
const parent = byId.get(current.parentFrameId);
if (!parent || parent.target.sessionId !== current.target.sessionId) break;
current = parent;
}
return current;
}
function frameAncestry(graph: CdpFrameGraph, frameId: string): CdpFrame[] | null {
const byId = frameMap(graph);
const path: CdpFrame[] = [];
const seen = new Set<string>();
let current = byId.get(frameId);
if (!current) return null;
while (current.parentFrameId) {
if (seen.has(current.frameId)) return null;
seen.add(current.frameId);
path.push(current);
const parent = byId.get(current.parentFrameId);
if (!parent) return null;
current = parent;
}
return path;
}
async function targetViewport(cdp: CdpRunner, target: CdpTarget): Promise<Size | null> {
const metrics = await sendToCdpTarget<{
cssLayoutViewport?: { clientWidth?: number; clientHeight?: number };
layoutViewport?: { clientWidth?: number; clientHeight?: number };
}>(cdp, target, "Page.getLayoutMetrics", {});
const viewport = metrics.cssLayoutViewport ?? metrics.layoutViewport ?? {};
const width = viewport.clientWidth ?? 0;
const height = viewport.clientHeight ?? 0;
return width > 0 && height > 0 ? { width, height } : null;
}
async function ownerContentQuad(
cdp: CdpRunner,
parent: CdpFrame,
ownerBackendNodeId: number,
): Promise<Quad | null> {
const result = await sendToCdpTarget<{ model?: { content?: number[] } }>(
cdp,
parent.target,
"DOM.getBoxModel",
{ backendNodeId: ownerBackendNodeId },
);
return parseCdpQuad(result.model?.content);
}
async function sameTargetFrameClips(
cdp: CdpRunner,
byId: Map<string, CdpFrame>,
frame: CdpFrame,
targetRoot: CdpFrame,
): Promise<Polygon[] | null> {
const clips: Polygon[] = [];
let current = frame;
const seen = new Set<string>();
while (current.frameId !== targetRoot.frameId) {
if (seen.has(current.frameId) || !current.parentFrameId) return null;
seen.add(current.frameId);
const parent = byId.get(current.parentFrameId);
if (!parent || current.ownerBackendNodeId === undefined) return null;
const clip = await ownerContentQuad(cdp, parent, current.ownerBackendNodeId);
if (!clip) return null;
clips.push(clip);
current = parent;
}
return clips;
}
export async function resolveFrameProjection(
cdp: CdpRunner,
graph: CdpFrameGraph,
frameId: string,
): Promise<GeometryProjection | null> {
const byId = frameMap(graph);
const frame = byId.get(frameId);
if (!frame) return null;
let targetRoot = targetRootFrame(byId, frame);
if (!targetRoot) return null;
const sourceViewport = await targetViewport(cdp, targetRoot.target);
if (!sourceViewport) return null;
const sourceClips = await sameTargetFrameClips(cdp, byId, frame, targetRoot);
if (!sourceClips) return null;
const edges: ProjectiveEdge[] = [];
while (targetRoot.parentFrameId) {
const parent = byId.get(targetRoot.parentFrameId);
if (!parent || targetRoot.ownerBackendNodeId === undefined) return null;
const destinationQuad = await ownerContentQuad(cdp, parent, targetRoot.ownerBackendNodeId);
if (!destinationQuad) return null;
const source =
edges.length === 0 ? sourceViewport : await targetViewport(cdp, targetRoot.target);
if (!source) return null;
const parentTargetRoot = targetRootFrame(byId, parent);
if (!parentTargetRoot) return null;
const destinationClips = await sameTargetFrameClips(cdp, byId, parent, parentTargetRoot);
if (!destinationClips) return null;
edges.push({ sourceViewport: source, destinationQuad, destinationClips });
targetRoot = parentTargetRoot;
}
const topViewport =
edges.length === 0 ? sourceViewport : await targetViewport(cdp, targetRoot.target);
return topViewport ? { sourceClips, edges, topViewport } : null;
}
async function loadFrameGraph(cdp: CdpRunner, tabId: number): Promise<CdpFrameGraph | null> {
if (!cdp.getFrameGraph) return null;
try {
return await cdp.getFrameGraph(tabId);
} catch (error) {
console.debug("[bsk frame-geometry] frame graph resolution failed", error);
return null;
}
}
async function scrollFrameOwners(
cdp: CdpRunner,
tabId: number,
graph: CdpFrameGraph,
frameId: string,
): Promise<RpcError | null> {
const byId = frameMap(graph);
const ancestry = frameAncestry(graph, frameId);
if (!ancestry) return geometryError(`could not resolve frame ancestry for ${frameId}`);
for (const child of [...ancestry].reverse()) {
const parent = child.parentFrameId ? byId.get(child.parentFrameId) : undefined;
if (!parent || child.ownerBackendNodeId === undefined) {
return geometryError(`could not resolve frame owner for ${child.frameId}`);
}
const error = await scrollNodeIntoView(
cdpRunnerForTarget(cdp, parent.target),
tabId,
child.ownerBackendNodeId,
);
if (error) return error;
}
return null;
}
async function scrollElementWithFrameGraph(
cdp: CdpRunner,
tabId: number,
target: CdpTarget,
backendNodeId: number,
frameId: string | undefined,
graph: CdpFrameGraph | null,
): Promise<RpcError | null> {
if (frameId) {
if (!graph) return geometryError(`could not resolve frame graph for ${frameId}`);
const error = await scrollFrameOwners(cdp, tabId, graph, frameId);
if (error) return error;
} else if (target.sessionId) {
return geometryError("an OOPIF node address requires frameId");
}
return scrollNodeIntoView(cdpRunnerForTarget(cdp, target), tabId, backendNodeId);
}
export async function scrollElementAndFramesIntoView(
cdp: CdpRunner,
tabId: number,
target: CdpTarget,
backendNodeId: number,
frameId?: string,
): Promise<RpcError | null> {
const graph = frameId ? await loadFrameGraph(cdp, tabId) : null;
return scrollElementWithFrameGraph(cdp, tabId, target, backendNodeId, frameId, graph);
}
function largestRegion(regions: Region): Polygon | null {
let largest: { polygon: Polygon; area: number } | null = null;
for (const polygon of regions) {
const area = polygonArea(polygon);
if (area <= 0) continue;
if (!largest || area > largest.area) largest = { polygon, area };
}
return largest?.polygon ?? null;
}
export async function resolveNodeGeometry(
cdp: CdpRunner,
tabId: number,
address: NodeAddress,
options: { scrollIntoView?: boolean } = {},
): Promise<ResolvedNodeGeometry | RpcError> {
try {
if (address.target.sessionId && !address.frameId) {
return geometryError("an OOPIF node address requires frameId");
}
const graph = address.frameId ? await loadFrameGraph(cdp, tabId) : null;
if (address.frameId && !graph) {
return geometryError(`could not resolve frame graph for ${address.frameId}`);
}
if (options.scrollIntoView) {
const scrollError = await scrollElementWithFrameGraph(
cdp,
tabId,
address.target,
address.backendNodeId,
address.frameId,
graph,
);
if (scrollError) return scrollError;
}
const localRegion = await nodeContentRegion(
cdpRunnerForTarget(cdp, address.target),
tabId,
address.backendNodeId,
);
if (isRpcError(localRegion)) return localRegion;
let topVisibleRegions: Region;
if (address.frameId && graph) {
const projection = await resolveFrameProjection(cdp, graph, address.frameId);
if (!projection)
return geometryError(`could not resolve frame geometry for ${address.frameId}`);
topVisibleRegions = projectRegionToViewport(localRegion, projection);
} else {
const viewport = await targetViewport(cdp, address.target);
if (!viewport) return geometryError("could not resolve top viewport geometry");
topVisibleRegions = localRegion
.map((polygon) =>
clipPolygon(polygon, rectPolygon({ x: 0, y: 0, w: viewport.width, h: viewport.height })),
)
.filter((polygon) => polygon.length >= 3);
}
const topBounds = regionBounds(topVisibleRegions);
const actionRegion = largestRegion(topVisibleRegions);
const actionPoint = actionRegion ? polygonCentroid(actionRegion) : null;
if (!topBounds || !actionPoint) {
return { code: "permission_denied", message: "element not visible" };
}
return { topVisibleRegions, topBounds, actionPoint };
} catch (error) {
return geometryError(error instanceof Error ? error.message : String(error));
}
}
+266
View File
@@ -0,0 +1,266 @@
export interface Point {
x: number;
y: number;
}
export type Polygon = Point[];
export type Region = Polygon[];
export type Quad = [Point, Point, Point, Point];
export interface Size {
width: number;
height: number;
}
export interface ViewportRect {
x: number;
y: number;
width: number;
height: number;
}
export interface ProjectiveEdge {
sourceViewport: Size;
destinationQuad: Quad;
destinationClips?: Polygon[];
}
export interface GeometryProjection {
sourceClips: Polygon[];
edges: ProjectiveEdge[];
topViewport: Size;
}
export function parseCdpQuad(raw: number[] | undefined): Quad | null {
if (!raw || raw.length !== 8 || !raw.every(Number.isFinite)) return null;
return [
{ x: raw[0], y: raw[1] },
{ x: raw[2], y: raw[3] },
{ x: raw[4], y: raw[5] },
{ x: raw[6], y: raw[7] },
];
}
export function rectPolygon(rect: { x: number; y: number; w: number; h: number }): Quad {
return [
{ x: rect.x, y: rect.y },
{ x: rect.x + rect.w, y: rect.y },
{ x: rect.x + rect.w, y: rect.y + rect.h },
{ x: rect.x, y: rect.y + rect.h },
];
}
export function viewportPolygon(viewport: Size): Quad {
return rectPolygon({ x: 0, y: 0, w: viewport.width, h: viewport.height });
}
/** Project a normalized unit-square point into an arbitrary content quad. */
export function projectUnitPoint(point: Point, quad: Quad): Point {
const [p0, p1, p2, p3] = quad;
const dx1 = p1.x - p2.x;
const dx2 = p3.x - p2.x;
const dx3 = p0.x - p1.x + p2.x - p3.x;
const dy1 = p1.y - p2.y;
const dy2 = p3.y - p2.y;
const dy3 = p0.y - p1.y + p2.y - p3.y;
let a: number;
let b: number;
let c: number;
let d: number;
let e: number;
let f: number;
let g = 0;
let h = 0;
if (Math.abs(dx3) < 1e-9 && Math.abs(dy3) < 1e-9) {
a = p1.x - p0.x;
b = p3.x - p0.x;
c = p0.x;
d = p1.y - p0.y;
e = p3.y - p0.y;
f = p0.y;
} else {
const denominator = dx1 * dy2 - dx2 * dy1;
if (Math.abs(denominator) < 1e-9) {
return {
x: p0.x + (p1.x - p0.x) * point.x + (p3.x - p0.x) * point.y,
y: p0.y + (p1.y - p0.y) * point.x + (p3.y - p0.y) * point.y,
};
}
g = (dx3 * dy2 - dx2 * dy3) / denominator;
h = (dx1 * dy3 - dx3 * dy1) / denominator;
a = p1.x - p0.x + g * p1.x;
b = p3.x - p0.x + h * p3.x;
c = p0.x;
d = p1.y - p0.y + g * p1.y;
e = p3.y - p0.y + h * p3.y;
f = p0.y;
}
const scale = g * point.x + h * point.y + 1;
return {
x: (a * point.x + b * point.y + c) / scale,
y: (d * point.x + e * point.y + f) / scale,
};
}
export function projectPolygon(polygon: Polygon, edge: ProjectiveEdge): Polygon {
const { width, height } = edge.sourceViewport;
if (width <= 0 || height <= 0) return [];
return polygon.map((point) =>
projectUnitPoint({ x: point.x / width, y: point.y / height }, edge.destinationQuad),
);
}
function polygonSignedArea(points: Polygon): number {
let area = 0;
for (let index = 0; index < points.length; index += 1) {
const current = points[index];
const next = points[(index + 1) % points.length];
area += current.x * next.y - next.x * current.y;
}
return area / 2;
}
function edgeIntersection(start: Point, end: Point, clipStart: Point, clipEnd: Point): Point {
const subjectX = end.x - start.x;
const subjectY = end.y - start.y;
const clipX = clipEnd.x - clipStart.x;
const clipY = clipEnd.y - clipStart.y;
const denominator = subjectX * clipY - subjectY * clipX;
if (Math.abs(denominator) < 1e-9) return end;
const offsetX = clipStart.x - start.x;
const offsetY = clipStart.y - start.y;
const scale = (offsetX * clipY - offsetY * clipX) / denominator;
return { x: start.x + scale * subjectX, y: start.y + scale * subjectY };
}
/** Clip a polygon to a convex polygon, preserving every resulting vertex. */
export function clipPolygon(subject: Polygon, clip: Polygon): Polygon {
if (subject.length < 3 || clip.length < 3) return [];
const orientation = polygonSignedArea(clip) >= 0 ? 1 : -1;
let output = [...subject];
for (let edge = 0; edge < clip.length && output.length > 0; edge += 1) {
const clipStart = clip[edge];
const clipEnd = clip[(edge + 1) % clip.length];
const input = output;
output = [];
const inside = (point: Point) =>
orientation *
((clipEnd.x - clipStart.x) * (point.y - clipStart.y) -
(clipEnd.y - clipStart.y) * (point.x - clipStart.x)) >=
-1e-9;
let start = input[input.length - 1];
for (const end of input) {
const startInside = inside(start);
const endInside = inside(end);
if (endInside) {
if (!startInside) output.push(edgeIntersection(start, end, clipStart, clipEnd));
output.push(end);
} else if (startInside) {
output.push(edgeIntersection(start, end, clipStart, clipEnd));
}
start = end;
}
}
return output;
}
export function polygonBounds(points: Polygon): ViewportRect | null {
if (points.length < 3) return null;
const xs = points.map((point) => point.x);
const ys = points.map((point) => point.y);
const x = Math.min(...xs);
const y = Math.min(...ys);
const right = Math.max(...xs);
const bottom = Math.max(...ys);
if (right <= x || bottom <= y) return null;
return { x, y, width: right - x, height: bottom - y };
}
export function regionBounds(region: Region): ViewportRect | null {
return polygonBounds(region.flat());
}
export function polygonArea(points: Polygon): number {
return Math.abs(polygonSignedArea(points));
}
export function polygonCentroid(points: Polygon): Point | null {
if (points.length < 3) return null;
let twiceArea = 0;
let x = 0;
let y = 0;
for (let index = 0; index < points.length; index += 1) {
const current = points[index];
const next = points[(index + 1) % points.length];
const cross = current.x * next.y - next.x * current.y;
twiceArea += cross;
x += (current.x + next.x) * cross;
y += (current.y + next.y) * cross;
}
if (Math.abs(twiceArea) < 1e-9) return null;
return { x: x / (3 * twiceArea), y: y / (3 * twiceArea) };
}
export function projectAndClipRegion(
region: Region,
edges: ProjectiveEdge[],
topViewport: Size,
): Region {
let projected = region;
for (const edge of edges) {
const sourceClip = viewportPolygon(edge.sourceViewport);
projected = projected
.map((polygon) => clipPolygon(polygon, sourceClip))
.filter((polygon) => polygon.length >= 3)
.map((polygon) =>
[edge.destinationQuad, ...(edge.destinationClips ?? [])].reduce(
(current, clip) => clipPolygon(current, clip),
projectPolygon(polygon, edge),
),
)
.filter((polygon) => polygon.length >= 3);
}
const topClip = viewportPolygon(topViewport);
return projected
.map((polygon) => clipPolygon(polygon, topClip))
.filter((polygon) => polygon.length >= 3);
}
export function projectRegionToViewport(region: Region, projection: GeometryProjection): Region {
const clipped = region
.map((polygon) =>
projection.sourceClips.reduce((current, clip) => clipPolygon(current, clip), polygon),
)
.filter((polygon) => polygon.length >= 3);
return projectAndClipRegion(clipped, projection.edges, projection.topViewport);
}
export function projectRectToViewport(
rect: { x: number; y: number; w: number; h: number } | null,
projection: GeometryProjection,
): ViewportRect | null {
if (!rect) return null;
return regionBounds(projectRegionToViewport([rectPolygon(rect)], projection));
}
export function childFrameProjection(
parent: GeometryProjection,
ownerRectInParent: { x: number; y: number; w: number; h: number },
): GeometryProjection {
const destinationQuad = rectPolygon(ownerRectInParent);
return {
sourceClips: [],
edges: [
{
sourceViewport: { width: ownerRectInParent.w, height: ownerRectInParent.h },
destinationQuad,
destinationClips: parent.sourceClips,
},
...parent.edges,
],
topViewport: parent.topViewport,
};
}
+62 -68
View File
@@ -8,6 +8,7 @@ import {
HELP_REQUEST,
type HelpCancelMessage,
type HelpFinishMessage,
type HelpHighlightRect,
type HelpQueryResponse,
type HelpRequestMessage,
isHelpAckMessage,
@@ -25,6 +26,7 @@ import type {
ResolvedTarget,
RpcError,
} from "@/transport/types";
import { resolveNodeGeometry } from "./frame-geometry";
import {
type CdpRunner,
type ChromeTabsApi,
@@ -37,7 +39,6 @@ import {
import { lookupSnapshotRef } from "./snapshot-ref";
const DEFAULT_HELP_TIMEOUT_MS = 300_000;
const HELP_ATTR = "data-bsk-help";
const HELP_SEND_RETRIES = 3;
const HELP_SEND_RETRY_DELAY_MS = 350;
const HELP_REARM_DEBOUNCE_MS = 150;
@@ -74,6 +75,7 @@ interface ActiveHelpRequest {
title?: string;
targets: HelpTarget[];
selectors: string[];
rects: HelpHighlightRect[];
timeoutMs: number;
notificationId: string;
resolvedTargets?: ResolvedTarget[];
@@ -134,6 +136,7 @@ function helpRequestMessage(help: ActiveHelpRequest, tabId: number): HelpRequest
...(help.title ? { title: help.title } : {}),
displayMode: isPrimaryTab ? "full" : "compact",
selectors: isPrimaryTab ? help.selectors : [],
rects: isPrimaryTab ? help.rects : [],
timeoutMs: help.timeoutMs,
};
}
@@ -165,33 +168,6 @@ async function sendHelpRequestWithAck(
throw lastError ?? new Error("failed to show help overlay");
}
async function tagRefTarget(
cdp: CdpRunner,
tabId: number,
backendNodeId: number,
index: number,
): Promise<string | null> {
let objectId: string | undefined;
try {
const resolved = await cdp.send<{ object?: { objectId?: string } }>(tabId, "DOM.resolveNode", {
backendNodeId,
});
objectId = resolved.object?.objectId;
if (!objectId) return null;
await cdp.send(tabId, "Runtime.callFunctionOn", {
objectId,
functionDeclaration: `function(){ this.setAttribute("${HELP_ATTR}", "${index}"); }`,
});
return `[${HELP_ATTR}="${index}"]`;
} catch {
return null;
} finally {
if (objectId) {
await cdp.send(tabId, "Runtime.releaseObject", { objectId }).catch(() => {});
}
}
}
async function selectorExists(
cdp: CdpRunner,
tabId: number,
@@ -209,36 +185,22 @@ async function selectorExists(
}
}
async function clearRefTags(cdp: CdpRunner | undefined, tabId: number): Promise<void> {
if (!cdp) return;
try {
const doc = await cdp.send<{ root?: { nodeId?: number } }>(tabId, "DOM.getDocument", {
depth: 0,
});
const rootId = doc.root?.nodeId;
if (rootId === undefined) return;
const { nodeIds } = await cdp.send<{ nodeIds: number[] }>(tabId, "DOM.querySelectorAll", {
nodeId: rootId,
selector: `[${HELP_ATTR}]`,
});
for (const nodeId of nodeIds ?? []) {
await cdp.send(tabId, "DOM.removeAttribute", { nodeId, name: HELP_ATTR }).catch(() => {});
}
} catch {
// Best-effort cleanup.
}
}
async function resolveHelpTargets(
ctx: ReturnType<SessionManager["get"]>,
tabId: number,
targets: HelpTarget[],
deps: RequestHelpDeps,
): Promise<{ selectors: string[]; resolvedTargets?: ResolvedTarget[] }> {
options: { scrollIntoView: boolean },
): Promise<{
selectors: string[];
rects: HelpHighlightRect[];
resolvedTargets?: ResolvedTarget[];
}> {
const selectors: string[] = [];
const rects: HelpHighlightRect[] = [];
const resolved: ResolvedTarget[] = [];
let rootNodeId: number | undefined;
if (deps.cdp && targets.length > 0) {
if (deps.cdp && targets.some((target) => Boolean(target.selector))) {
try {
const doc = await deps.cdp.send<{ root?: { nodeId?: number } }>(tabId, "DOM.getDocument", {
depth: 0,
@@ -262,28 +224,55 @@ async function resolveHelpTargets(
resolved.push({ matched, selector: tgt.selector });
} else if (tgt.ref) {
const looked = ctx ? lookupSnapshotRef(ctx, tgt.ref, tabId) : null;
const backendNodeId = looked?.backendNodeId ?? null;
let sel: string | null = null;
if (backendNodeId !== null && deps.cdp) {
sel = await tagRefTarget(deps.cdp, tabId, backendNodeId, i);
let rect: HelpHighlightRect | null = null;
if (looked && deps.cdp) {
const target = {
tabId,
...(looked.cdpSessionId ? { sessionId: looked.cdpSessionId } : {}),
};
const geometry = await resolveNodeGeometry(
deps.cdp,
tabId,
{
target,
backendNodeId: looked.backendNodeId,
...(looked.frameId ? { frameId: looked.frameId } : {}),
},
{ scrollIntoView: options.scrollIntoView },
);
if (!isRpcError(geometry)) {
rect = {
top: geometry.topBounds.y,
left: geometry.topBounds.x,
width: geometry.topBounds.width,
height: geometry.topBounds.height,
};
}
}
if (sel) selectors.push(sel);
resolved.push({ matched: sel !== null, ref: tgt.ref });
if (rect) rects.push(rect);
resolved.push({
matched: rect !== null,
ref: tgt.ref,
});
}
}
return { selectors, resolvedTargets: resolved.length > 0 ? resolved : undefined };
return { selectors, rects, resolvedTargets: resolved.length > 0 ? resolved : undefined };
}
async function refreshHelpTargets(help: ActiveHelpRequest): Promise<void> {
await clearRefTags(help.deps.cdp, help.primaryTabId);
const { selectors, resolvedTargets } = await resolveHelpTargets(
async function refreshHelpTargets(
help: ActiveHelpRequest,
options: { scrollIntoView: boolean } = { scrollIntoView: true },
): Promise<void> {
const { selectors, rects, resolvedTargets } = await resolveHelpTargets(
help.ctx,
help.primaryTabId,
help.targets,
help.deps,
options,
);
help.selectors = selectors;
help.rects = rects;
help.resolvedTargets = resolvedTargets;
}
@@ -294,12 +283,7 @@ async function cleanupHelp(help: ActiveHelpRequest): Promise<void> {
const tabsToCancel = new Set([help.primaryTabId, ...help.overlayTabIds]);
await Promise.all(
[...tabsToCancel].map((tabId) =>
Promise.all([
clearRefTags(help.deps.cdp, tabId),
help.deps
.sendToTab(tabId, { type: HELP_CANCEL, requestId: help.requestId })
.catch(() => {}),
]),
help.deps.sendToTab(tabId, { type: HELP_CANCEL, requestId: help.requestId }).catch(() => {}),
),
);
}
@@ -476,7 +460,9 @@ function attachHelpRuntimeListener(deps: RequestHelpDeps): () => void {
sendResponse({ active: false });
return;
}
if (tabId === help.primaryTabId) await refreshHelpTargets(help);
if (tabId === help.primaryTabId) {
await refreshHelpTargets(help, { scrollIntoView: false });
}
help.overlayTabIds.add(tabId);
sendResponse({
active: true,
@@ -486,6 +472,7 @@ function attachHelpRuntimeListener(deps: RequestHelpDeps): () => void {
...(help.title ? { title: help.title } : {}),
displayMode: tabId === help.primaryTabId ? "full" : "compact",
selectors: tabId === help.primaryTabId ? help.selectors : [],
rects: tabId === help.primaryTabId ? help.rects : [],
timeoutMs: help.timeoutMs,
},
});
@@ -686,7 +673,13 @@ export async function handleRequestHelp(
if (deps.autoAttachLifecycle !== false) ensureHelpLifecycleListeners(deps);
const tabId = target.tabId;
const initialTargets = params.targets ?? [];
const { selectors, resolvedTargets } = await resolveHelpTargets(ctx, tabId, initialTargets, deps);
const { selectors, rects, resolvedTargets } = await resolveHelpTargets(
ctx,
tabId,
initialTargets,
deps,
{ scrollIntoView: true },
);
await deps.windows.update(target.windowId, { focused: true }).catch(() => {});
await deps.activateTab(tabId).catch(() => {});
@@ -729,6 +722,7 @@ export async function handleRequestHelp(
...(params.title ? { title: params.title } : {}),
targets: initialTargets,
selectors,
rects,
timeoutMs,
notificationId,
resolvedTargets,
+124 -62
View File
@@ -3,14 +3,15 @@
//
// All interaction tools:
// 1. Resolve target tab (sandbox: must be inside Agent Window).
// 2. Resolve target element by `ref` (RefStore.resolve with tabId
// binding) or `selector` (DOM.querySelector + describeNode).
// 3. Scroll the node into view, then dispatch the appropriate
// `Input.*` CDP events.
// 2. Resolve target element by `ref` (compound frame/session identity)
// or `selector` (DOM.querySelector + describeNode).
// 3. Scroll the node and its frame owners into view, project its live
// content quad into the top viewport, then dispatch `Input.*` events.
// 4. Honour `AbortSignal` so canceled calls don't issue follow-up CDP
// commands.
import { ChromiumCdp } from "@/browser-driver/chromium-cdp";
import type { CdpTarget } from "@/browser-driver/frame-graph";
import type { SessionContext, SessionManager } from "@/session-manager/manager";
import type {
ClickParams,
@@ -28,21 +29,18 @@ import type {
SelectResult,
} from "@/transport/types";
import { attachDialogs, markDialogCursor } from "./dialogs";
import {
backendNodeToObject,
boxCentre,
nodeCentre,
quadCentre,
scrollNodeIntoView,
} from "./element-geometry";
import { backendNodeToObject } from "./element-geometry";
import { rpcError } from "./errors";
import { resolveNodeGeometry, scrollElementAndFramesIntoView } from "./frame-geometry";
import {
type CdpRunner,
type ChromeTabsApi,
cdpRunnerForTarget,
chromeTabsApi,
enforceAgentWindow,
isRpcError,
lookupSession,
type ResolvedTargetTab,
resolveTargetTab,
} from "./shared";
import { resolveSnapshotRef } from "./snapshot-ref";
@@ -59,6 +57,15 @@ export interface InteractionDeps {
keepOverlayBypassAfterHover?: boolean;
}
export interface ResolvedActionTarget {
tab: ResolvedTargetTab;
backendNodeId: number;
cdpTarget: CdpTarget;
frameId?: string;
usedRef?: string;
usedSelector?: string;
}
const DEFAULT_TIMEOUT_MS = 30_000;
const DEFAULT_HOVER_SETTLE_MS = 200;
@@ -131,13 +138,22 @@ async function wait(ms: number, signal?: AbortSignal): Promise<void> {
* `RpcError` if the caller supplied neither (or both), or if neither
* lookup matched.
*/
async function resolveBackendNode(
export async function resolveBackendNode(
cdp: CdpRunner,
ctx: SessionContext,
target: { tabId: number },
params: { ref?: string; selector?: string },
toolName: string,
): Promise<{ backendNodeId: number; usedRef?: string; usedSelector?: string } | RpcError> {
): Promise<
| {
backendNodeId: number;
cdpTarget: CdpTarget;
frameId?: string;
usedRef?: string;
usedSelector?: string;
}
| RpcError
> {
const hasRef = typeof params.ref === "string" && params.ref.length > 0;
const hasSelector = typeof params.selector === "string" && params.selector.length > 0;
if (hasRef && hasSelector) {
@@ -155,7 +171,15 @@ async function resolveBackendNode(
if (hasRef) {
const resolved = resolveSnapshotRef(ctx, params.ref as string, target.tabId);
if (isRpcError(resolved)) return resolved;
return { backendNodeId: resolved.backendNodeId, usedRef: resolved.refKey };
return {
backendNodeId: resolved.backendNodeId,
cdpTarget: {
tabId: target.tabId,
...(resolved.cdpSessionId ? { sessionId: resolved.cdpSessionId } : {}),
},
...(resolved.frameId ? { frameId: resolved.frameId } : {}),
usedRef: resolved.refKey,
};
}
// selector path
try {
@@ -192,7 +216,7 @@ async function resolveBackendNode(
message: "DOM.describeNode returned no backendNodeId",
};
}
return { backendNodeId, usedSelector: params.selector };
return { backendNodeId, cdpTarget: { tabId: target.tabId }, usedSelector: params.selector };
} catch (err) {
return {
code: "cdp_failed",
@@ -201,6 +225,17 @@ async function resolveBackendNode(
}
}
export async function resolveActionTarget(
cdp: CdpRunner,
ctx: SessionContext,
target: ResolvedTargetTab,
params: { ref?: string; selector?: string },
toolName: string,
): Promise<ResolvedActionTarget | RpcError> {
const node = await resolveBackendNode(cdp, ctx, target, params, toolName);
return isRpcError(node) ? node : { tab: target, ...node };
}
// ---------------------------------------------------------------------------
// tool.click
// ---------------------------------------------------------------------------
@@ -219,28 +254,37 @@ export async function handleClick(
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "click");
if (denied) return denied;
const dialogCursor = markDialogCursor(deps.cdp, target.tabId);
const resolved = await resolveActionTarget(deps.cdp, ctx, target, params, "click");
if (isRpcError(resolved)) return resolved;
return clickResolvedTarget(ctx, resolved, params, deps);
}
const node = await resolveBackendNode(deps.cdp, ctx, target, params, "click");
if (isRpcError(node)) return node;
export async function clickResolvedTarget(
ctx: SessionContext,
resolved: ResolvedActionTarget,
params: Pick<ClickParams, "button" | "click_count" | "modifiers">,
deps: InteractionDeps,
): Promise<ClickResult | RpcError> {
const { tab: target } = resolved;
const dialogCursor = markDialogCursor(deps.cdp, target.tabId);
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "click aborted" };
}
try {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const scrollErr = await scrollNodeIntoView(deps.cdp, target.tabId, node.backendNodeId);
if (scrollErr) return scrollErr;
} catch (err) {
return {
code: "cdp_failed",
message: err instanceof Error ? err.message : String(err),
};
}
const centre = await nodeCentre(deps.cdp, target.tabId, node.backendNodeId);
if (isRpcError(centre)) return centre;
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const geometry = await resolveNodeGeometry(
deps.cdp,
target.tabId,
{
target: resolved.cdpTarget,
backendNodeId: resolved.backendNodeId,
...(resolved.frameId ? { frameId: resolved.frameId } : {}),
},
{ scrollIntoView: true },
);
if (isRpcError(geometry)) return geometry;
const centre = geometry.actionPoint;
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "click aborted" };
@@ -323,8 +367,8 @@ export async function handleClick(
return attachDialogs(deps.cdp, target.tabId, dialogCursor, {
tab_id: target.tabId,
used_ref: node.usedRef,
used_selector: node.usedSelector,
used_ref: resolved.usedRef,
used_selector: resolved.usedSelector,
x: centre.x,
y: centre.y,
});
@@ -353,19 +397,19 @@ export async function handleHover(
const node = await resolveBackendNode(deps.cdp, ctx, target, params, "hover");
if (isRpcError(node)) return node;
try {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const scrollErr = await scrollNodeIntoView(deps.cdp, target.tabId, node.backendNodeId);
if (scrollErr) return scrollErr;
} catch (err) {
return {
code: "cdp_failed",
message: err instanceof Error ? err.message : String(err),
};
}
const centre = await nodeCentre(deps.cdp, target.tabId, node.backendNodeId);
if (isRpcError(centre)) return centre;
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const geometry = await resolveNodeGeometry(
deps.cdp,
target.tabId,
{
target: node.cdpTarget,
backendNodeId: node.backendNodeId,
...(node.frameId ? { frameId: node.frameId } : {}),
},
{ scrollIntoView: true },
);
if (isRpcError(geometry)) return geometry;
const centre = geometry.actionPoint;
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "hover aborted" };
@@ -585,10 +629,11 @@ export async function handleFill(
const node = await resolveBackendNode(deps.cdp, ctx, target, params, "fill");
if (isRpcError(node)) return node;
const nodeCdp = cdpRunnerForTarget(deps.cdp, node.cdpTarget);
try {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const described = await deps.cdp.send<{ node?: DescribedNode }>(
const described = await nodeCdp.send<{ node?: DescribedNode }>(
target.tabId,
"DOM.describeNode",
{
@@ -602,12 +647,18 @@ export async function handleFill(
`element ${described.node?.nodeName ?? "?"} not fillable (need input/textarea/contenteditable)`,
);
}
const scrollErr = await scrollNodeIntoView(deps.cdp, target.tabId, node.backendNodeId);
const scrollErr = await scrollElementAndFramesIntoView(
deps.cdp,
target.tabId,
node.cdpTarget,
node.backendNodeId,
node.frameId,
);
if (scrollErr) return scrollErr;
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "fill aborted" };
}
await deps.cdp.send(target.tabId, "DOM.focus", { backendNodeId: node.backendNodeId });
await nodeCdp.send(target.tabId, "DOM.focus", { backendNodeId: node.backendNodeId });
} catch (err) {
return {
code: "cdp_failed",
@@ -619,7 +670,7 @@ export async function handleFill(
return { code: "cancelled", message: "fill aborted" };
}
const objectIdOrErr = await backendNodeToObject(deps.cdp, target.tabId, node.backendNodeId);
const objectIdOrErr = await backendNodeToObject(nodeCdp, target.tabId, node.backendNodeId);
if (isRpcError(objectIdOrErr)) return objectIdOrErr;
const objectId = objectIdOrErr;
const clearBefore = params.clear_before ?? true;
@@ -628,7 +679,7 @@ export async function handleFill(
if (clearBefore) {
// Clear input/textarea value or wipe contenteditable innerText,
// then fire `input` so frameworks observe the empty state.
await deps.cdp.send(target.tabId, "Runtime.callFunctionOn", {
await nodeCdp.send(target.tabId, "Runtime.callFunctionOn", {
objectId,
functionDeclaration: `function() {
if (this.isContentEditable) { this.textContent = ''; }
@@ -655,7 +706,7 @@ export async function handleFill(
return { code: "cancelled", message: "fill aborted" };
}
// Fire `input` + `change` so React / Vue controlled inputs commit.
await deps.cdp.send(target.tabId, "Runtime.callFunctionOn", {
await nodeCdp.send(target.tabId, "Runtime.callFunctionOn", {
objectId,
functionDeclaration: `function() {
this.dispatchEvent(new Event('input', { bubbles: true }));
@@ -853,14 +904,21 @@ export async function handlePress(
if (params.ref || params.selector) {
const node = await resolveBackendNode(deps.cdp, ctx, target, params, "press");
if (isRpcError(node)) return node;
const nodeCdp = cdpRunnerForTarget(deps.cdp, node.cdpTarget);
try {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const scrollErr = await scrollNodeIntoView(deps.cdp, target.tabId, node.backendNodeId);
const scrollErr = await scrollElementAndFramesIntoView(
deps.cdp,
target.tabId,
node.cdpTarget,
node.backendNodeId,
node.frameId,
);
if (scrollErr) return scrollErr;
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "press aborted" };
}
await deps.cdp.send(target.tabId, "DOM.focus", { backendNodeId: node.backendNodeId });
await nodeCdp.send(target.tabId, "DOM.focus", { backendNodeId: node.backendNodeId });
} catch (err) {
return {
code: "cdp_failed",
@@ -965,10 +1023,11 @@ export async function handleSelect(
const node = await resolveBackendNode(deps.cdp, ctx, target, params, "select");
if (isRpcError(node)) return node;
const nodeCdp = cdpRunnerForTarget(deps.cdp, node.cdpTarget);
try {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const described = await deps.cdp.send<{ node?: DescribedNode }>(
const described = await nodeCdp.send<{ node?: DescribedNode }>(
target.tabId,
"DOM.describeNode",
{ backendNodeId: node.backendNodeId },
@@ -991,12 +1050,18 @@ export async function handleSelect(
"single-select <select> requires exactly one value",
);
}
const scrollErr = await scrollNodeIntoView(deps.cdp, target.tabId, node.backendNodeId);
const scrollErr = await scrollElementAndFramesIntoView(
deps.cdp,
target.tabId,
node.cdpTarget,
node.backendNodeId,
node.frameId,
);
if (scrollErr) return scrollErr;
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "select aborted" };
}
await deps.cdp.send(target.tabId, "DOM.focus", { backendNodeId: node.backendNodeId });
await nodeCdp.send(target.tabId, "DOM.focus", { backendNodeId: node.backendNodeId });
} catch (err) {
return {
code: "cdp_failed",
@@ -1008,12 +1073,12 @@ export async function handleSelect(
return { code: "cancelled", message: "select aborted" };
}
const objectIdOrErr = await backendNodeToObject(deps.cdp, target.tabId, node.backendNodeId);
const objectIdOrErr = await backendNodeToObject(nodeCdp, target.tabId, node.backendNodeId);
if (isRpcError(objectIdOrErr)) return objectIdOrErr;
const objectId = objectIdOrErr;
try {
const evaluated = await deps.cdp.send<{
const evaluated = await nodeCdp.send<{
result?: { value?: SelectMutationResult | null };
}>(target.tabId, "Runtime.callFunctionOn", {
objectId,
@@ -1072,9 +1137,6 @@ export async function handleSelect(
export const __testing__ = {
DEFAULT_TIMEOUT_MS,
quadCentre,
boxCentre,
resolveBackendNode,
nodeCentre,
isFillable,
};
+204 -37
View File
@@ -13,6 +13,7 @@ import {
type VomScene,
} from "@browser-skill/vom";
import { ChromiumCdp } from "@/browser-driver/chromium-cdp";
import type { CdpTarget } from "@/browser-driver/frame-graph";
import {
type CaptureSuppressSendToTab,
withOverlaysHiddenForCapture,
@@ -30,8 +31,8 @@ import type {
SnapshotResult,
} from "@/transport/types";
import { attachDialogs, markDialogCursor } from "./dialogs";
import { nodeBoundingRect, scrollNodeIntoView } from "./element-geometry";
import { rpcError } from "./errors";
import { resolveNodeGeometry } from "./frame-geometry";
import {
type ChromeTabsApi,
enforceToolTargetScope,
@@ -40,6 +41,7 @@ import {
type ResolvedTargetTab,
resolveCdpAccessibleTargetTab,
type CdpRunner as SharedCdpRunner,
sendToCdpTarget,
normaliseRef as sharedNormaliseRef,
type ToolEffect,
} from "./shared";
@@ -50,6 +52,7 @@ import {
captureViewModel,
collectOverlayExcludedBackendIds,
} from "./vom/capture";
import { type CapturedFrameDocument, captureFrameData } from "./vom/frame-capture";
// ---------------------------------------------------------------------------
// Shared helpers (legacy aliases — observation.ts kept exporting these
@@ -162,26 +165,30 @@ function throwIfAborted(signal: AbortSignal | undefined, tool: string): void {
async function captureElementScreenshot(
cdp: SharedCdpRunner,
tabId: number,
target: CdpTarget,
backendNodeId: number,
frameId?: string,
signal?: AbortSignal,
): Promise<{ image_base64: string; width: number; height: number } | RpcError> {
if (signal?.aborted) return cancelled("screenshot");
const scrollErr = await scrollNodeIntoView(cdp, tabId, backendNodeId);
if (scrollErr) return scrollErr;
if (signal?.aborted) return cancelled("screenshot");
const rectOrErr = await nodeBoundingRect(cdp, tabId, backendNodeId);
if (isRpcError(rectOrErr)) return rectOrErr;
const geometry = await resolveNodeGeometry(
cdp,
tabId,
{ target, backendNodeId, ...(frameId ? { frameId } : {}) },
{ scrollIntoView: true },
);
if (isRpcError(geometry)) return geometry;
if (signal?.aborted) return cancelled("screenshot");
const rect = geometry.topBounds;
try {
const shot = await cdp.send<{ data?: string }>(tabId, "Page.captureScreenshot", {
format: "png",
clip: {
x: rectOrErr.x,
y: rectOrErr.y,
width: rectOrErr.width,
height: rectOrErr.height,
x: rect.x,
y: rect.y,
width: rect.width,
height: rect.height,
scale: 1,
},
});
@@ -191,8 +198,8 @@ async function captureElementScreenshot(
return { code: "cdp_failed", message: "Page.captureScreenshot returned no data" };
}
const dims = parsePngDimensions(image_base64) ?? {
width: Math.round(rectOrErr.width),
height: Math.round(rectOrErr.height),
width: Math.round(rect.width),
height: Math.round(rect.height),
};
return { image_base64, width: dims.width, height: dims.height };
} catch (err) {
@@ -296,9 +303,21 @@ export async function handleScreenshot(
await deps.cdp.ensureAttachedToUrl?.(target.tabId, target.url);
if (signal?.aborted) return cancelled("screenshot");
const cdp = deps.cdp;
const nodeTarget = {
tabId: target.tabId,
...(node.cdpSessionId ? { sessionId: node.cdpSessionId } : {}),
};
const captured = await withOverlaysHiddenForCapture(
target.tabId,
() => captureElementScreenshot(cdp, target.tabId, node.backendNodeId, signal),
() =>
captureElementScreenshot(
cdp,
target.tabId,
nodeTarget,
node.backendNodeId,
node.frameId,
signal,
),
deps.sendToTab,
);
if (isRpcError(captured)) return captured;
@@ -352,6 +371,7 @@ export type CdpRunner = SharedCdpRunner;
/** Subset of CDP `AXNode` we care about — see `Accessibility.AXNode`. */
export interface CdpAxNode {
nodeId: string;
frameId?: string;
parentId?: string;
backendDOMNodeId?: number;
ignored?: boolean;
@@ -584,7 +604,7 @@ function iframeNameFor(node: CapturedNode): string | undefined {
function isRenderableIframeControl(node: CapturedNode): boolean {
const tag = normalizeTag(node.tag);
if (!IFRAME_RENDERABLE_TAGS.has(tag)) return false;
if (!node.rect) return false;
if (!node.localRect && !node.rect) return false;
if (node.pointerEvents === "none") return false;
if ((node.attrs.type ?? "").toLowerCase() === "hidden") return false;
@@ -756,7 +776,12 @@ function applyCapturedSignals(
signals: VomNodeDomSignals,
): VomNode {
if (!capturedNode) return node;
const attrs = capturedNode.attrs;
const attrs =
node.sensitive && Object.prototype.hasOwnProperty.call(capturedNode.attrs, "value")
? Object.fromEntries(
Object.entries(capturedNode.attrs).filter(([name]) => name.toLowerCase() !== "value"),
)
: capturedNode.attrs;
const text = cleanAttr(capturedNode.textContent);
const nearbyText = nearbyTextFor(capturedNode, signals.childrenByParentId);
const placeholder = cleanAttr(capturedNode.formPlaceholder) ?? cleanAttr(attrs.placeholder);
@@ -823,8 +848,7 @@ function vomNodeFromCaptured(
signals: VomNodeDomSignals,
): VomNode {
const sensitive = isSensitive(capturedNode);
const value =
capturedNode.formValue ?? (sensitive ? (capturedNode.attrs.value ?? "") : undefined);
const value = sensitive ? undefined : capturedNode.formValue;
const tag = normalizeTag(capturedNode.tag);
const node = applyCapturedSignals(
{
@@ -860,8 +884,17 @@ function axVomNode(
): VomNode {
const role = axString(axNode.role);
const capturedValue = capturedNode?.formValue;
const value = capturedValue ?? axValue(axNode.value);
const sourceValue = capturedValue ?? axValue(axNode.value);
const signalsForNode = axSignals.get(axNode.nodeId);
const sensitive = isSensitive(capturedNode) || signalsForNode?.sensitive === true;
const value = sensitive ? undefined : sourceValue;
const inputState =
inputStateFor(capturedNode, sourceValue) ??
(sensitive && ["textbox", "searchbox"].includes(role?.toLowerCase() ?? "")
? sourceValue === undefined || sourceValue === ""
? "empty"
: "filled"
: undefined);
let name = FORM_CONTROL_TAGS.has(normalizeTag(capturedNode?.tag))
? formControlName(capturedNode, axString(axNode.name), axSiblingLabel(axNode, axById), signals)
: (axString(axNode.name) ?? signalsForNode?.aggregatedText);
@@ -877,8 +910,8 @@ function axVomNode(
position: capturedNode?.position || "static",
pointerEvents: capturedNode?.pointerEvents || "auto",
modal: isModalSignal(axNode, capturedNode),
sensitive: isSensitive(capturedNode) || signalsForNode?.sensitive === true,
inputState: inputStateFor(capturedNode, value),
sensitive,
inputState,
};
if (role) node.role = role;
if (name) node.name = name;
@@ -1141,6 +1174,8 @@ export interface BuildVomSceneOptions {
pageUrl?: string;
}
export type VomFrameDocument = CapturedFrameDocument<CdpAxNode>;
function externalHrefHost(
href: string | undefined,
pageUrl: string | undefined,
@@ -1234,6 +1269,119 @@ export function buildVomScene(
};
}
export function buildFrameVomScene(
documents: VomFrameDocument[],
captured: CapturedViewModel,
options: BuildVomSceneOptions = {},
): VomScene {
const documentByFrameId = new Map(documents.map((document) => [document.frameId, document]));
const rootFrameId =
captured.rootFrameId ?? documents.find((document) => !document.parentFrameId)?.frameId;
const orderedFrameIds = [
...(rootFrameId ? [rootFrameId] : []),
...documents.map((document) => document.frameId),
].filter((frameId, index, all) => all.indexOf(frameId) === index);
const localScenes = new Map<string, VomScene>();
for (const frameId of orderedFrameIds) {
const document = documentByFrameId.get(frameId);
if (!document) continue;
localScenes.set(
frameId,
buildVomScene(
document.axNodes,
{
viewport: captured.viewport,
nodes: document.domNodes,
iframeNodes: new Map(),
surfaceProbes: frameId === rootFrameId ? captured.surfaceProbes : [],
excludedBackendNodeIds:
frameId === rootFrameId ? captured.excludedBackendNodeIds : new Set(),
},
{ pageUrl: document.url ?? options.pageUrl },
),
);
}
const sceneIdByFrameAndBackend = new Map<string, number>();
let nextSceneId = 1;
for (const [frameId, scene] of localScenes) {
for (const node of scene.nodes) {
sceneIdByFrameAndBackend.set(`${frameId}:${node.id}`, nextSceneId);
nextSceneId += 1;
}
}
const nodes: VomNode[] = [];
const surfaces: CondSurface[] = [];
const activeScopeBlocks: ActiveScopeBlock[] = [];
for (const [frameId, scene] of localScenes) {
const document = documentByFrameId.get(frameId);
const ownerBackendNodeId = document?.ownerBackendNodeId;
const parentFrameId = document?.parentFrameId;
const ownerSceneId =
ownerBackendNodeId !== undefined && parentFrameId
? sceneIdByFrameAndBackend.get(`${parentFrameId}:${ownerBackendNodeId}`)
: undefined;
if (frameId !== rootFrameId && ownerSceneId === undefined) continue;
const suppressedRoots = new Set(
frameId === rootFrameId
? []
: scene.nodes
.filter(
(node) =>
node.parentId === null &&
["rootwebarea", "webarea"].includes(node.role?.toLowerCase() ?? ""),
)
.map((node) => node.id),
);
for (const node of scene.nodes) {
if (suppressedRoots.has(node.id)) continue;
const sceneNodeId = sceneIdByFrameAndBackend.get(`${frameId}:${node.id}`) as number;
let parentId =
node.parentId !== null
? sceneIdByFrameAndBackend.get(`${frameId}:${node.parentId}`)
: ownerSceneId;
if (node.parentId !== null && suppressedRoots.has(node.parentId)) parentId = ownerSceneId;
const remapId = (id: number) => sceneIdByFrameAndBackend.get(`${frameId}:${id}`);
nodes.push({
...node,
id: sceneNodeId,
backendNodeId: node.id,
frameId,
contextScopeId: document?.contextScopeId ?? frameId,
parentId: parentId ?? null,
...(node.domParentId !== undefined
? { domParentId: node.domParentId === null ? null : (remapId(node.domParentId) ?? null) }
: {}),
...(node.domAncestorIds
? { domAncestorIds: node.domAncestorIds.flatMap((id) => remapId(id) ?? []) }
: {}),
});
}
for (const surface of scene.surfaces ?? []) {
const triggerId = sceneIdByFrameAndBackend.get(`${frameId}:${surface.triggerId}`);
if (triggerId !== undefined) surfaces.push({ ...surface, triggerId });
}
for (const block of scene.activeScopeBlocks ?? []) {
const triggerId = sceneIdByFrameAndBackend.get(`${frameId}:${block.triggerId}`);
if (triggerId !== undefined) activeScopeBlocks.push({ ...block, triggerId });
}
}
return {
viewport: captured.viewport,
nodes,
...(rootFrameId ? { rootFrameId } : {}),
...(surfaces.length > 0 ? { surfaces } : {}),
...(activeScopeBlocks.length > 0 ? { activeScopeBlocks } : {}),
};
}
export interface SnapshotDeps {
cdp: CdpRunner;
tabsApi: {
@@ -1322,9 +1470,15 @@ export async function handleGetHtml(
if (params.ref) {
const resolved = resolveSnapshotRef(ctx, params.ref, target.tabId);
if (isRpcError(resolved)) return resolved;
const resp = await deps.cdp.send<{ outerHTML?: string }>(target.tabId, "DOM.getOuterHTML", {
backendNodeId: resolved.backendNodeId,
});
const resp = await sendToCdpTarget<{ outerHTML?: string }>(
deps.cdp,
{
tabId: target.tabId,
...(resolved.cdpSessionId ? { sessionId: resolved.cdpSessionId } : {}),
},
"DOM.getOuterHTML",
{ backendNodeId: resolved.backendNodeId },
);
throwIfAborted(signal, "get_html");
html = resp.outerHTML ?? "";
} else {
@@ -1447,15 +1601,6 @@ async function handleVomObservation(
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
await deps.cdp.ensureAttachedToUrl?.(target.tabId, target.url);
throwIfAborted(signal, toolName);
await deps.cdp.send<unknown>(target.tabId, "Accessibility.enable", {});
throwIfAborted(signal, toolName);
const result = await deps.cdp.send<{ nodes: CdpAxNode[] }>(
target.tabId,
"Accessibility.getFullAXTree",
{},
);
throwIfAborted(signal, toolName);
const axNodes = result.nodes ?? [];
const effectiveConditionalSurfaceProbe =
deps.conditionalSurfaceProbe ?? conditionalSurfaceProbe;
const captured = await captureForVom(
@@ -1466,17 +1611,39 @@ async function handleVomObservation(
signal,
);
throwIfAborted(signal, toolName);
const scene = buildVomScene(axNodes, captured, { pageUrl: target.url });
const frameDocuments = await captureFrameData<CdpAxNode>(
deps.cdp,
target.tabId,
captured,
signal,
);
throwIfAborted(signal, toolName);
const scene =
captured.rootFrameId && captured.frameNodes?.size
? buildFrameVomScene(frameDocuments, captured, { pageUrl: target.url })
: buildVomScene(frameDocuments[0]?.axNodes ?? [], captured, { pageUrl: target.url });
const rendered = renderVom(scene, {
maxDepth: params.max_depth,
maxTokens: params.max_tokens,
activeRegionPolicy: true,
});
throwIfAborted(signal, toolName);
const targetByFrameId = new Map(
frameDocuments.map((document) => [document.frameId, document.target]),
);
ctx.refStore.replace(
rendered.refs.map(
(r) => [r.ref, { backendNodeId: r.backendNodeId, tabId: target.tabId }] as const,
),
rendered.refs.map((ref) => {
const refTarget = ref.frameId ? targetByFrameId.get(ref.frameId) : undefined;
return [
ref.ref,
{
backendNodeId: ref.backendNodeId,
tabId: target.tabId,
...(ref.frameId ? { frameId: ref.frameId } : {}),
...(refTarget?.sessionId ? { cdpSessionId: refTarget.sessionId } : {}),
},
] as const;
}),
);
return attachDialogs(deps.cdp, target.tabId, dialogCursor, {
text: rendered.text,
+25 -2
View File
@@ -6,7 +6,8 @@
// exactly the same sandbox + visibility rules as the M6 observation
// handlers (review parity).
import type { DialogCursor } from "@/browser-driver/chromium-cdp";
import type { CdpDebuggee, DialogCursor } from "@/browser-driver/chromium-cdp";
import type { CdpFrameGraph, CdpTarget } from "@/browser-driver/frame-graph";
import type { SessionContext, SessionManager } from "@/session-manager/manager";
import { normaliseRef } from "@/session-manager/ref-store";
import type { ConsoleResult, JavaScriptDialogInfo, RpcError } from "@/transport/types";
@@ -51,9 +52,12 @@ export type { DialogCursor };
*/
export interface CdpRunner {
send<T = unknown>(tabId: number, method: string, params?: object): Promise<T>;
sendToTarget?<T = unknown>(target: CdpTarget, method: string, params?: object): Promise<T>;
detach?(tabId: number): Promise<void>;
getFrameGraph?(tabId: number): Promise<CdpFrameGraph>;
ensureAttachedToUrl?(tabId: number, expectedUrl: string | undefined): Promise<void>;
trackSessionTab?(sessionId: string, tabId: number): void;
onEvent?(handler: (source: chrome.debugger.Debuggee, method: string, params: unknown) => void): {
onEvent?(handler: (source: CdpDebuggee, method: string, params: unknown) => void): {
dispose(): void;
};
dialogCursor?(tabId: number): DialogCursor;
@@ -68,6 +72,25 @@ export interface CdpRunner {
): ConsoleResult;
}
export function sendToCdpTarget<T = unknown>(
cdp: CdpRunner,
target: CdpTarget,
method: string,
params?: object,
): Promise<T> {
if (target.sessionId && cdp.sendToTarget) {
return cdp.sendToTarget<T>(target, method, params);
}
return cdp.send<T>(target.tabId, method, params);
}
export function cdpRunnerForTarget(cdp: CdpRunner, target: CdpTarget): CdpRunner {
return {
send: <T = unknown>(_tabId: number, method: string, params?: object) =>
sendToCdpTarget<T>(cdp, target, method, params),
};
}
export interface BufferedReadBounds {
since: number | undefined;
limit: number;
+17 -5
View File
@@ -1,5 +1,5 @@
// Snapshot ref resolution — normalise `@eN` / `eN`, look up
// `backendNodeId` via the session RefStore, and build stable
// compound CDP node identity via the session RefStore, and build stable
// `ref_not_found` errors for hard-failure tool paths.
import type { SessionContext } from "@/session-manager/manager";
@@ -10,6 +10,13 @@ import { rpcError } from "./errors";
export interface SnapshotRefLookup {
backendNodeId: number;
refKey: string;
frameId?: string;
cdpSessionId?: string;
}
function refEntryForTab(ctx: SessionContext, refKey: string, tabId: number) {
const entry = ctx.refStore.resolveEntry(refKey);
return entry?.tabId === tabId ? entry : null;
}
/**
@@ -23,9 +30,14 @@ export function lookupSnapshotRef(
tabId: number,
): SnapshotRefLookup | null {
const refKey = normaliseRef(ref);
const backendNodeId = ctx.refStore.resolve(refKey, { tabId });
if (backendNodeId === null) return null;
return { backendNodeId, refKey };
const entry = refEntryForTab(ctx, refKey, tabId);
if (!entry) return null;
return {
backendNodeId: entry.backendNodeId,
refKey,
...(entry.frameId ? { frameId: entry.frameId } : {}),
...(entry.cdpSessionId ? { cdpSessionId: entry.cdpSessionId } : {}),
};
}
/**
@@ -37,7 +49,7 @@ export function resolveSnapshotRef(
ctx: SessionContext,
ref: string,
tabId: number,
): { backendNodeId: number; refKey: string } | RpcError {
): SnapshotRefLookup | RpcError {
const looked = lookupSnapshotRef(ctx, ref, tabId);
if (looked === null) {
return rpcError(
+59
View File
@@ -0,0 +1,59 @@
// Upload orchestration: validate the session-scoped request, resolve its click
// target, then delegate the browser protocol transaction to the file-input
// transaction module.
import type { SessionManager } from "@/session-manager/manager";
import type { RpcError, UploadParams, UploadResult } from "@/transport/types";
import { uploadThroughActivatedFileInput } from "./file-input-transaction";
import { clickResolvedTarget, type InteractionDeps, resolveActionTarget } from "./interaction";
import {
type CdpRunner,
enforceAgentWindow,
isRpcError,
lookupSession,
resolveTargetTab,
} from "./shared";
const DEFAULT_TIMEOUT_MS = 120_000;
export interface UploadDeps extends InteractionDeps {
cdp: CdpRunner;
}
export async function handleUpload(
manager: SessionManager,
params: UploadParams,
deps: UploadDeps,
): Promise<UploadResult | RpcError> {
const ctx = lookupSession(manager, params, "upload");
if (isRpcError(ctx)) return ctx;
const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "upload");
if (denied) return denied;
if (
params.files.length === 0 ||
params.files.length > 20 ||
params.files.some((file) => !file.staged_path)
) {
return { code: "invalid_params", message: "upload requires daemon-staged files" };
}
const address = await resolveActionTarget(deps.cdp, ctx, target, params, "upload");
if (isRpcError(address)) return address;
const timeoutMs = params.timeout_ms ?? DEFAULT_TIMEOUT_MS;
const transaction = await uploadThroughActivatedFileInput({
cdp: deps.cdp,
actionTarget: address,
files: params.files.map((file) => file.staged_path as string),
timeoutMs,
signal: deps.signal,
trigger: () => clickResolvedTarget(ctx, address, {}, deps),
});
if (isRpcError(transaction)) return transaction;
return {
tab_id: target.tabId,
used_ref: transaction.click.used_ref,
used_selector: transaction.click.used_selector,
file_names: params.files.map((file) => file.name),
};
}
@@ -18,6 +18,8 @@ function fakeSnapshotReply() {
"auto",
"type",
"password",
"value",
"hunter2",
];
const i = (s: string) => S.indexOf(s);
return {
@@ -29,7 +31,8 @@ function fakeSnapshotReply() {
nodeType: [1, 1, 1, 1],
nodeName: [i("html"), i("body"), i("div"), i("input")],
backendNodeId: [10, 11, 12, 13],
attributes: [[], [], [], [i("type"), i("password")]],
attributes: [[], [], [], [i("type"), i("password"), i("value"), i("hunter2")]],
inputValue: { index: [3], value: [i("hunter2")] },
},
layout: {
nodeIndex: [1, 2, 3],
@@ -217,7 +220,8 @@ function makeCdp(snapshot: unknown) {
{
state: "filled",
sensitive: true,
defaultValue: "",
defaultValue: "hunter2",
value: "hunter2",
placeholder: "secret",
},
],
@@ -262,10 +266,11 @@ describe("captureViewModel", () => {
expect(input).toMatchObject({
formState: "filled",
formDefaultValue: "",
formPlaceholder: "secret",
attrs: { type: "password" },
});
expect(input?.formValue).toBeUndefined();
expect(input?.formDefaultValue).toBeUndefined();
expect(cdp.send).toHaveBeenCalledWith(
4,
"Runtime.evaluate",
@@ -692,9 +697,7 @@ describe("captureViewModel", () => {
};
const { nodes } = await captureViewModel(cdp, 4);
const div = nodes.find((n) => n.backendNodeId === 12);
// local rect preserves viewport-relative scroll math.
expect(div?.localRect?.y).toBe(-200);
// exported rect is clipped to the top-level viewport before VOM consumes it.
expect(div?.rect).toMatchObject({ y: 0, h: 600 });
});
@@ -839,6 +842,7 @@ describe("captureViewModel", () => {
strings: S,
documents: [
{
frameId: "main-frame",
scrollOffsetX: 0,
scrollOffsetY: 0,
nodes: {
@@ -866,6 +870,7 @@ describe("captureViewModel", () => {
},
// documents[1]: the iframe's sub-document with a login input
{
frameId: "child-frame",
scrollOffsetX: 0,
scrollOffsetY: 0,
nodes: {
@@ -895,7 +900,14 @@ describe("captureViewModel", () => {
throw new Error(method);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const { nodes, iframeNodes } = await captureViewModel(cdp, 4);
const {
nodes,
iframeNodes,
frameNodes,
frameOwnerBackendNodeIds,
frameParentIds,
rootFrameId,
} = await captureViewModel(cdp, 4);
// Main frame has 4 nodes; iframe is included
expect(nodes.find((n) => n.backendNodeId === 13)?.tag).toBe("iframe");
// iframeNodes keyed by the <iframe> element's backendNodeId=13
@@ -908,9 +920,13 @@ describe("captureViewModel", () => {
expect(input?.ownerFrameBackendNodeId).toBe(13);
expect(input?.localRect).toEqual({ x: 0, y: 0, w: 200, h: 40 });
expect(input?.rect).toEqual({ x: 100, y: 300, w: 200, h: 40 });
expect(rootFrameId).toBe("main-frame");
expect(frameNodes?.get("child-frame")).toBe(subNodes);
expect(frameOwnerBackendNodeIds?.get("child-frame")).toBe(13);
expect(frameParentIds?.get("child-frame")).toBe("main-frame");
});
it("clips iframe sub-document rects to the iframe viewport in top coordinates", async () => {
it("captures iframe documents without requiring owner-frame geometry", async () => {
const S = [
"html",
"body",
@@ -938,16 +954,10 @@ describe("captureViewModel", () => {
contentDocumentIndex: { index: [2], value: [1] },
},
layout: {
nodeIndex: [1, 2],
styles: [
[i("static"), i("auto")],
[i("static"), i("auto")],
],
bounds: [
[0, 0, 1000, 800],
[950, 760, 200, 100],
],
paintOrders: [0, 1],
nodeIndex: [1],
styles: [[i("static"), i("auto")]],
bounds: [[0, 0, 1000, 800]],
paintOrders: [0],
},
},
{
@@ -985,7 +995,7 @@ describe("captureViewModel", () => {
const input = iframeNodes.get(13)?.find((n) => n.backendNodeId === 101);
expect(input?.localRect).toEqual({ x: 20, y: 20, w: 120, h: 60 });
expect(input?.rect).toEqual({ x: 970, y: 780, w: 30, h: 20 });
expect(input?.rect).toBeNull();
});
it("recursively normalizes nested iframe sub-documents", async () => {
@@ -1080,8 +1090,10 @@ describe("captureViewModel", () => {
const nestedIframe = iframeNodes.get(13)?.find((n) => n.backendNodeId === 23);
const input = iframeNodes.get(23)?.find((n) => n.backendNodeId === 31);
expect(nestedIframe?.localRect).toEqual({ x: 5, y: 6, w: 100, h: 80 });
expect(nestedIframe?.rect).toEqual({ x: 15, y: 26, w: 100, h: 80 });
expect(input?.ownerFrameBackendNodeId).toBe(23);
expect(input?.localRect).toEqual({ x: 1, y: 2, w: 40, h: 20 });
expect(input?.rect).toEqual({ x: 16, y: 28, w: 40, h: 20 });
});
@@ -0,0 +1,135 @@
import { describe, expect, it, vi } from "vitest";
import type { CdpRunner } from "../../shared";
import type { CapturedNode, CapturedViewModel } from "../capture";
import { captureFrameData } from "../frame-capture";
function ownerNode(backendNodeId: number, x: number): CapturedNode {
return {
backendNodeId,
parentBackendNodeId: 1,
frameId: "main",
ownerFrameBackendNodeId: null,
tag: "iframe",
attrs: {},
rect: { x, y: 100, w: 300, h: 200 },
localRect: { x, y: 100, w: 300, h: 200 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
};
}
function childSnapshot(frameId: string, backendNodeId: number) {
const strings = [frameId, "body", "button", "static", "auto", "pointer"];
return {
strings,
documents: [
{
frameId,
nodes: {
parentIndex: [-1, 0],
nodeName: [1, 2],
backendNodeId: [backendNodeId - 1, backendNodeId],
attributes: [[], []],
},
layout: {
nodeIndex: [0, 1],
styles: [
[3, 4, 4],
[3, 4, 5],
],
bounds: [
[0, 0, 300, 200],
[10, 20, 100, 40],
],
paintOrders: [0, 1],
},
},
],
};
}
describe("captureFrameData", () => {
it("captures and positions multiple OOPIF documents missing from the root snapshot", async () => {
const leftOwner = ownerNode(10, 50);
const rightOwner = ownerNode(20, 500);
const captured: CapturedViewModel = {
nodes: [leftOwner, rightOwner],
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
frameNodes: new Map([["main", [leftOwner, rightOwner]]]),
frameOwnerBackendNodeIds: new Map(),
rootFrameId: "main",
excludedBackendNodeIds: new Set(),
};
const sendToTarget = vi.fn(async (target, method) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 300, clientHeight: 200, pageX: 0, pageY: 0 } };
}
if (method === "DOMSnapshot.enable" || method === "Accessibility.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") {
return target.sessionId === "left-session"
? childSnapshot("left", 101)
: childSnapshot("right", 201);
}
if (method === "Accessibility.getFullAXTree") return { nodes: [] };
throw new Error(`unexpected ${method}`);
});
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method, params) => {
if (method === "Accessibility.enable") return {};
if (method === "Accessibility.getFullAXTree") return { nodes: [] };
if (method === "DOM.getBoxModel") {
const backendNodeId = (params as { backendNodeId?: number })?.backendNodeId;
const x = backendNodeId === 10 ? 50 : 500;
return { model: { content: [x, 100, x + 300, 100, x + 300, 300, x, 300] } };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 800 } };
}
throw new Error(`unexpected root ${method}`);
}) as CdpRunner["send"],
sendToTarget: sendToTarget as unknown as NonNullable<CdpRunner["sendToTarget"]>,
getFrameGraph: vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "left",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4, sessionId: "left-session" },
},
{
frameId: "right",
parentFrameId: "main",
ownerBackendNodeId: 20,
target: { tabId: 4, sessionId: "right-session" },
},
],
})),
};
const trees = await captureFrameData(cdp, 4, captured);
expect(trees.map((tree) => tree.frameId)).toEqual(["main", "left", "right"]);
expect(
captured.frameNodes?.get("left")?.find((node) => node.backendNodeId === 101)?.rect,
).toEqual({ x: 60, y: 120, w: 100, h: 40 });
expect(
captured.frameNodes?.get("right")?.find((node) => node.backendNodeId === 201)?.rect,
).toEqual({ x: 510, y: 120, w: 100, h: 40 });
expect(captured.frameOwnerBackendNodeIds).toEqual(
new Map([
["left", 10],
["right", 20],
]),
);
expect(captured.frameParentIds).toEqual(
new Map([
["left", "main"],
["right", "main"],
]),
);
});
});
@@ -0,0 +1,174 @@
import { describe, expect, it } from "vitest";
import type { CdpFrameGraph } from "@/browser-driver/frame-graph";
import type { CapturedNode, CapturedViewModel } from "../capture";
import { buildFrameDocuments, type FrameAxBatch, type FrameOwnedAxNode } from "../frame-document";
function domNode(frameId: string, backendNodeId: number): CapturedNode {
return {
backendNodeId,
parentBackendNodeId: null,
frameId,
tag: "div",
attrs: {},
rect: { x: 0, y: 0, w: 20, h: 20 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
};
}
function captured(frameNodes: Map<string, CapturedNode[]>): CapturedViewModel {
return {
nodes: frameNodes.get("main") ?? [],
viewport: { width: 800, height: 600 },
iframeNodes: new Map(),
frameNodes,
rootFrameId: "main",
excludedBackendNodeIds: new Set(),
};
}
describe("buildFrameDocuments", () => {
it("partitions mixed AX results by node ownership and cuts cross-frame parent edges", () => {
const graph: CdpFrameGraph = {
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{ frameId: "child", parentFrameId: "main", target: { tabId: 4 } },
],
};
const batches: FrameAxBatch<FrameOwnedAxNode>[] = [
{
frame: graph.frames[0],
nodes: [
{ nodeId: "root", frameId: "main", backendDOMNodeId: 1, childIds: ["h"] },
{
nodeId: "h",
frameId: "child",
parentId: "root",
backendDOMNodeId: 100,
childIds: ["virtual"],
},
{ nodeId: "virtual", parentId: "h" },
],
},
];
const documents = buildFrameDocuments(
graph,
batches,
captured(
new Map([
["main", [domNode("main", 1)]],
["child", [domNode("child", 100)]],
]),
),
);
expect(documents.find((document) => document.frameId === "main")?.axNodes).toHaveLength(1);
const childNodes = documents.find((document) => document.frameId === "child")?.axNodes ?? [];
expect(childNodes.map((node) => node.nodeId)).toEqual(["h", "virtual"]);
expect(childNodes.find((node) => node.nodeId === "h")?.parentId).toBeUndefined();
expect(childNodes.find((node) => node.nodeId === "virtual")?.frameId).toBe("child");
});
it("deduplicates overlapping same-target AX results using the strongest ownership", () => {
const graph: CdpFrameGraph = {
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{ frameId: "child", parentFrameId: "main", target: { tabId: 4 } },
],
};
const childNode = { nodeId: "shared", backendDOMNodeId: 100 };
const batches: FrameAxBatch<FrameOwnedAxNode>[] = [
{
frame: graph.frames[0],
nodes: [{ ...childNode, frameId: "child" }],
},
{
frame: graph.frames[1],
nodes: [childNode],
},
];
const documents = buildFrameDocuments(
graph,
batches,
captured(
new Map([
["main", [domNode("main", 1)]],
["child", [domNode("child", 100)]],
]),
),
);
expect(documents.find((document) => document.frameId === "main")?.axNodes).toEqual([]);
expect(documents.find((document) => document.frameId === "child")?.axNodes).toHaveLength(1);
});
it("uses target-scoped backend ownership when OOPIF backend ids collide", () => {
const graph: CdpFrameGraph = {
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "left",
parentFrameId: "main",
target: { tabId: 4, sessionId: "left-session" },
},
{
frameId: "right",
parentFrameId: "main",
target: { tabId: 4, sessionId: "right-session" },
},
],
};
const batches: FrameAxBatch<FrameOwnedAxNode>[] = graph.frames.slice(1).map((frame) => ({
frame,
nodes: [{ nodeId: `${frame.frameId}-node`, backendDOMNodeId: 7 }],
}));
const documents = buildFrameDocuments(
graph,
batches,
captured(
new Map([
["main", []],
["left", [domNode("left", 7)]],
["right", [domNode("right", 7)]],
]),
),
);
expect(documents.find((document) => document.frameId === "left")?.axNodes[0]?.nodeId).toBe(
"left-node",
);
expect(documents.find((document) => document.frameId === "right")?.axNodes[0]?.nodeId).toBe(
"right-node",
);
});
it("uses captured frame parent identity instead of guessing from backend ids", () => {
const frameNodes = new Map([
["main", [domNode("main", 1)]],
["left", [domNode("left", 7)]],
["right", [domNode("right", 7)]],
["child", [domNode("child", 9)]],
]);
const capture = captured(frameNodes);
capture.frameOwnerBackendNodeIds = new Map([["child", 7]]);
capture.frameParentIds = new Map([["child", "left"]]);
const frames = ["main", "left", "right", "child"].map((frameId) => ({
frame: { frameId, target: { tabId: 4 } },
nodes: [],
}));
const documents = buildFrameDocuments(null, frames, capture);
expect(documents.find((document) => document.frameId === "child")).toMatchObject({
parentFrameId: "left",
ownerBackendNodeId: 7,
});
});
});
+138 -56
View File
@@ -7,6 +7,7 @@
import type { Rect, Viewport } from "@browser-skill/vom";
import { evaluateHoverTrigger } from "@/lib/hover-trigger-policy";
import { isOverlayHostNode, OVERLAY_HOST_SELECTOR } from "../../lib/overlay-bridge";
import { childFrameProjection, type GeometryProjection, projectRectToViewport } from "../geometry";
import type { CdpRunner } from "../shared";
const REQUESTED_STYLES = ["position", "pointer-events", "cursor"] as const;
@@ -17,13 +18,14 @@ const STYLE_COL = Object.fromEntries(
export interface CapturedNode {
backendNodeId: number;
parentBackendNodeId: number | null;
frameId?: string;
/** Owning iframe backend node id; `null` for the top-level document. */
ownerFrameBackendNodeId?: number | null;
tag: string;
attrs: Record<string, string>;
/** Top-level viewport-relative CSS px, clipped to the owning frame viewport. */
rect: Rect | null;
/** Frame-local viewport-relative CSS px before top-level translation/clipping. */
/** Frame-local viewport-relative CSS px before top-level projection. */
localRect?: Rect | null;
paintOrder: number;
position: string;
@@ -57,6 +59,11 @@ export interface CapturedViewModel {
nodes: CapturedNode[];
viewport: Viewport;
iframeNodes: CapturedIframeNodes;
frameNodes?: Map<string, CapturedNode[]>;
frameOwnerBackendNodeIds?: Map<string, number>;
/** Explicit DOMSnapshot frame ancestry; never inferred from backend node ids. */
frameParentIds?: Map<string, string>;
rootFrameId?: string;
surfaceProbes?: CapturedSurfaceProbe[];
/** Backend node ids belonging to the agent overlay host + its shadow subtree. */
excludedBackendNodeIds: Set<number>;
@@ -92,7 +99,12 @@ interface SparseArray {
value: number[];
}
interface RareBooleanData {
index: number[];
}
interface SnapshotDocument {
frameId?: string | number;
scrollOffsetX?: number;
scrollOffsetY?: number;
nodes?: {
@@ -107,6 +119,10 @@ interface SnapshotDocument {
nodeValue?: number[];
/** Maps node array index → index into `documents[]` for frame content. */
contentDocumentIndex?: SparseArray;
inputValue?: SparseArray;
textValue?: SparseArray;
inputChecked?: RareBooleanData;
optionSelected?: RareBooleanData;
};
layout?: {
nodeIndex?: number[];
@@ -116,6 +132,12 @@ interface SnapshotDocument {
};
}
function snapshotFrameId(document: SnapshotDocument, strings: string[]): string | undefined {
if (typeof document.frameId === "string") return document.frameId || undefined;
if (typeof document.frameId === "number") return str(strings, document.frameId) || undefined;
return undefined;
}
interface SnapshotReply {
strings?: string[];
documents?: SnapshotDocument[];
@@ -135,6 +157,25 @@ function isFormControlTag(tag: string): boolean {
return tag === "input" || tag === "textarea" || tag === "select";
}
function isSensitiveFormControl(node: Pick<CapturedNode, "tag" | "attrs">): boolean {
return node.tag === "input" && (node.attrs.type ?? "").toLowerCase() === "password";
}
function snapshotFormState(
value: string | undefined,
defaultValue: string,
hasDefaultValue: boolean,
sensitive: boolean,
): CapturedNode["formState"] {
if (sensitive) {
if (value === undefined && !hasDefaultValue) return undefined;
return (value ?? defaultValue) === "" ? "empty" : "filled";
}
if (value === undefined) return undefined;
if (value === "") return "empty";
return value === defaultValue ? "default" : "filled";
}
const MAX_FORM_ENRICH_CONTROLS = 250;
interface CapturedFormState {
@@ -166,9 +207,8 @@ function formStateBatchExpression(maxControls: number): string {
return {
state,
sensitive,
defaultValue,
placeholder,
...(sensitive ? {} : { value: rawValue }),
...(sensitive ? {} : { value: rawValue, defaultValue }),
};
};
const collect = (doc) => {
@@ -210,10 +250,14 @@ function applyFormStates(nodes: CapturedNode[], states: CapturedFormState[]): vo
index += 1;
if (!state) continue;
node.formState = state.state;
node.formDefaultValue = state.defaultValue ?? "";
node.formPlaceholder = state.placeholder ?? "";
if (!state.sensitive && state.value !== undefined) {
node.formValue = state.value;
if (state.sensitive || isSensitiveFormControl(node)) {
delete node.formValue;
delete node.formDefaultValue;
delete node.attrs.value;
} else {
node.formDefaultValue = state.defaultValue ?? "";
if (state.value !== undefined) node.formValue = state.value;
}
}
}
@@ -272,9 +316,9 @@ interface ParseDocumentResult {
}
interface FrameContext {
frameId?: string;
ownerFrameBackendNodeId: number | null;
originInTop: { x: number; y: number };
clipRectInTop: Rect;
projection: GeometryProjection | null;
scrollX: number;
scrollY: number;
}
@@ -293,38 +337,6 @@ function devicePixelRatio(metrics: LayoutMetricsReply): number {
return dpr >= 1 ? dpr : 1;
}
function viewportRect(viewport: Viewport): Rect {
return { x: 0, y: 0, w: Math.max(0, viewport.width), h: Math.max(0, viewport.height) };
}
function intersectRects(a: Rect, b: Rect): Rect | null {
const x1 = Math.max(a.x, b.x);
const y1 = Math.max(a.y, b.y);
const x2 = Math.min(a.x + a.w, b.x + b.w);
const y2 = Math.min(a.y + a.h, b.y + b.h);
if (x2 <= x1 || y2 <= y1) return null;
return { x: x1, y: y1, w: x2 - x1, h: y2 - y1 };
}
function rectInTop(localRect: Rect, context: FrameContext): Rect | null {
const translated = {
x: context.originInTop.x + localRect.x,
y: context.originInTop.y + localRect.y,
w: localRect.w,
h: localRect.h,
};
return intersectRects(translated, context.clipRectInTop);
}
function unclipRectInTop(localRect: Rect, context: FrameContext): Rect {
return {
x: context.originInTop.x + localRect.x,
y: context.originInTop.y + localRect.y,
w: localRect.w,
h: localRect.h,
};
}
function sparseIndexMap(sparse: SparseArray | undefined): Map<number, number> {
const out = new Map<number, number>();
if (!sparse?.index || !sparse.value) return out;
@@ -805,6 +817,10 @@ function parseDocumentNodes(
const posCol = STYLE_COL.position;
const peCol = STYLE_COL["pointer-events"];
const cursorCol = STYLE_COL.cursor;
const inputValues = sparseIndexMap(dn.inputValue);
const textValues = sparseIndexMap(dn.textValue);
const checkedInputs = new Set(dn.inputChecked?.index ?? []);
const selectedOptions = new Set(dn.optionSelected?.index ?? []);
// Collect visible text from #text child nodes so element CapturedNodes
// carry a textContent value usable as a button/link label fallback.
@@ -855,7 +871,10 @@ function parseDocumentNodes(
w: b[2] / dpr,
h: b[3] / dpr,
};
rect = rectInTop(localRect, context);
const bounds = context.projection
? projectRectToViewport(localRect, context.projection)
: null;
rect = bounds ? { x: bounds.x, y: bounds.y, w: bounds.width, h: bounds.height } : null;
}
paintOrder = dl?.paintOrders?.[li] ?? 0;
const styleRow = dl?.styles?.[li] ?? [];
@@ -870,9 +889,24 @@ function parseDocumentNodes(
const textContent = nodeTextContent.get(n);
const rawFormValueIndex = tag === "textarea" ? textValues.get(n) : inputValues.get(n);
const rawFormValue =
rawFormValueIndex !== undefined ? str(strings, rawFormValueIndex) : undefined;
const sensitive = isSensitiveFormControl({ tag, attrs });
const formDefaultValue = attrs.value ?? "";
const formValue = sensitive ? undefined : rawFormValue;
const formState = snapshotFormState(
rawFormValue,
formDefaultValue,
Object.prototype.hasOwnProperty.call(attrs, "value"),
sensitive,
);
if (sensitive) delete attrs.value;
nodes.push({
backendNodeId,
parentBackendNodeId,
...(context.frameId ? { frameId: context.frameId } : {}),
ownerFrameBackendNodeId: context.ownerFrameBackendNodeId,
tag,
attrs,
@@ -883,6 +917,16 @@ function parseDocumentNodes(
pointerEvents,
cursor,
textContent,
...(formValue !== undefined ? { formValue } : {}),
...(tag === "input" || tag === "textarea"
? {
formPlaceholder: attrs.placeholder ?? "",
...(!sensitive ? { formDefaultValue } : {}),
...(formState ? { formState } : {}),
}
: {}),
...(checkedInputs.has(n) ? { formValue: "true", formState: "filled" } : {}),
...(selectedOptions.has(n) ? { formValue: attrs.value ?? textContent ?? "" } : {}),
});
}
return { nodes, excludedBackendNodeIds };
@@ -890,6 +934,8 @@ function parseDocumentNodes(
interface ParseFrameDocumentsResult {
iframeNodes: CapturedIframeNodes;
frameOwnerBackendNodeIds: Map<string, number>;
frameParentIds: Map<string, string>;
excludedBackendNodeIds: Set<number>;
}
@@ -903,10 +949,14 @@ function parseChildFrameDocuments(
visited = new Set<number>(),
): ParseFrameDocumentsResult {
const iframeNodes: CapturedIframeNodes = new Map();
const frameOwnerBackendNodeIds = new Map<string, number>();
const frameParentIds = new Map<string, string>();
const excludedBackendNodeIds = new Set<number>();
const parentDoc = documents[parentDocIndex];
const cdi = sparseIndexMap(parentDoc?.nodes?.contentDocumentIndex);
if (cdi.size === 0) return { iframeNodes, excludedBackendNodeIds };
if (cdi.size === 0) {
return { iframeNodes, frameOwnerBackendNodeIds, frameParentIds, excludedBackendNodeIds };
}
const parentBackendIds = parentDoc?.nodes?.backendNodeId ?? [];
const parentNodeByBackendId = new Map(parentNodes.map((node) => [node.backendNodeId, node]));
@@ -918,19 +968,15 @@ function parseChildFrameDocuments(
const iframeBackendId = parentBackendIds[nodeArrayIdx];
if (iframeBackendId === undefined) continue;
const iframeNode = parentNodeByBackendId.get(iframeBackendId);
if (!iframeNode?.localRect) continue;
const iframeRectInTop = unclipRectInTop(iframeNode.localRect, parentContext);
const childClip = intersectRects(iframeRectInTop, parentContext.clipRectInTop);
if (!childClip) {
iframeNodes.set(iframeBackendId, []);
continue;
}
const projection =
parentContext.projection && iframeNode?.localRect
? childFrameProjection(parentContext.projection, iframeNode.localRect)
: null;
const childContext: FrameContext = {
frameId: snapshotFrameId(childDoc, strings),
ownerFrameBackendNodeId: iframeBackendId,
originInTop: { x: iframeRectInTop.x, y: iframeRectInTop.y },
clipRectInTop: childClip,
projection,
scrollX: childDoc.scrollOffsetX ?? 0,
scrollY: childDoc.scrollOffsetY ?? 0,
};
@@ -938,6 +984,10 @@ function parseChildFrameDocuments(
nextVisited.add(childDocIndex);
const parsed = parseDocumentNodes(childDoc, strings, dpr, childContext);
iframeNodes.set(iframeBackendId, parsed.nodes);
if (childContext.frameId) {
frameOwnerBackendNodeIds.set(childContext.frameId, iframeBackendId);
if (parentContext.frameId) frameParentIds.set(childContext.frameId, parentContext.frameId);
}
for (const id of parsed.excludedBackendNodeIds) excludedBackendNodeIds.add(id);
const nested = parseChildFrameDocuments(
@@ -952,10 +1002,16 @@ function parseChildFrameDocuments(
for (const [nestedFrameId, nestedNodes] of nested.iframeNodes) {
iframeNodes.set(nestedFrameId, nestedNodes);
}
for (const [frameId, ownerBackendNodeId] of nested.frameOwnerBackendNodeIds) {
frameOwnerBackendNodeIds.set(frameId, ownerBackendNodeId);
}
for (const [frameId, parentFrameId] of nested.frameParentIds) {
frameParentIds.set(frameId, parentFrameId);
}
for (const id of nested.excludedBackendNodeIds) excludedBackendNodeIds.add(id);
}
return { iframeNodes, excludedBackendNodeIds };
return { iframeNodes, frameOwnerBackendNodeIds, frameParentIds, excludedBackendNodeIds };
}
export async function captureViewModel(
@@ -998,9 +1054,13 @@ export async function captureViewModel(
}
const topContext: FrameContext = {
frameId: snapshotFrameId(doc0, strings),
ownerFrameBackendNodeId: null,
originInTop: { x: 0, y: 0 },
clipRectInTop: viewportRect(viewport),
projection: {
sourceClips: [],
edges: [],
topViewport: viewport,
},
scrollX,
scrollY,
};
@@ -1022,5 +1082,27 @@ export async function captureViewModel(
: [];
throwIfAborted(options.signal);
return { nodes, viewport, iframeNodes, surfaceProbes, excludedBackendNodeIds };
const frameNodes = new Map<string, CapturedNode[]>();
if (topContext.frameId) frameNodes.set(topContext.frameId, nodes);
for (const iframeFrameNodes of iframeNodes.values()) {
const frameId = iframeFrameNodes.find((node) => node.frameId)?.frameId;
if (frameId) frameNodes.set(frameId, iframeFrameNodes);
}
for (const [frameId, ownerBackendNodeId] of frameParsed.frameOwnerBackendNodeIds) {
if (!frameNodes.has(frameId)) {
frameNodes.set(frameId, iframeNodes.get(ownerBackendNodeId) ?? []);
}
}
return {
nodes,
viewport,
iframeNodes,
frameNodes,
frameOwnerBackendNodeIds: frameParsed.frameOwnerBackendNodeIds,
frameParentIds: frameParsed.frameParentIds,
...(topContext.frameId ? { rootFrameId: topContext.frameId } : {}),
surfaceProbes,
excludedBackendNodeIds,
};
}
@@ -0,0 +1,179 @@
import type { CdpFrame, CdpFrameGraph } from "@/browser-driver/frame-graph";
import { resolveFrameProjection } from "../frame-geometry";
import { type GeometryProjection, projectRectToViewport } from "../geometry";
import { type CdpRunner, cdpRunnerForTarget, sendToCdpTarget } from "../shared";
import { type CapturedNode, type CapturedViewModel, captureViewModel } from "./capture";
import {
buildFrameDocuments,
type FrameAxBatch,
type FrameDocument,
type FrameOwnedAxNode,
} from "./frame-document";
export type FrameAxNode = FrameOwnedAxNode;
export type CapturedFrameDocument<T extends FrameAxNode> = FrameDocument<T>;
function throwIfAborted(signal?: AbortSignal): void {
if (signal?.aborted) throw new DOMException("observation aborted", "AbortError");
}
function isAbortError(err: unknown): boolean {
return err instanceof Error && err.name === "AbortError";
}
async function discoverFrameGraph(cdp: CdpRunner, tabId: number): Promise<CdpFrameGraph | null> {
if (!cdp.getFrameGraph) return null;
try {
return await cdp.getFrameGraph(tabId);
} catch (err) {
console.debug("[bsk observation] frame graph capture failed", err);
return null;
}
}
function transformFrameNodes(
nodes: CapturedNode[],
projection: GeometryProjection,
): CapturedNode[] {
return nodes.map((node) => ({
...node,
rect: (() => {
const bounds = projectRectToViewport(node.rect, projection);
return bounds ? { x: bounds.x, y: bounds.y, w: bounds.width, h: bounds.height } : null;
})(),
}));
}
function frameOwnerNode(captured: CapturedViewModel, frame: CdpFrame): CapturedNode | undefined {
if (frame.ownerBackendNodeId === undefined) return undefined;
const parentNodes = frame.parentFrameId
? captured.frameNodes?.get(frame.parentFrameId)
: captured.nodes;
return parentNodes?.find((node) => node.backendNodeId === frame.ownerBackendNodeId);
}
async function captureMissingFrameDocuments(
cdp: CdpRunner,
tabId: number,
graph: CdpFrameGraph,
captured: CapturedViewModel,
signal?: AbortSignal,
): Promise<void> {
if (!captured.frameNodes) captured.frameNodes = new Map();
if (!captured.frameOwnerBackendNodeIds) captured.frameOwnerBackendNodeIds = new Map();
if (!captured.frameParentIds) captured.frameParentIds = new Map();
for (const frame of graph.frames) {
if (!frame.target.sessionId || captured.frameNodes.has(frame.frameId)) continue;
const owner = frameOwnerNode(captured, frame);
if (!owner?.rect || frame.ownerBackendNodeId === undefined) continue;
try {
const child = await captureViewModel(cdpRunnerForTarget(cdp, frame.target), tabId, {
signal,
});
const projection = await resolveFrameProjection(cdp, graph, frame.frameId);
if (!projection) continue;
const rootFrameId = child.rootFrameId ?? frame.frameId;
const childFrames = child.frameNodes ?? new Map<string, CapturedNode[]>();
if (!childFrames.has(rootFrameId)) childFrames.set(rootFrameId, child.nodes);
if (rootFrameId !== frame.frameId && !childFrames.has(frame.frameId)) {
childFrames.set(
frame.frameId,
child.nodes.map((node) => ({ ...node, frameId: frame.frameId })),
);
}
for (const [childFrameId, nodes] of childFrames) {
captured.frameNodes.set(childFrameId, transformFrameNodes(nodes, projection));
}
captured.iframeNodes.set(
frame.ownerBackendNodeId,
captured.frameNodes.get(frame.frameId) ?? [],
);
captured.frameOwnerBackendNodeIds.set(frame.frameId, frame.ownerBackendNodeId);
if (frame.parentFrameId) captured.frameParentIds.set(frame.frameId, frame.parentFrameId);
for (const [childFrameId, ownerBackendNodeId] of child.frameOwnerBackendNodeIds ?? []) {
captured.frameOwnerBackendNodeIds.set(childFrameId, ownerBackendNodeId);
}
for (const [childFrameId, parentFrameId] of child.frameParentIds ?? []) {
captured.frameParentIds.set(childFrameId, parentFrameId);
}
} catch (err) {
if (isAbortError(err)) throw err;
console.debug("[bsk observation] child frame DOM capture failed", {
frameId: frame.frameId,
err,
});
}
}
}
async function captureAxTrees<T extends FrameAxNode>(
cdp: CdpRunner,
tabId: number,
discoveredFrames: CdpFrame[],
captured: CapturedViewModel,
signal?: AbortSignal,
): Promise<FrameAxBatch<T>[]> {
let frames = [...discoveredFrames];
const capturedFrameIds = [...(captured.frameNodes?.keys() ?? [])];
if (frames.length === 0) {
const rootFrameId = captured.rootFrameId ?? capturedFrameIds[0] ?? "root";
frames = [{ frameId: rootFrameId, target: { tabId } }];
}
for (const frameId of capturedFrameIds) {
if (!frames.some((frame) => frame.frameId === frameId)) {
frames.push({ frameId, target: { tabId } });
}
}
let firstFailure: unknown;
const trees = await Promise.all(
frames.map(async (frame): Promise<FrameAxBatch<T>> => {
try {
throwIfAborted(signal);
await sendToCdpTarget(cdp, frame.target, "Accessibility.enable", {});
const result = await sendToCdpTarget<{ nodes?: T[] }>(
cdp,
frame.target,
"Accessibility.getFullAXTree",
frame.frameId === "root" ? {} : { frameId: frame.frameId },
);
throwIfAborted(signal);
return {
frame,
nodes: result.nodes ?? [],
};
} catch (err) {
if (isAbortError(err)) throw err;
firstFailure ??= err;
console.debug("[bsk observation] frame AX capture failed", {
frameId: frame.frameId,
err,
});
return { frame, nodes: [] };
}
}),
);
const capturedNodeCount = [...captured.iframeNodes.values()].reduce(
(count, nodes) => count + nodes.length,
captured.nodes.length,
);
if (firstFailure && capturedNodeCount === 0 && trees.every((tree) => tree.nodes.length === 0)) {
throw firstFailure;
}
return trees;
}
export async function captureFrameData<T extends FrameAxNode>(
cdp: CdpRunner,
tabId: number,
captured: CapturedViewModel,
signal?: AbortSignal,
): Promise<CapturedFrameDocument<T>[]> {
const graph = await discoverFrameGraph(cdp, tabId);
const frames = graph?.frames ?? [];
if (graph) await captureMissingFrameDocuments(cdp, tabId, graph, captured, signal);
throwIfAborted(signal);
const batches = await captureAxTrees<T>(cdp, tabId, frames, captured, signal);
return buildFrameDocuments(graph, batches, captured);
}
@@ -0,0 +1,170 @@
import {
type CdpFrame,
type CdpFrameGraph,
type CdpTarget,
cdpTargetKey,
} from "@/browser-driver/frame-graph";
import type { CapturedNode, CapturedViewModel } from "./capture";
export interface FrameOwnedAxNode {
nodeId: string;
frameId?: string;
parentId?: string;
childIds?: string[];
backendDOMNodeId?: number;
}
export interface FrameAxBatch<T extends FrameOwnedAxNode> {
frame: CdpFrame;
nodes: T[];
}
export interface FrameDocument<T extends FrameOwnedAxNode> extends CdpFrame {
contextScopeId: string;
axNodes: T[];
domNodes: CapturedNode[];
}
interface Ownership {
frameId: string;
strength: number;
}
interface OwnedCandidate<T> {
node: T;
target: CdpTarget;
ownership: Ownership;
}
function targetNodeKey(target: CdpTarget, nodeId: string): string {
return `${cdpTargetKey(target)}:${nodeId}`;
}
function targetBackendKey(target: CdpTarget, backendNodeId: number): string {
return `${cdpTargetKey(target)}:${backendNodeId}`;
}
function frameList<T extends FrameOwnedAxNode>(
graph: CdpFrameGraph | null,
batches: FrameAxBatch<T>[],
captured: CapturedViewModel,
): CdpFrame[] {
const frames = new Map<string, CdpFrame>();
for (const frame of graph?.frames ?? []) frames.set(frame.frameId, frame);
for (const batch of batches) {
if (!frames.has(batch.frame.frameId)) frames.set(batch.frame.frameId, batch.frame);
}
for (const frameId of captured.frameNodes?.keys() ?? []) {
if (!frames.has(frameId))
frames.set(frameId, { frameId, target: { tabId: batches[0]?.frame.target.tabId ?? 0 } });
}
return [...frames.values()].map((frame) => {
const ownerBackendNodeId =
frame.ownerBackendNodeId ?? captured.frameOwnerBackendNodeIds?.get(frame.frameId);
const parentFrameId = frame.parentFrameId ?? captured.frameParentIds?.get(frame.frameId);
return {
...frame,
...(ownerBackendNodeId !== undefined ? { ownerBackendNodeId } : {}),
...(parentFrameId ? { parentFrameId } : {}),
};
});
}
export function buildFrameDocuments<T extends FrameOwnedAxNode>(
graph: CdpFrameGraph | null,
batches: FrameAxBatch<T>[],
captured: CapturedViewModel,
): FrameDocument<T>[] {
const frames = frameList(graph, batches, captured);
const frameById = new Map(frames.map((frame) => [frame.frameId, frame]));
const backendOwner = new Map<string, string>();
for (const frame of frames) {
for (const node of captured.frameNodes?.get(frame.frameId) ?? []) {
backendOwner.set(targetBackendKey(frame.target, node.backendNodeId), frame.frameId);
}
}
const candidates = new Map<string, OwnedCandidate<T>>();
for (const batch of batches) {
const nodeById = new Map(batch.nodes.map((node) => [node.nodeId, node]));
const ownershipByNodeId = new Map<string, Ownership>();
const resolving = new Set<string>();
const resolveOwnership = (node: T): Ownership => {
const cached = ownershipByNodeId.get(node.nodeId);
if (cached) return cached;
if (node.frameId && frameById.has(node.frameId)) {
const ownership = { frameId: node.frameId, strength: 4 };
ownershipByNodeId.set(node.nodeId, ownership);
return ownership;
}
if (typeof node.backendDOMNodeId === "number") {
const frameId = backendOwner.get(
targetBackendKey(batch.frame.target, node.backendDOMNodeId),
);
if (frameId) {
const ownership = { frameId, strength: 3 };
ownershipByNodeId.set(node.nodeId, ownership);
return ownership;
}
}
if (node.parentId && !resolving.has(node.nodeId)) {
const parent = nodeById.get(node.parentId);
if (parent) {
resolving.add(node.nodeId);
const inherited = resolveOwnership(parent);
resolving.delete(node.nodeId);
const ownership = {
frameId: inherited.frameId,
strength: Math.min(2, inherited.strength),
};
ownershipByNodeId.set(node.nodeId, ownership);
return ownership;
}
}
const ownership = { frameId: batch.frame.frameId, strength: 1 };
ownershipByNodeId.set(node.nodeId, ownership);
return ownership;
};
for (const node of batch.nodes) {
const ownership = resolveOwnership(node);
const key = targetNodeKey(batch.frame.target, node.nodeId);
const existing = candidates.get(key);
if (!existing || ownership.strength > existing.ownership.strength) {
candidates.set(key, { node, target: batch.frame.target, ownership });
}
}
}
const ownershipByTargetNode = new Map(
[...candidates].map(([key, candidate]) => [key, candidate.ownership.frameId]),
);
const axNodesByFrame = new Map<string, T[]>();
for (const candidate of candidates.values()) {
const { node, target, ownership } = candidate;
const parentFrameId = node.parentId
? ownershipByTargetNode.get(targetNodeKey(target, node.parentId))
: undefined;
const childIds = node.childIds?.filter(
(childId) => ownershipByTargetNode.get(targetNodeKey(target, childId)) === ownership.frameId,
);
const ownedNode = {
...node,
frameId: ownership.frameId,
...(parentFrameId === ownership.frameId
? { parentId: node.parentId }
: { parentId: undefined }),
...(childIds ? { childIds } : {}),
};
const nodes = axNodesByFrame.get(ownership.frameId) ?? [];
nodes.push(ownedNode);
axNodesByFrame.set(ownership.frameId, nodes);
}
return frames.map((frame) => ({
...frame,
contextScopeId: frame.frameId,
axNodes: axNodesByFrame.get(frame.frameId) ?? [],
domNodes: captured.frameNodes?.get(frame.frameId) ?? [],
}));
}
+56
View File
@@ -33,10 +33,22 @@ export type RpcErrorReason =
| "restricted_tab_url"
| "borrow_conflict"
| "screenshot_capture_failed"
| "file_input_probe_failed"
| "file_input_not_activated"
| "set_file_input_failed"
| "download_capture_failed"
| "transfer_outcome_unknown"
| "transfer_timeout"
| "cleanup_failed";
export type TransferEffectState = "none" | "committed" | "unknown";
export type TransferCleanupState = "complete" | "failed";
export interface RpcErrorData {
reason?: RpcErrorReason;
effect_state?: TransferEffectState;
phase?: string;
cleanup_state?: TransferCleanupState;
[key: string]: unknown;
}
@@ -500,6 +512,50 @@ export interface SelectResult {
dialogs?: JavaScriptDialogInfo[];
}
export interface UploadFile {
transfer_id: string;
name: string;
staged_path?: string;
}
export interface UploadParams {
session_id: string;
ref?: string;
selector?: string;
tab_id?: number;
files: UploadFile[];
timeout_ms?: number;
}
export interface UploadResult {
tab_id: number;
used_ref?: string;
used_selector?: string;
file_names: string[];
}
export interface DownloadParams {
session_id: string;
ref?: string;
selector?: string;
tab_id?: number;
timeout_ms?: number;
browser_relative_dir?: string;
max_byte_size?: number;
}
export interface DownloadResult {
tab_id: number;
used_ref?: string;
used_selector?: string;
suggested_filename: string;
byte_size: number;
mime?: string;
danger?: string;
browser_path?: string;
transfer_id?: string;
}
// --------------------------------------------------------------------------
// M9 tool payloads — evaluate / wait_for_navigation / wait_ms
// --------------------------------------------------------------------------
+3 -4
View File
@@ -25,13 +25,12 @@ export default defineConfig({
name: "BrowserSkill",
description:
"Let AI agents use your logged-in browser in a separate Agent Window—without interrupting your work. Powered by the bsk CLI.",
// Chrome 116 is the first version where WebSocket activity resets the
// service-worker idle timer, which the 20s heartbeat relies on to
// keep the daemon link alive.
minimum_chrome_version: "116",
// Flat debugger sessions are required to address out-of-process iframes.
minimum_chrome_version: "125",
permissions: [
"alarms",
"debugger",
"downloads",
"idle",
"notifications",
"tabs",
+5
View File
@@ -39,6 +39,11 @@
"**",
"!**/node_modules/**",
"!**/dist/**",
"!packages/dsh-plugin-browserskill/lib/**",
"!packages/dsh-plugin-browserskill/src/skill-content.generated.ts",
"!packages/dsh-plugin-browserskill/src/client/bsk-ui.nomodule.css",
"!packages/dsh-plugin-browserskill/.bsk-ui.raw.css",
"!packages/dsh-plugin-browserskill/.bsk-ui.input.css",
"!**/target/**",
"!**/.wxt/**",
"!**/.output/**",
+2
View File
@@ -53,6 +53,7 @@ flate2 = { workspace = true }
tar = { workspace = true }
zip = { workspace = true }
sha2 = { workspace = true }
base64 = { workspace = true }
[target.'cfg(unix)'.dependencies]
nix = { workspace = true }
@@ -63,6 +64,7 @@ windows-sys = { version = "0.59", features = [
"Win32_Foundation",
"Win32_System_Threading",
"Win32_Security",
"Win32_Storage_FileSystem",
] }
[dev-dependencies]
+23
View File
@@ -206,6 +206,29 @@ Both capture from the moment the tab is attached and read a bounded per-tab buff
| `bsk select <ref-or-selector> --value <v>` | Set `<select>` option(s) by `value` (repeat `--value` for multi-select) |
| `bsk press <key>` | Key/combo (`Enter`, `Ctrl+A`, …; optional `--ref` to focus first) |
### File transfer (require `--session`)
| Command | Summary |
|---------|---------|
| `bsk upload <ref-or-selector> --file <path>` | Click one upload trigger and attach an agent-readable local file (`--file` is repeatable) |
| `bsk download <ref-or-selector> --out <path>` | Click one download trigger and copy the single completed file to an exact local path (`--overwrite` is opt-in) |
The agent/harness decides whether a file transfer is appropriate and which local path belongs to the task. Treat upload as disclosure of that file to the current website, and download as accepting website-controlled bytes onto the local filesystem. Use only paths that are necessary for the user's bounded goal.
BrowserSkill enforces the mechanical boundary: files are staged under a session-scoped opaque transfer, only daemon-minted capabilities reach the extension, upload/download still obey Agent Window tab checks, and transfers are chunk/size bounded. Upload intercepts the native chooser for one transaction, locates the input activated in the resolved target's document, and assigns only the staged file paths. Download uniquely correlates one exact-target browser intent with one Chrome download in either event order, routes it through a daemon-minted relative directory, and lets the daemon validate and import it. Upload staging remains available for a later form submission and is removed when the session ends. Downloads cannot overwrite an existing destination unless `--overwrite` is explicit. BrowserSkill does not inspect file content or decide whether its meaning is sensitive.
Do not use `request-help` merely because a native file chooser or browser download is involved; try these commands first. For transfer failures, use the structured error instead of retrying blindly:
- `reason=file_input_not_activated` means the requested click did not activate exactly one `<input type="file">`; the page may use a non-input picker such as `window.showOpenFilePicker()`. Do not retry blindly. If human help is available, call `request-help` and tell the user the exact original local path to choose. The staged daemon path is internal and must not be shown to the user.
- `reason=file_input_probe_failed` means BrowserSkill could not safely establish the browser-side upload transaction. Do not repeat the same action; use `request-help` when available.
- `reason=set_file_input_failed` means BrowserSkill found the activated file input but Chrome rejected the staged path or assignment. Check the extension's file-URL access permission; otherwise use `request-help`.
- `reason=download_capture_failed` means BrowserSkill could not attribute exactly one completed download to the requested target. Do not retry blindly or accept an unrelated browser download; use `request-help` when available.
- `effect_state=none` means BrowserSkill confirmed that no file-transfer effect was committed. Follow the accompanying reason; a corrected target or explicit human fallback may be attempted.
- `effect_state=unknown` means the browser may already have attached or created the file. Do not repeat the transfer. Observe the page if that can establish the result; otherwise stop and report the uncertainty.
- `effect_state=committed` means the browser-side effect occurred even if later completion or cleanup failed. Do not repeat it; continue only after verifying the resulting page/download state.
If `request-help` returns `outcome="disabled"`, do not retry it. Stop gracefully and report the transfer mechanism that requires human intervention.
### Scripting & timing
| Command | Summary |
+75
View File
@@ -0,0 +1,75 @@
//! Atomic visibility boundary for files received into a same-directory temp.
use std::path::Path;
pub fn commit(temp: &Path, out: &Path, overwrite: bool) -> std::io::Result<()> {
if !overwrite {
std::fs::hard_link(temp, out)?;
std::fs::remove_file(temp)?;
return Ok(());
}
replace(temp, out)
}
#[cfg(unix)]
fn replace(temp: &Path, out: &Path) -> std::io::Result<()> {
// POSIX rename replaces an existing non-directory destination atomically.
std::fs::rename(temp, out)
}
#[cfg(windows)]
fn replace(temp: &Path, out: &Path) -> std::io::Result<()> {
use std::os::windows::ffi::OsStrExt;
use windows_sys::Win32::Storage::FileSystem::{
MOVEFILE_REPLACE_EXISTING, MOVEFILE_WRITE_THROUGH, MoveFileExW,
};
let from: Vec<u16> = temp.as_os_str().encode_wide().chain(Some(0)).collect();
let to: Vec<u16> = out.as_os_str().encode_wide().chain(Some(0)).collect();
// SAFETY: both buffers are NUL-terminated and remain alive for the call.
let ok = unsafe {
MoveFileExW(
from.as_ptr(),
to.as_ptr(),
MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH,
)
};
if ok == 0 {
Err(std::io::Error::last_os_error())
} else {
Ok(())
}
}
#[cfg(not(any(unix, windows)))]
fn replace(temp: &Path, out: &Path) -> std::io::Result<()> {
std::fs::rename(temp, out)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn no_replace_never_overwrites_existing_output() {
let dir = tempfile::tempdir().unwrap();
let temp = dir.path().join("new.part");
let out = dir.path().join("out.bin");
std::fs::write(&temp, b"new").unwrap();
std::fs::write(&out, b"old").unwrap();
assert!(commit(&temp, &out, false).is_err());
assert_eq!(std::fs::read(&out).unwrap(), b"old");
}
#[test]
fn overwrite_replaces_existing_output_without_predelete() {
let dir = tempfile::tempdir().unwrap();
let temp = dir.path().join("new.part");
let out = dir.path().join("out.bin");
std::fs::write(&temp, b"new").unwrap();
std::fs::write(&out, b"old").unwrap();
commit(&temp, &out, true).unwrap();
assert_eq!(std::fs::read(&out).unwrap(), b"new");
assert!(!temp.exists());
}
}
+37 -11
View File
@@ -18,6 +18,12 @@ use crate::daemon::state::PROTOCOL_VERSION;
const EXTENSION_STORE_URL: &str =
"https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi";
/// Edge Add-ons listing for the browser-skill extension.
const EXTENSION_STORE_URL_EDGE: &str = "https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg";
/// Store listings highlighted in repair hints, in the order they appear.
const EXTENSION_STORE_URLS: [&str; 2] = [EXTENSION_STORE_URL, EXTENSION_STORE_URL_EDGE];
/// Status of a single doctor check. `Ok` / `Fail` are the legacy two
/// states; `NotApplicable` (review M2) is reported as "N/A" in human
/// output and as `"status": "na"` in `--json` output, so a check that
@@ -420,7 +426,10 @@ fn check_extension_connected(status: Option<&StatusResult>) -> CheckResult {
CheckResult::fail(
name,
"0 browsers connected",
format!("install the extension from {EXTENSION_STORE_URL} and load it in Chromium"),
format!(
"install the extension from {EXTENSION_STORE_URL} (Chrome) \
or {EXTENSION_STORE_URL_EDGE} (Edge) and load it in the browser"
),
)
}
}
@@ -428,17 +437,14 @@ fn check_extension_connected(status: Option<&StatusResult>) -> CheckResult {
/// Highlight known URLs in repair hints for terminal output. Plain
/// text is preserved in `--json` and in stored [`CheckResult::hint`].
fn style_hint(hint: &str) -> String {
if !hint.contains(EXTENSION_STORE_URL) {
return hint.to_string();
let mut styled = hint.to_string();
for url in EXTENSION_STORE_URLS {
if !styled.contains(url) {
continue;
}
styled = styled.replace(url, &style(url).cyan().bold().underlined().to_string());
}
hint.replace(
EXTENSION_STORE_URL,
&style(EXTENSION_STORE_URL)
.cyan()
.bold()
.underlined()
.to_string(),
)
styled
}
fn render_human(checks: &[CheckResult]) {
@@ -504,6 +510,26 @@ mod m2_tests {
hint.contains(EXTENSION_STORE_URL),
"hint should include Chrome Web Store URL: {hint}"
);
assert!(
hint.contains(EXTENSION_STORE_URL_EDGE),
"hint should include Edge Add-ons URL: {hint}"
);
}
#[test]
fn style_hint_preserves_every_store_url() {
let hint = check_extension_connected(Some(&fake_status(Vec::new(), Vec::new())))
.hint
.expect("extension disconnected should include a hint");
let styled = style_hint(&hint);
// Whether or not the terminal accepts colors, styling must never drop or
// mangle a URL the user has to click.
for url in EXTENSION_STORE_URLS {
assert!(
styled.contains(url),
"styled hint should still contain {url}: {styled}"
);
}
}
#[test]
+147
View File
@@ -0,0 +1,147 @@
//! `bsk download` — capture one browser download and atomically commit it.
use std::fs::OpenOptions;
use std::io::Write;
use std::path::{Path, PathBuf};
use std::time::Duration;
use anyhow::Context;
use base64::Engine;
use bsk_protocol::Method;
use bsk_protocol::tools::{
DownloadParams, DownloadResult, TransferChunkParams, TransferChunkResult, TransferIdParams,
TransferReleaseResult,
};
use clap::Args;
use uuid::Uuid;
use crate::cli::atomic_output;
use crate::cli::ensure_daemon::ensure_daemon;
use crate::cli::error::{CliError, Format};
use crate::cli::interaction::split_target;
use crate::cli::navigate::parse_timeout_ms;
#[derive(Debug, Clone, Args)]
pub struct DownloadArgs {
/// Snapshot ref (`@e3`) or CSS selector for the download trigger.
pub target: Option<String>,
#[arg(long = "ref")]
pub ref_: Option<String>,
#[arg(long = "selector")]
pub selector: Option<String>,
#[arg(long)]
pub out: PathBuf,
#[arg(long)]
pub session: String,
#[arg(long = "tab-id")]
pub tab_id: Option<i64>,
#[arg(long, default_value = "2m", value_parser = parse_timeout_ms)]
pub timeout: u32,
#[arg(long)]
pub overwrite: bool,
}
pub fn dispatch(args: DownloadArgs, format: Format) -> Result<(), CliError> {
if args.out.exists() && !args.overwrite {
return Err(CliError::Local(anyhow::anyhow!(
"output already exists (pass --overwrite to replace it): {}",
args.out.display()
)));
}
let info = ensure_daemon().context("ensure daemon is running")?;
let (ref_, selector) = split_target(args.target, args.ref_, args.selector)?;
let params = DownloadParams {
session_id: args.session,
ref_,
selector,
tab_id: args.tab_id,
timeout_ms: Some(args.timeout),
browser_relative_dir: None,
max_byte_size: None,
};
let reply: DownloadResult = crate::cli::business_rpc::call(
info.sock_path.clone(),
"download",
Method::ToolDownload,
Some(params),
ipc_timeout(args.timeout),
)?;
let transfer_id = reply.transfer_id.clone().ok_or_else(|| {
CliError::Local(anyhow::anyhow!("daemon returned no download transfer id"))
})?;
let write_result = write_transfer(&info.sock_path, &transfer_id, &args.out, args.overwrite);
let _: Result<TransferReleaseResult, CliError> = crate::cli::business_rpc::call(
info.sock_path,
"transfer-release",
Method::TransferRelease,
Some(TransferIdParams { transfer_id }),
Duration::from_secs(5),
);
write_result?;
match format {
Format::Json => {
let mut value = serde_json::to_value(&reply).unwrap();
value["path"] = serde_json::json!(args.out.to_string_lossy());
if let Some(obj) = value.as_object_mut() {
obj.remove("transfer_id");
obj.remove("browser_path");
}
println!("{}", serde_json::to_string_pretty(&value).unwrap());
}
Format::Human => println!("{}", args.out.display()),
}
Ok(())
}
fn write_transfer(sock: &Path, id: &str, out: &Path, overwrite: bool) -> Result<(), CliError> {
let parent = out
.parent()
.filter(|p| !p.as_os_str().is_empty())
.unwrap_or(Path::new("."));
let temp = parent.join(format!(".bsk-download-{}.part", Uuid::new_v4().simple()));
let result = (|| {
let mut file = OpenOptions::new()
.write(true)
.create_new(true)
.open(&temp)
.with_context(|| format!("create temporary download {}", temp.display()))
.map_err(CliError::Local)?;
let mut offset = 0u64;
loop {
let chunk: TransferChunkResult = crate::cli::business_rpc::call(
sock.to_path_buf(),
"transfer-read",
Method::TransferRead,
Some(TransferChunkParams {
transfer_id: id.to_string(),
offset,
data_base64: String::new(),
}),
Duration::from_secs(30),
)?;
let encoded = chunk.data_base64.as_deref().unwrap_or("");
let bytes = base64::engine::general_purpose::STANDARD
.decode(encoded)
.map_err(|e| CliError::Local(anyhow::anyhow!("decode download chunk: {e}")))?;
file.write_all(&bytes)
.map_err(|e| CliError::Local(e.into()))?;
offset = chunk.next_offset;
if chunk.eof {
break;
}
}
file.sync_all().map_err(|e| CliError::Local(e.into()))?;
drop(file);
atomic_output::commit(&temp, out, overwrite)
.with_context(|| format!("atomically commit download to {}", out.display()))
.map_err(CliError::Local)
})();
if result.is_err() {
let _ = std::fs::remove_file(&temp);
}
result
}
fn ipc_timeout(timeout_ms: u32) -> Duration {
Duration::from_millis(u64::from(timeout_ms) + 5_000)
}
+1 -1
View File
@@ -71,7 +71,7 @@ pub(crate) fn parse_modifiers(input: &str) -> Result<Vec<KeyModifier>, String> {
Ok(out)
}
fn split_target(
pub(crate) fn split_target(
positional: Option<String>,
explicit_ref: Option<String>,
explicit_selector: Option<String>,
+11
View File
@@ -2,6 +2,7 @@
use std::time::Duration;
mod atomic_output;
pub mod browser_wait;
pub mod browsers;
pub mod business_rpc;
@@ -9,6 +10,7 @@ pub mod console;
pub mod daemon;
pub mod dialogs;
pub mod doctor;
pub mod download;
pub mod emulate;
pub mod ensure_daemon;
pub mod error;
@@ -30,6 +32,7 @@ pub mod snapshot;
pub mod status;
pub mod tab;
pub mod update;
pub mod upload;
pub mod waits;
pub mod window;
@@ -37,6 +40,7 @@ use clap::{Args, Parser, Subcommand};
use crate::cli::console::ConsoleArgs;
use crate::cli::daemon::DaemonCmd;
use crate::cli::download::DownloadArgs;
use crate::cli::emulate::EmulateArgs;
use crate::cli::evaluate::EvaluateArgs;
use crate::cli::get_html::GetHtmlArgs;
@@ -52,6 +56,7 @@ use crate::cli::session::SessionCmd;
use crate::cli::snapshot::SnapshotArgs;
use crate::cli::tab::TabCmd;
use crate::cli::update::UpdateArgs;
use crate::cli::upload::UploadArgs;
use crate::cli::waits::{WaitForNavigationArgs, WaitMsArgs};
use crate::cli::window::WindowCmd;
@@ -176,6 +181,12 @@ pub enum Command {
/// Set `<select>` option values by `value` attribute.
Select(SelectArgs),
/// Upload one or more local files through a page file chooser.
Upload(UploadArgs),
/// Capture one browser download and write it to a local path.
Download(DownloadArgs),
/// Evaluate a JavaScript expression inside the Agent Window.
Evaluate(EvaluateArgs),
+82
View File
@@ -49,6 +49,12 @@ pub mod reason {
pub const TAB_NOT_ACTIVE: &str = "tab_not_active";
pub const BORROW_CONFLICT: &str = "borrow_conflict";
pub const SCREENSHOT_CAPTURE_FAILED: &str = "screenshot_capture_failed";
pub const FILE_INPUT_PROBE_FAILED: &str = "file_input_probe_failed";
pub const FILE_INPUT_NOT_ACTIVATED: &str = "file_input_not_activated";
pub const SET_FILE_INPUT_FAILED: &str = "set_file_input_failed";
pub const DOWNLOAD_CAPTURE_FAILED: &str = "download_capture_failed";
pub const TRANSFER_OUTCOME_UNKNOWN: &str = "transfer_outcome_unknown";
pub const TRANSFER_TIMEOUT: &str = "transfer_timeout";
pub const SESSION_BUSY: &str = crate::rpc_reason::SESSION_BUSY;
}
@@ -196,6 +202,20 @@ pub fn info_for_error(code: ErrorCode, data: Option<&serde_json::Value>) -> Rend
return base;
};
match (code, reason) {
(_, reason::TRANSFER_OUTCOME_UNKNOWN) => RenderInfo {
summary: "the file transfer outcome could not be confirmed",
hint: Some(
"do not retry the transfer: the browser may already have applied it; inspect the page or stop safely",
),
exit_code: base.exit_code,
},
(ErrorCode::Timeout, reason::TRANSFER_TIMEOUT) => RenderInfo {
summary: "the file transfer timed out after browser dispatch",
hint: Some(
"do not retry when effect_state is unknown or committed; inspect the page before taking another action",
),
exit_code: base.exit_code,
},
(ErrorCode::PermissionDenied, reason::ELEMENT_NOT_VISIBLE) => RenderInfo {
summary: "target element has no visible geometry",
hint: Some(
@@ -269,6 +289,36 @@ pub fn info_for_error(code: ErrorCode, data: Option<&serde_json::Value>) -> Rend
),
exit_code: base.exit_code,
},
(ErrorCode::Unsupported, reason::FILE_INPUT_NOT_ACTIVATED) => RenderInfo {
summary: "the upload trigger did not activate a file input",
hint: Some(
"the page may use a non-input picker such as showOpenFilePicker(); do not retry blindly — use `bsk request-help` with the exact original local path, or stop if human help is disabled",
),
exit_code: base.exit_code,
},
(ErrorCode::Timeout | ErrorCode::CdpFailed, reason::FILE_INPUT_PROBE_FAILED) => {
RenderInfo {
summary: "the browser upload transaction could not be established",
hint: Some(
"do not retry the same upload blindly; use `bsk request-help` when human help is available",
),
exit_code: base.exit_code,
}
}
(ErrorCode::Timeout | ErrorCode::CdpFailed, reason::SET_FILE_INPUT_FAILED) => RenderInfo {
summary: "the browser could not attach the staged file to the input",
hint: Some(
"check that BrowserSkill has Chrome's 'Allow access to file URLs' permission; otherwise use `bsk request-help`",
),
exit_code: base.exit_code,
},
(ErrorCode::CdpFailed, reason::DOWNLOAD_CAPTURE_FAILED) => RenderInfo {
summary: "the browser download could not be attributed or completed",
hint: Some(
"do not retry blindly; the target browser may not expose a reliable download intent signal",
),
exit_code: base.exit_code,
},
_ => base,
}
}
@@ -392,6 +442,38 @@ mod tests {
assert_eq!(info.exit_code, 3);
}
#[test]
fn file_transfer_reasons_render_actionable_fallbacks() {
let unsupported = serde_json::json!({ "reason": reason::FILE_INPUT_NOT_ACTIVATED });
let info = info_for_error(ErrorCode::Unsupported, Some(&unsupported));
assert_eq!(
info.summary,
"the upload trigger did not activate a file input"
);
assert!(info.hint.unwrap().contains("exact original local path"));
let control = serde_json::json!({ "reason": reason::FILE_INPUT_PROBE_FAILED });
let info = info_for_error(ErrorCode::Timeout, Some(&control));
assert!(
info.summary
.contains("transaction could not be established")
);
assert!(info.hint.unwrap().contains("do not retry"));
let download = serde_json::json!({ "reason": reason::DOWNLOAD_CAPTURE_FAILED });
let info = info_for_error(ErrorCode::CdpFailed, Some(&download));
assert!(info.summary.contains("download could not be attributed"));
let unknown = serde_json::json!({ "reason": reason::TRANSFER_OUTCOME_UNKNOWN });
let info = info_for_error(ErrorCode::ProtocolError, Some(&unknown));
assert!(info.summary.contains("outcome could not be confirmed"));
assert!(info.hint.unwrap().contains("do not retry"));
let timeout = serde_json::json!({ "reason": reason::TRANSFER_TIMEOUT });
let info = info_for_error(ErrorCode::Timeout, Some(&timeout));
assert!(info.summary.contains("timed out after browser dispatch"));
}
#[test]
fn ref_not_found_overrides_not_found_copy() {
let data = serde_json::json!({ "reason": reason::REF_NOT_FOUND });
+185
View File
@@ -0,0 +1,185 @@
//! `bsk upload` — stage caller-readable files and attach them to a page.
use std::fs::File;
use std::io::Read;
use std::path::{Path, PathBuf};
use std::time::Duration;
use anyhow::Context;
use base64::Engine;
use bsk_protocol::Method;
use bsk_protocol::tools::{
TransferBeginParams, TransferBeginResult, TransferChunkParams, TransferChunkResult,
TransferIdParams, TransferReadyResult, TransferReleaseResult, UploadFile, UploadParams,
UploadResult,
};
use clap::Args;
use crate::cli::ensure_daemon::ensure_daemon;
use crate::cli::error::{CliError, Format};
use crate::cli::interaction::split_target;
use crate::cli::navigate::parse_timeout_ms;
#[derive(Debug, Clone, Args)]
pub struct UploadArgs {
/// Snapshot ref (`@e3`) or CSS selector for the file input / chooser trigger.
pub target: Option<String>,
#[arg(long = "ref")]
pub ref_: Option<String>,
#[arg(long = "selector")]
pub selector: Option<String>,
/// Local file to upload. Repeat for a multiple-file input.
#[arg(long = "file", required = true)]
pub files: Vec<PathBuf>,
#[arg(long)]
pub session: String,
#[arg(long = "tab-id")]
pub tab_id: Option<i64>,
#[arg(long, default_value = "2m", value_parser = parse_timeout_ms)]
pub timeout: u32,
}
pub fn dispatch(args: UploadArgs, format: Format) -> Result<(), CliError> {
if args.files.len() > crate::daemon::file_transfer::MAX_UPLOAD_FILES {
return Err(CliError::Local(anyhow::anyhow!(
"upload accepts at most {} files",
crate::daemon::file_transfer::MAX_UPLOAD_FILES
)));
}
let info = ensure_daemon().context("ensure daemon is running")?;
let (ref_, selector) = split_target(args.target, args.ref_, args.selector)?;
let mut staged = Vec::new();
for path in &args.files {
match stage_file(&info.sock_path, &args.session, path) {
Ok(file) => staged.push(file),
Err(err) => {
for (id, _) in &staged {
let _ = release(&info.sock_path, id);
}
return Err(err);
}
}
}
let params = UploadParams {
session_id: args.session,
ref_,
selector,
tab_id: args.tab_id,
files: staged
.iter()
.map(|(id, name)| UploadFile {
transfer_id: id.clone(),
name: name.clone(),
staged_path: None,
})
.collect(),
timeout_ms: Some(args.timeout),
};
let result = crate::cli::business_rpc::call::<_, UploadResult>(
info.sock_path.clone(),
"upload",
Method::ToolUpload,
Some(params),
ipc_timeout(args.timeout),
);
// Once tool.upload is dispatched, staging ownership belongs to the
// session. A transport timeout cannot prove that Chrome did not attach
// the file, so releasing here could invalidate a late successful attach.
// Session teardown remains the single cleanup boundary after dispatch.
let reply = result?;
match format {
Format::Json => println!("{}", serde_json::to_string_pretty(&reply).unwrap()),
Format::Human => println!(
"upload ok tab={} files={}",
reply.tab_id,
reply.file_names.join(", ")
),
}
Ok(())
}
fn stage_file(sock: &Path, session: &str, path: &PathBuf) -> Result<(String, String), CliError> {
let mut file = File::open(path)
.with_context(|| format!("open upload file {}", path.display()))
.map_err(CliError::Local)?;
let meta = file
.metadata()
.with_context(|| format!("read upload file metadata {}", path.display()))
.map_err(CliError::Local)?;
if !meta.is_file() {
return Err(CliError::Local(anyhow::anyhow!(
"upload source is not a regular file: {}",
path.display()
)));
}
let name = path
.file_name()
.and_then(|v| v.to_str())
.filter(|v| !v.is_empty())
.ok_or_else(|| CliError::Local(anyhow::anyhow!("upload file has no valid basename")))?
.to_string();
let begin: TransferBeginResult = crate::cli::business_rpc::call(
sock.to_path_buf(),
"transfer-begin",
Method::TransferBegin,
Some(TransferBeginParams {
session_id: session.to_string(),
name: name.clone(),
byte_size: meta.len(),
}),
Duration::from_secs(10),
)?;
let staged = (|| {
let mut offset = 0u64;
let mut buf = vec![0u8; begin.chunk_size as usize];
loop {
let n = file.read(&mut buf).map_err(|e| CliError::Local(e.into()))?;
if n == 0 {
break;
}
let reply: TransferChunkResult = crate::cli::business_rpc::call(
sock.to_path_buf(),
"transfer-chunk",
Method::TransferChunk,
Some(TransferChunkParams {
transfer_id: begin.transfer_id.clone(),
offset,
data_base64: base64::engine::general_purpose::STANDARD.encode(&buf[..n]),
}),
Duration::from_secs(30),
)?;
offset = reply.next_offset;
}
let _: TransferReadyResult = crate::cli::business_rpc::call(
sock.to_path_buf(),
"transfer-finish",
Method::TransferFinish,
Some(TransferIdParams {
transfer_id: begin.transfer_id.clone(),
}),
Duration::from_secs(10),
)?;
Ok::<_, CliError>(())
})();
if let Err(err) = staged {
let _ = release(sock, &begin.transfer_id);
return Err(err);
}
Ok((begin.transfer_id, name))
}
fn release(sock: &Path, id: &str) -> Result<TransferReleaseResult, CliError> {
crate::cli::business_rpc::call(
sock.to_path_buf(),
"transfer-release",
Method::TransferRelease,
Some(TransferIdParams {
transfer_id: id.to_string(),
}),
Duration::from_secs(5),
)
}
fn ipc_timeout(timeout_ms: u32) -> Duration {
Duration::from_millis(u64::from(timeout_ms) + 5_000)
}
+728
View File
@@ -0,0 +1,728 @@
//! Session-scoped, content-transparent staging for upload/download tools.
//!
//! The registry deliberately exposes no arbitrary-path operations. Upload
//! bytes arrive from the invoking CLI in bounded chunks; browser downloads
//! land in a daemon-minted directory and are read back by the invoking CLI.
use std::collections::HashMap;
use std::fs::{self, File, OpenOptions};
use std::io::{Read, Seek, SeekFrom, Write};
use std::path::{Component, Path, PathBuf};
use std::sync::Mutex;
use base64::Engine;
use bsk_protocol::tools::{
TransferBeginParams, TransferBeginResult, TransferChunkParams, TransferChunkResult,
TransferIdParams, TransferReadyResult, TransferReleaseResult,
};
use bsk_protocol::{ErrorCode, RpcError};
use uuid::Uuid;
use super::paths;
pub const TRANSFER_CHUNK_SIZE: u32 = 512 * 1024;
pub const MAX_TRANSFER_BYTES: u64 = 512 * 1024 * 1024;
pub const MAX_UPLOAD_FILES: usize = 20;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Direction {
Upload,
Download,
}
#[derive(Debug)]
struct Entry {
session_id: String,
direction: Direction,
path: PathBuf,
expected_size: Option<u64>,
written: u64,
ready: bool,
}
/// Browser download path after the daemon has verified that it belongs to
/// the capability minted for this transfer. From this point onward the daemon
/// owns cleanup on every error path; unvalidated arbitrary paths are never
/// removed.
#[derive(Debug)]
struct ValidatedBrowserFile {
path: PathBuf,
parent: PathBuf,
cleanup_armed: bool,
}
impl ValidatedBrowserFile {
fn validate(reported_path: &Path, transfer_id: &str) -> Result<Self, RpcError> {
if !reported_path.is_absolute() {
return Err(permission(
"download path is outside its browser capability",
));
}
let expected_parent = Path::new("BrowserSkill").join(transfer_id);
let parent = reported_path
.parent()
.ok_or_else(|| permission("download path has no parent directory"))?;
if !parent.ends_with(&expected_parent) {
return Err(permission(
"download escaped its browser capability directory",
));
}
for path in [reported_path, parent] {
if fs::symlink_metadata(path)
.map_err(io_error)?
.file_type()
.is_symlink()
{
return Err(permission("download capability path contains a symlink"));
}
}
Ok(Self {
path: reported_path.to_path_buf(),
parent: parent.to_path_buf(),
cleanup_armed: true,
})
}
fn remove_source(mut self) -> Result<(), RpcError> {
remove_if_present(&self.path).map_err(io_error)?;
remove_dir_if_present(&self.parent).map_err(io_error)?;
self.cleanup_armed = false;
Ok(())
}
}
impl Drop for ValidatedBrowserFile {
fn drop(&mut self) {
if !self.cleanup_armed {
return;
}
let _ = remove_if_present(&self.path);
let _ = remove_dir_if_present(&self.parent);
}
}
#[derive(Debug)]
pub struct DownloadStaging {
pub transfer_id: String,
pub browser_relative_dir: String,
}
#[derive(Debug)]
pub struct TransferRegistry {
root: PathBuf,
initialized: Mutex<bool>,
entries: Mutex<HashMap<String, Entry>>,
}
impl TransferRegistry {
pub fn new() -> anyhow::Result<Self> {
Ok(Self {
root: paths::bsk_home()?.join("run").join("transfers"),
initialized: Mutex::new(false),
entries: Mutex::new(HashMap::new()),
})
}
#[cfg(test)]
fn at_root(root: PathBuf) -> anyhow::Result<Self> {
let registry = Self {
root,
initialized: Mutex::new(false),
entries: Mutex::new(HashMap::new()),
};
registry.ensure_root()?;
Ok(registry)
}
pub fn initialize(&self) -> anyhow::Result<()> {
self.ensure_root()
}
fn ensure_root(&self) -> anyhow::Result<()> {
let mut initialized = self.initialized.lock().unwrap();
if *initialized {
return Ok(());
}
if self.root.exists() {
// Transfers never survive a daemon; removing stale directories on
// startup gives crash cleanup without a second persistence model.
let _ = fs::remove_dir_all(&self.root);
}
fs::create_dir_all(&self.root)?;
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::set_permissions(&self.root, fs::Permissions::from_mode(0o700))?;
}
*initialized = true;
Ok(())
}
pub fn begin_upload(&self, p: TransferBeginParams) -> Result<TransferBeginResult, RpcError> {
self.ensure_root()
.map_err(|err| protocol_error(err.to_string()))?;
let basename = safe_upload_basename(&p.name)?;
if p.byte_size > MAX_TRANSFER_BYTES {
return Err(invalid(format!(
"file size {} exceeds transfer limit {}",
p.byte_size, MAX_TRANSFER_BYTES
)));
}
let mut entries = self.entries.lock().unwrap();
let session_bytes: u64 = entries
.values()
.filter(|entry| entry.session_id == p.session_id)
.map(|entry| entry.expected_size.unwrap_or(entry.written))
.sum();
if session_bytes.saturating_add(p.byte_size) > MAX_TRANSFER_BYTES {
return Err(invalid(format!(
"session transfer staging exceeds limit {}",
MAX_TRANSFER_BYTES
)));
}
let id = new_id();
let dir = self.root.join(&id);
fs::create_dir(&dir).map_err(io_error)?;
set_private_dir(&dir).map_err(io_error)?;
let path = dir.join(basename);
let file = OpenOptions::new()
.write(true)
.create_new(true)
.open(&path)
.map_err(io_error)?;
set_private_file(&file).map_err(io_error)?;
entries.insert(
id.clone(),
Entry {
session_id: p.session_id,
direction: Direction::Upload,
path,
expected_size: Some(p.byte_size),
written: 0,
ready: false,
},
);
Ok(TransferBeginResult {
transfer_id: id,
chunk_size: TRANSFER_CHUNK_SIZE,
})
}
pub fn write_chunk(&self, p: TransferChunkParams) -> Result<TransferChunkResult, RpcError> {
let bytes = base64::engine::general_purpose::STANDARD
.decode(p.data_base64)
.map_err(|e| invalid(format!("invalid transfer chunk: {e}")))?;
if bytes.len() > TRANSFER_CHUNK_SIZE as usize {
return Err(invalid("transfer chunk exceeds negotiated size"));
}
let mut entries = self.entries.lock().unwrap();
let entry = entries
.get_mut(&p.transfer_id)
.ok_or_else(|| not_found("transfer not found"))?;
if entry.direction != Direction::Upload || entry.ready {
return Err(invalid("transfer is not writable"));
}
if p.offset != entry.written {
return Err(invalid(format!(
"non-sequential transfer offset: expected {}, got {}",
entry.written, p.offset
)));
}
let next = entry.written.saturating_add(bytes.len() as u64);
if next > entry.expected_size.unwrap_or(MAX_TRANSFER_BYTES) {
return Err(invalid("transfer exceeds declared byte size"));
}
let mut file = OpenOptions::new()
.append(true)
.open(&entry.path)
.map_err(io_error)?;
file.write_all(&bytes).map_err(io_error)?;
entry.written = next;
Ok(TransferChunkResult {
next_offset: next,
eof: false,
data_base64: None,
})
}
pub fn finish_upload(&self, p: TransferIdParams) -> Result<TransferReadyResult, RpcError> {
let mut entries = self.entries.lock().unwrap();
let entry = entries
.get_mut(&p.transfer_id)
.ok_or_else(|| not_found("transfer not found"))?;
if entry.direction != Direction::Upload {
return Err(invalid("transfer is not an upload"));
}
if entry.written != entry.expected_size.unwrap_or(entry.written) {
return Err(invalid(format!(
"incomplete transfer: expected {} bytes, received {}",
entry.expected_size.unwrap_or(0),
entry.written
)));
}
entry.ready = true;
Ok(TransferReadyResult {
transfer_id: p.transfer_id,
byte_size: entry.written,
})
}
pub fn resolve_uploads(
&self,
session_id: &str,
ids: &[String],
) -> Result<Vec<PathBuf>, RpcError> {
let entries = self.entries.lock().unwrap();
ids.iter()
.map(|id| {
let entry = entries
.get(id)
.ok_or_else(|| not_found("upload transfer not found"))?;
if entry.session_id != session_id
|| entry.direction != Direction::Upload
|| !entry.ready
{
return Err(permission(
"upload transfer is outside this session or not ready",
));
}
Ok(entry.path.clone())
})
.collect()
}
pub fn begin_download(&self, session_id: &str) -> Result<DownloadStaging, RpcError> {
self.ensure_root()
.map_err(|err| protocol_error(err.to_string()))?;
let id = new_id();
let dir = self.root.join(&id);
fs::create_dir(&dir).map_err(io_error)?;
set_private_dir(&dir).map_err(io_error)?;
self.entries.lock().unwrap().insert(
id.clone(),
Entry {
session_id: session_id.to_string(),
direction: Direction::Download,
path: dir.clone(),
expected_size: None,
written: 0,
ready: false,
},
);
Ok(DownloadStaging {
browser_relative_dir: format!("BrowserSkill/{id}"),
transfer_id: id,
})
}
pub fn import_download(&self, id: &str, reported_path: &Path) -> Result<u64, RpcError> {
let mut entries = self.entries.lock().unwrap();
let entry = entries
.get_mut(id)
.ok_or_else(|| not_found("download transfer not found"))?;
if entry.direction != Direction::Download {
return Err(permission("transfer is not a download capability"));
}
let browser_file = ValidatedBrowserFile::validate(reported_path, id)?;
let canonical = browser_file.path.canonicalize().map_err(io_error)?;
let meta = fs::metadata(&canonical).map_err(io_error)?;
if !meta.is_file() || meta.len() > MAX_TRANSFER_BYTES {
return Err(invalid(
"download is not a regular file or exceeds the transfer limit",
));
}
let destination = entry.path.join("payload");
let source = File::open(&canonical).map_err(io_error)?;
let mut destination_file = OpenOptions::new()
.write(true)
.create_new(true)
.open(&destination)
.map_err(io_error)?;
let copied = std::io::copy(
&mut source.take(MAX_TRANSFER_BYTES.saturating_add(1)),
&mut destination_file,
)
.map_err(io_error)?;
if copied > MAX_TRANSFER_BYTES {
let _ = fs::remove_file(&destination);
return Err(invalid("download exceeds the transfer limit"));
}
entry.path = destination;
entry.written = copied;
entry.expected_size = Some(copied);
entry.ready = true;
browser_file.remove_source()?;
Ok(copied)
}
pub fn read_chunk(&self, p: TransferChunkParams) -> Result<TransferChunkResult, RpcError> {
let entries = self.entries.lock().unwrap();
let entry = entries
.get(&p.transfer_id)
.ok_or_else(|| not_found("transfer not found"))?;
if entry.direction != Direction::Download || !entry.ready {
return Err(invalid("transfer is not readable"));
}
if p.offset > entry.written {
return Err(invalid("read offset is beyond transfer length"));
}
let mut file = File::open(&entry.path).map_err(io_error)?;
file.seek(SeekFrom::Start(p.offset)).map_err(io_error)?;
let mut buf = vec![0u8; TRANSFER_CHUNK_SIZE as usize];
let count = file.read(&mut buf).map_err(io_error)?;
buf.truncate(count);
let next = p.offset + count as u64;
Ok(TransferChunkResult {
next_offset: next,
eof: next >= entry.written,
data_base64: Some(base64::engine::general_purpose::STANDARD.encode(buf)),
})
}
pub fn release(&self, p: TransferIdParams) -> TransferReleaseResult {
let entry = self.entries.lock().unwrap().remove(&p.transfer_id);
if let Some(entry) = entry {
let dir = if entry.path.is_dir() {
entry.path
} else {
entry.path.parent().unwrap_or(&self.root).to_path_buf()
};
let _ = fs::remove_dir_all(dir);
TransferReleaseResult { released: true }
} else {
TransferReleaseResult { released: false }
}
}
pub fn release_session(&self, session_id: &str) {
let ids: Vec<String> = self
.entries
.lock()
.unwrap()
.iter()
.filter(|(_, entry)| entry.session_id == session_id)
.map(|(id, _)| id.clone())
.collect();
for id in ids {
self.release(TransferIdParams { transfer_id: id });
}
}
}
fn new_id() -> String {
format!("tr_{}", Uuid::new_v4().simple())
}
fn set_private_dir(path: &Path) -> std::io::Result<()> {
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::set_permissions(path, fs::Permissions::from_mode(0o700))?;
}
Ok(())
}
fn set_private_file(file: &File) -> std::io::Result<()> {
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
file.set_permissions(fs::Permissions::from_mode(0o600))?;
}
Ok(())
}
fn remove_if_present(path: &Path) -> std::io::Result<()> {
match fs::remove_file(path) {
Ok(()) => Ok(()),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(err) => Err(err),
}
}
fn remove_dir_if_present(path: &Path) -> std::io::Result<()> {
match fs::remove_dir(path) {
Ok(()) => Ok(()),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(err) => Err(err),
}
}
fn safe_upload_basename(name: &str) -> Result<&str, RpcError> {
if name.is_empty() || name.contains(['/', '\\', '\0']) {
return Err(invalid("upload name must be a safe basename"));
}
let mut components = Path::new(name).components();
match (components.next(), components.next()) {
(Some(Component::Normal(_)), None) => Ok(name),
_ => Err(invalid("upload name must be a safe basename")),
}
}
fn invalid(message: impl Into<String>) -> RpcError {
RpcError {
code: ErrorCode::InvalidParams,
message: message.into(),
data: None,
}
}
fn not_found(message: impl Into<String>) -> RpcError {
RpcError {
code: ErrorCode::NotFound,
message: message.into(),
data: None,
}
}
fn permission(message: impl Into<String>) -> RpcError {
RpcError {
code: ErrorCode::PermissionDenied,
message: message.into(),
data: None,
}
}
fn io_error(err: std::io::Error) -> RpcError {
RpcError {
code: ErrorCode::ProtocolError,
message: err.to_string(),
data: None,
}
}
fn protocol_error(message: impl Into<String>) -> RpcError {
RpcError {
code: ErrorCode::ProtocolError,
message: message.into(),
data: None,
}
}
#[cfg(test)]
mod tests {
use super::*;
fn registry() -> (tempfile::TempDir, TransferRegistry) {
let temp = tempfile::tempdir().unwrap();
let registry = TransferRegistry::at_root(temp.path().join("transfers")).unwrap();
(temp, registry)
}
#[test]
fn upload_is_not_resolvable_until_complete_or_from_another_session() {
let (_temp, registry) = registry();
let begin = registry
.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: "image.png".into(),
byte_size: 3,
})
.unwrap();
assert!(
registry
.resolve_uploads("s1", std::slice::from_ref(&begin.transfer_id))
.is_err()
);
registry
.write_chunk(TransferChunkParams {
transfer_id: begin.transfer_id.clone(),
offset: 0,
data_base64: base64::engine::general_purpose::STANDARD.encode(b"abc"),
})
.unwrap();
registry
.finish_upload(TransferIdParams {
transfer_id: begin.transfer_id.clone(),
})
.unwrap();
assert!(
registry
.resolve_uploads("s2", std::slice::from_ref(&begin.transfer_id))
.is_err()
);
let [path] = registry
.resolve_uploads("s1", &[begin.transfer_id])
.unwrap()
.try_into()
.unwrap();
assert_eq!(path.file_name().unwrap(), "image.png");
assert_eq!(fs::read(path).unwrap(), b"abc");
}
#[test]
fn upload_name_must_be_one_safe_path_component() {
let (_temp, registry) = registry();
for name in ["", ".", "..", "../secret", "folder/file", "folder\\file"] {
let result = registry.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: name.into(),
byte_size: 0,
});
assert!(result.is_err(), "accepted unsafe upload name");
}
assert!(registry.entries.lock().unwrap().is_empty());
assert!(fs::read_dir(&registry.root).unwrap().next().is_none());
}
#[test]
fn same_upload_names_are_isolated_by_transfer_directory() {
let (_temp, registry) = registry();
let first = registry
.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: "image.png".into(),
byte_size: 0,
})
.unwrap();
let second = registry
.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: "image.png".into(),
byte_size: 0,
})
.unwrap();
let entries = registry.entries.lock().unwrap();
let first_path = &entries[&first.transfer_id].path;
let second_path = &entries[&second.transfer_id].path;
assert_ne!(first_path.parent(), second_path.parent());
assert_eq!(first_path.file_name().unwrap(), "image.png");
assert_eq!(second_path.file_name().unwrap(), "image.png");
}
#[cfg(unix)]
#[test]
fn upload_staging_is_private() {
use std::os::unix::fs::PermissionsExt;
let (_temp, registry) = registry();
let begin = registry
.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: "private.png".into(),
byte_size: 0,
})
.unwrap();
let entries = registry.entries.lock().unwrap();
let path = &entries[&begin.transfer_id].path;
assert_eq!(
fs::metadata(path.parent().unwrap())
.unwrap()
.permissions()
.mode()
& 0o777,
0o700
);
assert_eq!(
fs::metadata(path).unwrap().permissions().mode() & 0o777,
0o600
);
}
#[test]
fn download_must_import_from_its_browser_capability_directory() {
let (temp, registry) = registry();
let staging = registry.begin_download("s1").unwrap();
let outside = temp.path().join("outside.bin");
fs::write(&outside, b"secret").unwrap();
assert!(
registry
.import_download(&staging.transfer_id, &outside)
.is_err()
);
assert_eq!(fs::read(&outside).unwrap(), b"secret");
let browser_dir = temp
.path()
.join("Downloads")
.join("BrowserSkill")
.join(&staging.transfer_id);
fs::create_dir_all(&browser_dir).unwrap();
let inside = browser_dir.join("result.bin");
fs::write(&inside, b"result").unwrap();
assert_eq!(
registry
.import_download(&staging.transfer_id, &inside)
.unwrap(),
6
);
assert!(!inside.exists());
let chunk = registry
.read_chunk(TransferChunkParams {
transfer_id: staging.transfer_id,
offset: 0,
data_base64: String::new(),
})
.unwrap();
assert_eq!(
base64::engine::general_purpose::STANDARD
.decode(chunk.data_base64.unwrap())
.unwrap(),
b"result"
);
}
#[test]
fn validated_oversized_browser_download_is_removed_on_rejection() {
let (temp, registry) = registry();
let staging = registry.begin_download("s1").unwrap();
let browser_dir = temp
.path()
.join("Downloads")
.join("BrowserSkill")
.join(&staging.transfer_id);
fs::create_dir_all(&browser_dir).unwrap();
let inside = browser_dir.join("oversized.bin");
File::create(&inside)
.unwrap()
.set_len(MAX_TRANSFER_BYTES + 1)
.unwrap();
assert!(
registry
.import_download(&staging.transfer_id, &inside)
.is_err()
);
assert!(!inside.exists());
assert!(!browser_dir.exists());
}
#[test]
fn releasing_a_session_removes_all_staging() {
let (_temp, registry) = registry();
let first = registry.begin_download("s1").unwrap();
let second = registry.begin_download("s2").unwrap();
let first_directory = registry.root.join(&first.transfer_id);
let second_directory = registry.root.join(&second.transfer_id);
registry.release_session("s1");
assert!(!first_directory.exists());
assert!(second_directory.exists());
}
#[test]
fn upload_staging_is_bounded_across_a_session() {
let (_temp, registry) = registry();
registry
.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: "large.bin".into(),
byte_size: MAX_TRANSFER_BYTES,
})
.unwrap();
assert!(
registry
.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: "one-more-byte.bin".into(),
byte_size: 1,
})
.is_err()
);
assert!(
registry
.begin_upload(TransferBeginParams {
session_id: "s2".into(),
name: "other-session.bin".into(),
byte_size: 1,
})
.is_ok()
);
}
}
+209 -5
View File
@@ -27,7 +27,8 @@ use bsk_protocol::system::{
VersionSkewEntry,
};
use bsk_protocol::tools::{
HelpOutcome, RequestHelpResult, ReturnFailure, WaitMsParams, WaitMsResult,
DownloadParams, DownloadResult, HelpOutcome, RequestHelpResult, ReturnFailure,
TransferBeginParams, TransferIdParams, UploadParams, WaitMsParams, WaitMsResult,
};
use bsk_protocol::{
CancelParams, CancelResult, ErrorCode, Method, PingResult, ResponseBody, RpcError, RpcId,
@@ -61,6 +62,10 @@ pub type RpcHandler = Arc<
>;
const DEFAULT_RPC_TIMEOUT: Duration = Duration::from_secs(15);
// The upload transaction owns its operation deadline and may need a bounded
// cleanup before it can return a useful structured error. Keep only that
// transport alive slightly longer so it does not replace the result.
const EXTENSION_RESPONSE_GRACE: Duration = Duration::from_secs(2);
/// Upper bound on `wait_for_browser_ms` accepted over IPC.
const MAX_BROWSER_WAIT: Duration = Duration::from_secs(60);
// `session.stop` fast-fails while another tool is active; this budget
@@ -223,6 +228,11 @@ pub fn full_handler(status: DaemonStatus, state: Arc<DaemonState>) -> RpcHandler
Ok(v) => ResponseBody::Ok(v),
Err(e) => ResponseBody::Err(e),
},
Method::TransferBegin => handle_transfer_begin(&state, params),
Method::TransferChunk => handle_transfer_chunk(&state, params),
Method::TransferFinish => handle_transfer_finish(&state, params),
Method::TransferRead => handle_transfer_read(&state, params),
Method::TransferRelease => handle_transfer_release(&state, params),
Method::ToolTabList
| Method::ToolTabCreate
| Method::ToolTabClose
@@ -246,6 +256,8 @@ pub fn full_handler(status: DaemonStatus, state: Arc<DaemonState>) -> RpcHandler
| Method::ToolFill
| Method::ToolPress
| Method::ToolSelect
| Method::ToolUpload
| Method::ToolDownload
| Method::ToolEvaluate
| Method::ToolWaitForNavigation
| Method::ToolRequestHelp
@@ -324,7 +336,7 @@ async fn handle_tool_dispatch(
data: None,
});
}
let timeout = match tool_dispatch_timeout(&params) {
let timeout = match tool_dispatch_transport_timeout(&method, &params) {
Ok(timeout) => timeout,
Err(err) => return ResponseBody::Err(err),
};
@@ -338,24 +350,180 @@ async fn handle_tool_dispatch(
});
}
};
let mut params = params;
let mut download_transfer_id: Option<String> = None;
if method == Method::ToolUpload {
let mut upload: UploadParams = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
if upload.files.is_empty() || upload.files.len() > super::file_transfer::MAX_UPLOAD_FILES {
return ResponseBody::Err(invalid_params(format!(
"upload requires 1..={} files",
super::file_transfer::MAX_UPLOAD_FILES
)));
}
let ids: Vec<String> = upload.files.iter().map(|f| f.transfer_id.clone()).collect();
let paths = match state.transfers.resolve_uploads(&session_id.0, &ids) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(err),
};
for (file, path) in upload.files.iter_mut().zip(paths) {
file.staged_path = Some(path.to_string_lossy().into_owned());
}
params = serde_json::to_value(upload).unwrap_or(Value::Null);
} else if method == Method::ToolDownload {
let mut download: DownloadParams = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
let staging = match state.transfers.begin_download(&session_id.0) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(err),
};
download.browser_relative_dir = Some(staging.browser_relative_dir);
download.max_byte_size = Some(super::file_transfer::MAX_TRANSFER_BYTES);
download_transfer_id = Some(staging.transfer_id);
params = serde_json::to_value(download).unwrap_or(Value::Null);
}
let entry = inflight_guard.entry();
// `record_stop` must reach the extension while `record_await` holds the
// serial busy lock — finishing the recording unblocks await.
let outcome = if method == Method::ToolRecordStop {
state
.tool_queues
.dispatch_unlocked(&session_id, method, params, timeout, Some(entry))
.dispatch_unlocked(&session_id, method.clone(), params, timeout, Some(entry))
.await
} else {
state
.tool_queues
.dispatch(&session_id, method, params, timeout, Some(entry))
.dispatch(&session_id, method.clone(), params, timeout, Some(entry))
.await
};
drop(inflight_guard);
match outcome {
Ok(v) if method == Method::ToolDownload => {
let id = download_transfer_id.expect("download transfer allocated");
let mut result: DownloadResult = match serde_json::from_value(v) {
Ok(v) => v,
Err(err) => {
state
.transfers
.release(TransferIdParams { transfer_id: id });
return ResponseBody::Err(RpcError {
code: ErrorCode::ProtocolError,
message: format!("invalid tool.download result: {err}"),
data: None,
});
}
};
let Some(path) = result.browser_path.take() else {
state
.transfers
.release(TransferIdParams { transfer_id: id });
return ResponseBody::Err(RpcError {
code: ErrorCode::ProtocolError,
message: "tool.download returned no browser_path".into(),
data: None,
});
};
match state
.transfers
.import_download(&id, std::path::Path::new(&path))
{
Ok(size) => {
result.byte_size = size;
result.transfer_id = Some(id);
ResponseBody::Ok(serde_json::to_value(result).unwrap_or(Value::Null))
}
Err(err) => {
state
.transfers
.release(TransferIdParams { transfer_id: id });
ResponseBody::Err(err)
}
}
}
Ok(v) => ResponseBody::Ok(v),
Err(err) => ResponseBody::Err(err.into_rpc()),
Err(err) => {
if let Some(id) = download_transfer_id {
state
.transfers
.release(TransferIdParams { transfer_id: id });
}
ResponseBody::Err(err.into_rpc())
}
}
}
fn handle_transfer_begin(state: &Arc<DaemonState>, params: Value) -> ResponseBody {
let p: TransferBeginParams = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
if state
.sessions
.get(&SessionId(p.session_id.clone()))
.is_none()
{
return ResponseBody::Err(RpcError {
code: ErrorCode::NotFound,
message: format!("session {} unknown", p.session_id),
data: None,
});
}
match state.transfers.begin_upload(p) {
Ok(v) => ResponseBody::Ok(serde_json::to_value(v).unwrap_or(Value::Null)),
Err(err) => ResponseBody::Err(err),
}
}
fn handle_transfer_chunk(state: &Arc<DaemonState>, params: Value) -> ResponseBody {
let p = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
match state.transfers.write_chunk(p) {
Ok(v) => ResponseBody::Ok(serde_json::to_value(v).unwrap_or(Value::Null)),
Err(err) => ResponseBody::Err(err),
}
}
fn handle_transfer_finish(state: &Arc<DaemonState>, params: Value) -> ResponseBody {
let p = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
match state.transfers.finish_upload(p) {
Ok(v) => ResponseBody::Ok(serde_json::to_value(v).unwrap_or(Value::Null)),
Err(err) => ResponseBody::Err(err),
}
}
fn handle_transfer_read(state: &Arc<DaemonState>, params: Value) -> ResponseBody {
let p = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
match state.transfers.read_chunk(p) {
Ok(v) => ResponseBody::Ok(serde_json::to_value(v).unwrap_or(Value::Null)),
Err(err) => ResponseBody::Err(err),
}
}
fn handle_transfer_release(state: &Arc<DaemonState>, params: Value) -> ResponseBody {
let p: TransferIdParams = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
ResponseBody::Ok(serde_json::to_value(state.transfers.release(p)).unwrap_or(Value::Null))
}
fn invalid_params(message: impl Into<String>) -> RpcError {
RpcError {
code: ErrorCode::InvalidParams,
message: message.into(),
data: None,
}
}
@@ -532,6 +700,16 @@ fn tool_dispatch_timeout(params: &Value) -> Result<Duration, RpcError> {
Ok(Duration::from_millis(u64::from(ms)))
}
fn tool_dispatch_transport_timeout(method: &Method, params: &Value) -> Result<Duration, RpcError> {
tool_dispatch_timeout(params).map(|timeout| {
if matches!(method, Method::ToolUpload | Method::ToolDownload) {
timeout.saturating_add(EXTENSION_RESPONSE_GRACE)
} else {
timeout
}
})
}
// Local CLI-facing shapes. Intentionally distinct from
// `bsk_protocol::tools::SessionStart*` (which describes the WS-facing
// `tool.session_*` round-trip with the extension) so the two sides can
@@ -757,6 +935,7 @@ async fn handle_session_stop(state: &Arc<DaemonState>, params: Value) -> Result<
.await
{
Ok(stop) => {
state.transfers.release_session(&session_id.0);
let result = CliSessionStopResult {
stopped: vec![session_id.0],
failed: Vec::new(),
@@ -827,6 +1006,7 @@ async fn handle_session_stop_all(state: &Arc<DaemonState>) -> Result<Value, RpcE
.await
{
Ok(stop) => {
state.transfers.release_session(&id.0);
stopped.push(id.0);
returned_tab_ids.extend(stop.returned_tab_ids);
return_failures.extend(stop.return_failures);
@@ -1250,6 +1430,30 @@ mod tests {
assert_eq!(got, std::time::Duration::from_millis(300_000));
}
#[test]
fn extension_transport_outlives_the_operation_deadline() {
let params = serde_json::json!({
"session_id": "abcd",
"timeout_ms": 60_000,
});
let dispatch_timeout =
tool_dispatch_transport_timeout(&Method::ToolUpload, &params).unwrap();
assert_eq!(dispatch_timeout, Duration::from_secs(62));
}
#[test]
fn extension_response_grace_does_not_change_other_tools() {
let params = serde_json::json!({
"session_id": "abcd",
"timeout_ms": 60_000,
});
let dispatch_timeout =
tool_dispatch_transport_timeout(&Method::ToolClick, &params).unwrap();
assert_eq!(dispatch_timeout, Duration::from_secs(60));
}
#[test]
fn tool_dispatch_timeout_rejects_zero_timeout_ms() {
let params = serde_json::json!({
+1
View File
@@ -3,6 +3,7 @@
pub mod abort;
pub mod browsers;
mod cancel_forward;
pub mod file_transfer;
pub mod inflight;
pub mod info;
pub mod ipc;
+227 -19
View File
@@ -420,6 +420,11 @@ async fn dispatch_with_sender(
wait_for_capacity: bool,
inflight: Option<Arc<ToolInflightEntry>>,
) -> Result<Value, DispatchError> {
let response_grace = if is_effect_aware_transfer(&method) {
CANCEL_CLEANUP_TIMEOUT
} else {
Duration::ZERO
};
let (respond_tx, respond_rx) = oneshot::channel();
let job = ToolJob {
method,
@@ -453,8 +458,13 @@ async fn dispatch_with_sender(
mpsc::error::TrySendError::Closed(_) => DispatchError::QueueClosed,
});
}
let waited =
tokio::time::timeout(timeout.saturating_add(Duration::from_secs(1)), respond_rx).await;
let waited = tokio::time::timeout(
timeout
.saturating_add(response_grace)
.saturating_add(Duration::from_secs(1)),
respond_rx,
)
.await;
match waited {
Ok(Ok(Ok(v))) => Ok(v),
Ok(Ok(Err(rpc))) => Err(DispatchError::Rpc(rpc)),
@@ -576,32 +586,78 @@ async fn forward_one(
None
}
};
let send_deadline_cancel = || {
client
.sink
.send(Frame::Request(RequestFrame {
id: format!("deadline-cancel-{rpc_id}"),
method: Method::Cancel,
params: Some(serde_json::json!({ "rpc_id": rpc_id })),
}))
.is_ok()
};
let deadline_cancel: Option<&(dyn Fn() -> bool + Sync)> =
is_effect_aware_transfer(&job.method).then_some(&send_deadline_cancel);
let waited = await_with_optional_cancel(
job.timeout,
CANCEL_CLEANUP_TIMEOUT,
waiter,
cancel_token.as_ref(),
deadline_cancel,
)
.await;
let response = match waited {
WaitOutcome::Response(resp) => resp,
WaitOutcome::CancelledAfterResponse(resp) => {
// Cancellation wins the external verdict, but only after the
// extension's original RPC has settled. Preserve any non-cancel
// error so compensation failures remain explicit instead of
// being hidden behind a generic cancelled result.
if let ResponseBody::Err(err) = resp.body
&& !matches!(err.code, ErrorCode::Cancelled | ErrorCode::UserAborted)
{
return Err(err);
// File transfer commits are irreversible. A late cancel cannot
// overwrite a confirmed success, and an unknown transfer effect
// must remain explicit so callers do not retry or release upload
// staging as though nothing happened.
if is_effect_aware_transfer(&job.method) {
match &resp.body {
ResponseBody::Ok(_) => resp,
ResponseBody::Err(err)
if transfer_effect(err) == Some("unknown")
|| transfer_effect(err) == Some("committed") =>
{
return Err(err.clone());
}
ResponseBody::Err(err)
if !matches!(err.code, ErrorCode::Cancelled | ErrorCode::UserAborted) =>
{
return Err(err.clone());
}
ResponseBody::Err(_) => {
return Err(cancelled_error(
job.inflight.as_deref(),
"tool dispatch cancelled after extension cleanup",
));
}
}
} else {
// For ordinary tools cancellation keeps the existing verdict,
// while non-cancel errors still expose compensation failures.
if let ResponseBody::Err(err) = resp.body
&& !matches!(err.code, ErrorCode::Cancelled | ErrorCode::UserAborted)
{
return Err(err);
}
return Err(cancelled_error(
job.inflight.as_deref(),
"tool dispatch cancelled after extension cleanup",
));
}
return Err(cancelled_error(
job.inflight.as_deref(),
"tool dispatch cancelled after extension cleanup",
));
}
WaitOutcome::CleanupTimeout => {
client.pending.lock().unwrap().cancel(&rpc_id);
if is_effect_aware_transfer(&job.method) {
return Err(unknown_transfer_error(
ErrorCode::Timeout,
"cancelled file transfer did not confirm its outcome before cleanup timed out",
"cleanup",
true,
));
}
return Err(RpcError {
code: ErrorCode::Timeout,
message: format!(
@@ -611,8 +667,36 @@ async fn forward_one(
data: Some(serde_json::json!({ "reason": "cancel_cleanup_timeout" })),
});
}
WaitOutcome::TimedOutAfterResponse(resp) => match resp.body {
ResponseBody::Ok(value) => return Ok(value),
ResponseBody::Err(err)
if !matches!(err.code, ErrorCode::Cancelled | ErrorCode::UserAborted) =>
{
return Err(err);
}
ResponseBody::Err(err) => {
return Err(timed_out_transfer_error(&err));
}
},
WaitOutcome::TimeoutCleanupFailed => {
client.pending.lock().unwrap().cancel(&rpc_id);
return Err(unknown_transfer_error(
ErrorCode::Timeout,
"file transfer timed out and cleanup could not be confirmed",
"cleanup",
true,
));
}
WaitOutcome::WaiterClosed => {
client.pending.lock().unwrap().cancel(&rpc_id);
if is_effect_aware_transfer(&job.method) {
return Err(unknown_transfer_error(
ErrorCode::ProtocolError,
"file transfer transport closed after dispatch; outcome is unknown",
"transport",
false,
));
}
return Err(RpcError {
code: ErrorCode::ProtocolError,
message: "transport closed mid-call".into(),
@@ -621,6 +705,14 @@ async fn forward_one(
}
WaitOutcome::Timeout => {
client.pending.lock().unwrap().cancel(&rpc_id);
if is_effect_aware_transfer(&job.method) {
return Err(unknown_transfer_error(
ErrorCode::Timeout,
"file transfer timed out after dispatch; outcome is unknown",
"transport",
false,
));
}
return Err(RpcError {
code: ErrorCode::Timeout,
message: format!("tool RPC timed out after {:?}", job.timeout),
@@ -634,11 +726,65 @@ async fn forward_one(
}
}
fn is_effect_aware_transfer(method: &Method) -> bool {
matches!(method, Method::ToolUpload | Method::ToolDownload)
}
fn transfer_effect(err: &RpcError) -> Option<&str> {
err.data.as_ref()?.get("effect_state")?.as_str()
}
fn unknown_transfer_error(
code: ErrorCode,
message: impl Into<String>,
phase: &str,
cleanup_failed: bool,
) -> RpcError {
let mut data = serde_json::json!({
"reason": "transfer_outcome_unknown",
"effect_state": "unknown",
"phase": phase,
});
if cleanup_failed {
data["cleanup_state"] = serde_json::json!("failed");
}
RpcError {
code,
message: message.into(),
data: Some(data),
}
}
fn timed_out_transfer_error(err: &RpcError) -> RpcError {
let data = err.data.as_ref();
RpcError {
code: ErrorCode::Timeout,
message: "file transfer timed out after dispatch".into(),
data: Some(serde_json::json!({
"reason": "transfer_timeout",
"effect_state": data
.and_then(|value| value.get("effect_state"))
.and_then(Value::as_str)
.unwrap_or("unknown"),
"phase": data
.and_then(|value| value.get("phase"))
.and_then(Value::as_str)
.unwrap_or("cleanup"),
"cleanup_state": data
.and_then(|value| value.get("cleanup_state"))
.and_then(Value::as_str)
.unwrap_or("complete"),
})),
}
}
#[derive(Debug)]
enum WaitOutcome {
Response(bsk_protocol::ResponseFrame),
CancelledAfterResponse(bsk_protocol::ResponseFrame),
CleanupTimeout,
TimedOutAfterResponse(bsk_protocol::ResponseFrame),
TimeoutCleanupFailed,
WaiterClosed,
Timeout,
}
@@ -648,6 +794,7 @@ async fn await_with_optional_cancel(
cleanup_timeout: Duration,
mut waiter: oneshot::Receiver<bsk_protocol::ResponseFrame>,
cancel: Option<&super::abort::AbortToken>,
on_deadline: Option<&(dyn Fn() -> bool + Sync)>,
) -> WaitOutcome {
match cancel {
Some(token) => {
@@ -669,13 +816,31 @@ async fn await_with_optional_cancel(
Ok(resp) => WaitOutcome::Response(resp),
Err(_) => WaitOutcome::WaiterClosed,
},
_ = &mut deadline => WaitOutcome::Timeout,
_ = &mut deadline => match on_deadline {
Some(send_cancel) if send_cancel() => {
match tokio::time::timeout(cleanup_timeout, &mut waiter).await {
Ok(Ok(resp)) => WaitOutcome::TimedOutAfterResponse(resp),
Ok(Err(_)) | Err(_) => WaitOutcome::TimeoutCleanupFailed,
}
}
Some(_) => WaitOutcome::TimeoutCleanupFailed,
None => WaitOutcome::Timeout,
},
}
}
None => match tokio::time::timeout(timeout, &mut waiter).await {
Ok(Ok(resp)) => WaitOutcome::Response(resp),
Ok(Err(_)) => WaitOutcome::WaiterClosed,
Err(_) => WaitOutcome::Timeout,
Err(_) => match on_deadline {
Some(send_cancel) if send_cancel() => {
match tokio::time::timeout(cleanup_timeout, &mut waiter).await {
Ok(Ok(resp)) => WaitOutcome::TimedOutAfterResponse(resp),
Ok(Err(_)) | Err(_) => WaitOutcome::TimeoutCleanupFailed,
}
}
Some(_) => WaitOutcome::TimeoutCleanupFailed,
None => WaitOutcome::Timeout,
},
},
}
}
@@ -759,6 +924,7 @@ mod await_with_optional_cancel_tests {
Duration::from_secs(1),
rx,
Some(&token),
None,
)
.await;
assert!(
@@ -781,6 +947,7 @@ mod await_with_optional_cancel_tests {
Duration::from_secs(1),
rx,
Some(&token),
None,
)
.await;
match outcome {
@@ -795,9 +962,14 @@ mod await_with_optional_cancel_tests {
async fn no_cancel_token_still_returns_response() {
let (tx, rx) = oneshot::channel();
tx.send(dummy_response()).unwrap();
let outcome =
await_with_optional_cancel(Duration::from_secs(10), Duration::from_secs(1), rx, None)
.await;
let outcome = await_with_optional_cancel(
Duration::from_secs(10),
Duration::from_secs(1),
rx,
None,
None,
)
.await;
assert!(matches!(outcome, WaitOutcome::Response(_)));
}
@@ -812,6 +984,7 @@ mod await_with_optional_cancel_tests {
Duration::from_secs(1),
rx,
Some(&token),
None,
)
.await
});
@@ -835,10 +1008,45 @@ mod await_with_optional_cancel_tests {
Duration::from_millis(10),
rx,
Some(&token),
None,
)
.await;
assert!(matches!(outcome, WaitOutcome::CleanupTimeout));
}
#[tokio::test]
async fn transfer_deadline_sends_cancel_and_waits_for_cleanup_response() {
let (tx, rx) = oneshot::channel();
tokio::spawn(async move {
tokio::time::sleep(Duration::from_millis(5)).await;
tx.send(dummy_response()).unwrap();
});
let send_cancel = || true;
let outcome = await_with_optional_cancel(
Duration::from_millis(1),
Duration::from_secs(1),
rx,
None,
Some(&send_cancel),
)
.await;
assert!(matches!(outcome, WaitOutcome::TimedOutAfterResponse(_)));
}
#[tokio::test]
async fn transfer_deadline_is_unknown_when_cancel_cannot_be_sent() {
let (_tx, rx) = oneshot::channel();
let send_cancel = || false;
let outcome = await_with_optional_cancel(
Duration::from_millis(1),
Duration::from_secs(1),
rx,
None,
Some(&send_cancel),
)
.await;
assert!(matches!(outcome, WaitOutcome::TimeoutCleanupFailed));
}
}
#[cfg(test)]
+10 -1
View File
@@ -226,6 +226,10 @@ pub fn run_foreground(cfg: DaemonConfig) -> Result<()> {
.with_context(|| format!("bind IPC endpoint {}", sock_path.display()))?;
let state = Arc::new(DaemonState::new(cfg.clone()));
state
.transfers
.initialize()
.context("initialize transfer staging")?;
let session_idle_task = spawn_session_idle_reaper(Arc::clone(&state));
let browser_liveness_task = spawn_browser_liveness_reaper(Arc::clone(&state));
// Fired by the update check task after a successful auto-update:
@@ -458,6 +462,7 @@ pub(crate) fn spawn_browser_liveness_reaper(
for s in state.sessions.purge_browser(&client.id) {
state.tool_queues.remove(&s.id);
state.session_interrupts.drop_session(&s.id);
state.transfers.release_session(&s.id.0);
debug!(session = %s.id, "purged session on browser liveness timeout");
}
}
@@ -494,7 +499,10 @@ pub(crate) fn spawn_session_idle_reaper(state: Arc<DaemonState>) -> tokio::task:
)
.await
{
Ok(_) => info!(session = %session_id, "idle session stopped"),
Ok(_) => {
state.transfers.release_session(&session_id.0);
info!(session = %session_id, "idle session stopped");
}
Err(StopSessionError::SessionBusy | StopSessionError::Stopping) => {
debug!(session = %session_id, "idle session still active; retrying later");
}
@@ -505,6 +513,7 @@ pub(crate) fn spawn_session_idle_reaper(state: Arc<DaemonState>) -> tokio::task:
&state.session_interrupts,
&session_id,
);
state.transfers.release_session(&session_id.0);
}
Err(err) => {
warn!(session = %session_id, error = %err, "failed to stop idle session");
+6
View File
@@ -7,6 +7,7 @@ use tokio::task::JoinHandle;
use super::abort::AbortRegistry;
use super::browsers::BrowserRegistry;
use super::file_transfer::TransferRegistry;
use super::inflight::ToolInflightRegistry;
use super::ipc::IpcHandle;
use super::queue::ToolQueueRegistry;
@@ -51,6 +52,9 @@ pub struct DaemonState {
/// `SessionRegistry` because the signal is a transient runtime
/// control state.
pub session_interrupts: Arc<SessionInterruptRegistry>,
/// Operation-scoped local file staging. The extension only sees paths
/// minted here; agent-facing RPCs use opaque transfer ids.
pub transfers: Arc<TransferRegistry>,
}
impl DaemonState {
@@ -64,6 +68,7 @@ impl DaemonState {
));
let abort_registry = Arc::new(AbortRegistry::new());
let session_interrupts = Arc::new(SessionInterruptRegistry::new());
let transfers = Arc::new(TransferRegistry::new().expect("initialise transfer staging"));
Self {
config,
browsers,
@@ -72,6 +77,7 @@ impl DaemonState {
abort_registry,
tool_inflight,
session_interrupts,
transfers,
}
}
}
+3
View File
@@ -295,6 +295,7 @@ async fn drive_connection(
for session in state.sessions.purge_browser(&browser_id) {
state.tool_queues.remove(&session.id);
state.session_interrupts.drop_session(&session.id);
state.transfers.release_session(&session.id.0);
debug!(session = %session.id, "purged stale session before browser reconnect");
}
let (tx, mut rx) = mpsc::unbounded_channel::<Frame>();
@@ -407,6 +408,7 @@ async fn drive_connection(
for s in state.sessions.purge_browser(&browser_id) {
state.tool_queues.remove(&s.id);
state.session_interrupts.drop_session(&s.id);
state.transfers.release_session(&s.id.0);
debug!(session = %s.id, "purged session on browser disconnect");
}
} else {
@@ -503,6 +505,7 @@ fn handle_session_window_closed(
&state.session_interrupts,
&session_id,
) {
state.transfers.release_session(&session_id.0);
info!(session = %session_id, "session removed: user closed Agent Window");
} else {
debug!(session = %session_id, "session.window_closed for unknown session id");
+2
View File
@@ -97,6 +97,8 @@ fn dispatch(cli: Cli, format: Format) -> Result<(), CliError> {
Command::Fill(args) => cli::interaction::dispatch_fill(args, format),
Command::Press(args) => cli::interaction::dispatch_press(args, format),
Command::Select(args) => cli::interaction::dispatch_select(args, format),
Command::Upload(args) => cli::upload::dispatch(args, format),
Command::Download(args) => cli::download::dispatch(args, format),
Command::Evaluate(args) => cli::evaluate::dispatch(args, format),
Command::WaitForNavigation(args) => cli::waits::dispatch_wait_for_navigation(args, format),
Command::WaitMs(args) => cli::waits::dispatch_wait_ms(args, format),
+39
View File
@@ -213,6 +213,45 @@ fn rejects_zero_click_count() {
);
}
#[test]
fn parses_upload_with_repeated_files() {
let cli = parse(&[
"bsk",
"upload",
"@e3",
"--file",
"one.png",
"--file",
"two.png",
"--session",
"s1",
]);
let Command::Upload(args) = cli.command else {
panic!("expected upload command");
};
assert_eq!(args.target.as_deref(), Some("@e3"));
assert_eq!(args.files.len(), 2);
}
#[test]
fn parses_download_with_exact_output_policy() {
let cli = parse(&[
"bsk",
"download",
"#export",
"--out",
"result.zip",
"--session",
"s1",
"--overwrite",
]);
let Command::Download(args) = cli.command else {
panic!("expected download command");
};
assert_eq!(args.target.as_deref(), Some("#export"));
assert!(args.overwrite);
}
#[test]
fn parses_record_start_with_browser_and_url() {
let cli = parse(&[
@@ -0,0 +1,56 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "DownloadParams",
"type": "object",
"required": [
"session_id"
],
"properties": {
"browser_relative_dir": {
"description": "Daemon-injected relative directory beneath Chrome's Downloads root.",
"type": [
"string",
"null"
]
},
"max_byte_size": {
"description": "Daemon-injected authoritative transfer size limit. The extension uses it only for early cancellation; daemon import remains the final check.",
"type": [
"integer",
"null"
],
"format": "uint64",
"minimum": 0.0
},
"ref": {
"type": [
"string",
"null"
]
},
"selector": {
"type": [
"string",
"null"
]
},
"session_id": {
"type": "string"
},
"tab_id": {
"type": [
"integer",
"null"
],
"format": "int64"
},
"timeout_ms": {
"type": [
"integer",
"null"
],
"format": "uint32",
"minimum": 0.0
}
}
}
@@ -0,0 +1,62 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "DownloadResult",
"type": "object",
"required": [
"byte_size",
"suggested_filename",
"tab_id"
],
"properties": {
"browser_path": {
"description": "Extension-internal completed Chrome download path, stripped by the daemon.",
"type": [
"string",
"null"
]
},
"byte_size": {
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"danger": {
"type": [
"string",
"null"
]
},
"mime": {
"type": [
"string",
"null"
]
},
"suggested_filename": {
"type": "string"
},
"tab_id": {
"type": "integer",
"format": "int64"
},
"transfer_id": {
"description": "Opaque id returned by the daemon to the CLI.",
"type": [
"string",
"null"
]
},
"used_ref": {
"type": [
"string",
"null"
]
},
"used_selector": {
"type": [
"string",
"null"
]
}
}
}
@@ -0,0 +1,70 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "UploadParams",
"type": "object",
"required": [
"files",
"session_id"
],
"properties": {
"files": {
"type": "array",
"items": {
"$ref": "#/definitions/UploadFile"
}
},
"ref": {
"type": [
"string",
"null"
]
},
"selector": {
"type": [
"string",
"null"
]
},
"session_id": {
"type": "string"
},
"tab_id": {
"type": [
"integer",
"null"
],
"format": "int64"
},
"timeout_ms": {
"type": [
"integer",
"null"
],
"format": "uint32",
"minimum": 0.0
}
},
"definitions": {
"UploadFile": {
"type": "object",
"required": [
"name",
"transfer_id"
],
"properties": {
"name": {
"type": "string"
},
"staged_path": {
"type": [
"string",
"null"
]
},
"transfer_id": {
"type": "string"
}
}
}
}
}
@@ -0,0 +1,33 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "UploadResult",
"type": "object",
"required": [
"file_names",
"tab_id"
],
"properties": {
"file_names": {
"type": "array",
"items": {
"type": "string"
}
},
"tab_id": {
"type": "integer",
"format": "int64"
},
"used_ref": {
"type": [
"string",
"null"
]
},
"used_selector": {
"type": [
"string",
"null"
]
}
}
}
@@ -85,6 +85,10 @@ fn main() {
dump!(PressResult, "tool_press_result");
dump!(SelectParams, "tool_select_params");
dump!(SelectResult, "tool_select_result");
dump!(UploadParams, "tool_upload_params");
dump!(UploadResult, "tool_upload_result");
dump!(DownloadParams, "tool_download_params");
dump!(DownloadResult, "tool_download_result");
dump!(SnapshotParams, "tool_snapshot_params");
dump!(SnapshotResult, "tool_snapshot_result");
+22
View File
@@ -78,6 +78,10 @@ pub enum Method {
ToolPress,
#[serde(rename = "tool.select")]
ToolSelect,
#[serde(rename = "tool.upload")]
ToolUpload,
#[serde(rename = "tool.download")]
ToolDownload,
#[serde(rename = "tool.snapshot")]
ToolSnapshot,
#[serde(rename = "tool.observe")]
@@ -105,6 +109,17 @@ pub enum Method {
#[serde(rename = "tool.record_await")]
ToolRecordAwait,
#[serde(rename = "transfer.begin")]
TransferBegin,
#[serde(rename = "transfer.chunk")]
TransferChunk,
#[serde(rename = "transfer.finish")]
TransferFinish,
#[serde(rename = "transfer.read")]
TransferRead,
#[serde(rename = "transfer.release")]
TransferRelease,
#[serde(rename = "cancel")]
Cancel,
}
@@ -156,6 +171,8 @@ impl Method {
| Method::ToolFill
| Method::ToolPress
| Method::ToolSelect
| Method::ToolUpload
| Method::ToolDownload
| Method::ToolEvaluate
// May navigate via optional `url` and changes Agent Window
// chrome; gate behind pending-interrupt like other writes.
@@ -194,6 +211,11 @@ impl Method {
| Method::SystemPing
| Method::SystemStatus
| Method::BrowserList
| Method::TransferBegin
| Method::TransferChunk
| Method::TransferFinish
| Method::TransferRead
| Method::TransferRelease
| Method::Cancel => MethodEffect::ControlPlane,
}
}
@@ -0,0 +1,176 @@
//! Content-transparent file transfer contracts.
//!
//! Public tool calls carry opaque transfer ids. Filesystem paths are injected
//! by the daemon only on the daemon -> extension hop and are never accepted as
//! an agent-facing source or destination.
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct UploadFile {
pub transfer_id: String,
pub name: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub staged_path: Option<String>,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct UploadParams {
pub session_id: String,
#[serde(
rename = "ref",
alias = "ref_",
default,
skip_serializing_if = "Option::is_none"
)]
pub ref_: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub selector: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub tab_id: Option<i64>,
pub files: Vec<UploadFile>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub timeout_ms: Option<u32>,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct UploadResult {
pub tab_id: i64,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub used_ref: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub used_selector: Option<String>,
pub file_names: Vec<String>,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct DownloadParams {
pub session_id: String,
#[serde(
rename = "ref",
alias = "ref_",
default,
skip_serializing_if = "Option::is_none"
)]
pub ref_: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub selector: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub tab_id: Option<i64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub timeout_ms: Option<u32>,
/// Daemon-injected relative directory beneath Chrome's Downloads root.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub browser_relative_dir: Option<String>,
/// Daemon-injected authoritative transfer size limit. The extension uses
/// it only for early cancellation; daemon import remains the final check.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_byte_size: Option<u64>,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct DownloadResult {
pub tab_id: i64,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub used_ref: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub used_selector: Option<String>,
pub suggested_filename: String,
pub byte_size: u64,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub mime: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub danger: Option<String>,
/// Extension-internal completed Chrome download path, stripped by the daemon.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub browser_path: Option<String>,
/// Opaque id returned by the daemon to the CLI.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub transfer_id: Option<String>,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TransferBeginParams {
pub session_id: String,
pub name: String,
pub byte_size: u64,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TransferBeginResult {
pub transfer_id: String,
pub chunk_size: u32,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TransferChunkParams {
pub transfer_id: String,
pub offset: u64,
pub data_base64: String,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TransferChunkResult {
pub next_offset: u64,
#[serde(default)]
pub eof: bool,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub data_base64: Option<String>,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TransferIdParams {
pub transfer_id: String,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TransferReadyResult {
pub transfer_id: String,
pub byte_size: u64,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TransferReleaseResult {
pub released: bool,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn upload_agent_shape_omits_daemon_path_and_uses_ref_wire_name() {
let value = serde_json::to_value(UploadParams {
session_id: "s1".into(),
ref_: Some("@e3".into()),
selector: None,
tab_id: None,
files: vec![UploadFile {
transfer_id: "tr_1".into(),
name: "image.png".into(),
staged_path: None,
}],
timeout_ms: None,
})
.unwrap();
assert_eq!(value["ref"], "@e3");
assert!(value["files"][0].get("staged_path").is_none());
}
#[test]
fn download_agent_shape_omits_daemon_path() {
let value = serde_json::to_value(DownloadParams {
session_id: "s1".into(),
ref_: None,
selector: Some("#export".into()),
tab_id: None,
timeout_ms: None,
browser_relative_dir: None,
max_byte_size: None,
})
.unwrap();
assert!(value.get("browser_relative_dir").is_none());
assert!(value.get("max_byte_size").is_none());
}
}
+2
View File
@@ -3,6 +3,7 @@
pub mod console;
pub mod dialog;
pub mod emulate;
pub mod file_transfer;
pub mod human_loop;
pub mod interaction;
pub mod navigation;
@@ -18,6 +19,7 @@ pub mod window;
pub use console::*;
pub use dialog::*;
pub use emulate::*;
pub use file_transfer::*;
pub use human_loop::*;
pub use interaction::*;
pub use navigation::*;
+9
View File
@@ -143,6 +143,15 @@ flowchart LR
- No credential storage in bsk — cookies stay in the user's browser profile.
- `evaluate` restricted to Agent Window tabs in sandbox mode.
### File-transfer boundary
- The invoking agent/harness decides whether a transfer is authorized and supplies the task-local source or destination path.
- The CLI is the only component that reads an upload source or writes the final download destination. Before browser dispatch it owns rollback of partially staged uploads; after dispatch, ownership moves to the session because a transport timeout cannot prove that Chrome did not attach the file. Download output becomes visible through one atomic commit, and replacement is opt-in without a pre-delete window. The extension never receives either agent-facing path.
- The daemon is the authority for storage capabilities and limits. It issues opaque session-scoped transfer IDs, stages bounded chunks in a private runtime directory, and injects only private staged upload paths. For download it mints one relative Chrome directory capability. Only after validating the reported path, file type, symlink boundary, and authoritative byte limit does it take ownership of browser-file cleanup and import the bytes.
- The extension owns only the browser transaction. Both tools resolve one `ResolvedActionTarget` and use that same target for protocol setup and the triggering click. Upload arms Chrome's chooser interception before clicking, then accepts either an exact `Page.fileChooserOpened` input node or an independent probe anchored in the trigger node's own document; one verified input is committed with `DOM.setFileInputFiles`. A non-input picker is rejected immediately as unsupported. Download correlates exact-target CDP intent and `chrome.downloads` filename candidates in either arrival order, claims only one unique match, and never cancels an unclaimed candidate.
- Browser-side operations report `effect_state` (`none`, `committed`, or `unknown`), `phase`, and `cleanup_state`. Confirmed success wins over a late cancel; an unknown effect is preserved across timeout or transport loss and must not be retried blindly. A transfer deadline sends cancellation to the extension and keeps the session queue occupied for bounded compensation rather than abandoning an in-flight browser effect.
- Download staging is released after CLI commit. Upload staging remains until session teardown because the page may read an attached file only on a later form submission. Remaining staging is released on session stop/browser disconnect and on daemon startup after a crash. BrowserSkill does not inspect content or decide whether a transfer is appropriate.
## Repository layout
```
+1 -1
View File
@@ -9,7 +9,7 @@
"ext:dev": "pnpm --filter @browser-skill/extension dev",
"ext:test": "pnpm --filter @browser-skill/extension test",
"vom:test": "pnpm --filter @browser-skill/vom test",
"lint": "biome check . && stylelint \"**/*.css\"",
"lint": "biome check . && stylelint \"**/*.css\" && pnpm --filter @wxg-prc-cpg/browser-skill-dsh-plugin typecheck && pnpm --filter @wxg-prc-cpg/browser-skill-dsh-plugin test",
"format": "biome format --write ."
},
"devDependencies": {
@@ -0,0 +1,5 @@
lib/
src/client/bsk-ui.nomodule.css
.bsk-ui.input.css
.bsk-ui.raw.css
src/skill-content.generated.ts
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Tencent
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+195
View File
@@ -0,0 +1,195 @@
# dsh-plugin-browserskill
npm: [`@wxg-prc-cpg/browser-skill-dsh-plugin`](https://www.npmjs.com/package/@wxg-prc-cpg/browser-skill-dsh-plugin)
A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (dsh) tool plugin that exposes
[BrowserSkill](https://github.com/Tencent/BrowserSkill) (`bsk`) browser automation to the model.
Each tool maps to one `bsk <cmd> --json` invocation: the plugin spawns the bsk CLI, parses its
structured JSON output, and returns a canonical typed value. The bsk daemon, browser, and browser
extension keep owning the actual browser control — this package is a thin, well-typed bridge.
## Tools
| Tool | bsk command | Purpose |
| --- | --- | --- |
| `browser_session_start` | `bsk session start` | Open an Agent Window session; optional initial URL, window size, and device emulation preset. Returns the session id and makes it the current session. |
| `browser_session_stop` | `bsk session stop` | Stop a session (the current one by default) and close its Agent Window. Only plugin-created sessions can be stopped. |
| `browser_session_list` | — (registry only) | List the sessions this plugin created, marking the current one. Foreign daemon sessions are never shown. |
| `browser_navigate` | `bsk navigate` | Navigate the active tab, with `waitUntil` / timeout control. |
| `browser_snapshot` | `bsk snapshot` | Indented aria-tree snapshot with `@eN` refs for interaction tools. |
| `browser_observe` | `bsk observe` | Semantic VOM observation (read-only) with `@eN` refs. |
| `browser_click` | `bsk click` | Click a snapshot ref or CSS selector (button / click-count options). |
| `browser_fill` | `bsk fill` | Fill an input / textarea / contenteditable (clears first by default). |
| `browser_press` | `bsk press` | Dispatch a key or combo, optionally focusing a target first. |
| `browser_screenshot` | `bsk screenshot` | PNG capture of the tab or a ref-cropped element; inlines the image when the deployment supports image input, otherwise returns a file path. |
| `browser_emulate` | `bsk emulate` | Apply or clear mobile device emulation on the active tab. |
## Agent skill (progressive disclosure)
Beyond the tools, the plugin publishes the **`browser-skill` agent skill** through the harness's
official skill seam (`ctx.skills.register`): the catalog entry (name + routing description) is
resident in `<available_skills>`, and the body is loaded only when the model invokes the `skill`
tool. The body is assembled at build time from two parts, so there is exactly one source of
truth: a dsh-specific prelude (`skill/prelude.md` — tool↔CLI map, owned-session semantics,
plugin-only overrides) followed verbatim by the canonical CLI skill (`skill/SKILL.md` at the
repo root — the same file `crates/bsk-cli/build.rs` mirrors for the CLI package; workflows,
stop-when-done rules, refs usage, sandbox rules). Registration and every pre-step catalog
snapshot are pure in-memory reads (no disk/process/daemon); compositions without the skill
seam degrade silently.
## Multi-session model
One agent conversation can drive several browser sessions at once:
- `browser_session_start` returns the session id and makes it the **current session**.
- Every operation tool accepts an optional `session` argument. When omitted, the call acts on the
current session (the one most recently started or used); when given, that session becomes current.
- Every tool result echoes the session it actually acted on, so the model never has to guess.
- The number of concurrent sessions started through the plugin is capped (`maxSessions`, default 5).
- Unloading the plugin stops every session it started and kills in-flight bsk processes.
**Ownership boundary**: the bsk daemon may be shared with other agents, terminals, or dsh
instances. The plugin therefore only ever sees and operates on sessions it created itself —
an explicit `session` argument naming a foreign or unknown id is rejected, `browser_session_list`
shows plugin-created sessions only (no daemon-wide view), and stop/unload cleanup can never touch
a session owned by another program.
## Installation
The plugin follows the standard dsh bundle layout (`dsh.bundle` manifest + `cordis.patch.yml`):
```sh
dsh plugin --profile <name> add @wxg-prc-cpg/browser-skill-dsh-plugin
dsh --profile <name>
```
Prerequisite: the `bsk` CLI must be installed and on `PATH`, and the BrowserSkill browser extension
(Chrome or Edge) must be connected — see the
[BrowserSkill README](https://github.com/Tencent/BrowserSkill). When bsk is missing, tool calls fail
with install guidance instead of a bare spawn error.
## Configuration
All fields are optional and validated through the plugin's Schemastery `Config`:
```yaml
# cordis.patch.yml override example
- insert:
- id: browserskill
name: "@wxg-prc-cpg/browser-skill-dsh-plugin"
config:
bskPath: bsk # path to the bsk binary (default: resolve from PATH)
defaultTimeoutMs: 120000
maxSessions: 5
# observationEnabled: true # live PiP/overlay observation (below)
# thumbnailIntervalMs: 1500 # frame cadence while a session is active
# idleIntervalMs: 8000 # idle cadence / recent-activity window
# lazyTools: true # reveal browser_* tools only after the skill is invoked
```
- **`lazyTools` (default `true`)** — the final progressive-disclosure stage: the eleven
`browser_*` tool schemas stay OUT of the system prompt (zero schema tokens) until the
`browser-skill` skill is actually invoked — the skill catalog entry is the only
advertisement. One successful invocation (model tool call, or a `/browser-skill` user
gesture) registers the whole suite for the rest of the process; repeated invocations are
no-ops, and sessions resumed with a past invocation in their durable log reveal the suite
on entry. Set `false` for the legacy always-on registration.
## Observation overlay (PiP mini-window)
When the plugin runs inside the dsh Web UI, an **observation overlay** floats over the app
(registered into the `shell.overlay` seat): a breathing thumbnail per owned session plus its
current action and elapsed time. The card docks at the top-right of the content area (clear of
the composer and the shell's header controls) and wears the **BrowserSkill product family's own
look**: the overlay reuses `@browser-skill/ui` components (`Button`, `cn`) and its oklch design
tokens (`--card`, `--primary`, `--destructive`, `--ring`, …), status dots spec'd after the
extension popup's connection indicator, and Remix icons. The BSK utility sheet is compiled
scoped under the `.bsk-obs` root class (`scripts/build-client-css.mjs`), so the overlay looks
like the BrowserSkill extension without leaking a single selector into the host shell — and the
shell's theme cannot bleed back in.
- **Lifecycle**: hidden while the plugin owns no sessions; appears on the first
`browser_session_start`; disappears when all sessions stop (or the plugin unloads).
- **Focus view**: status row (green/idle/red dot + session + action + mm:ss), the latest page
frame (refreshes every ~1.5s while active, ~8s when idle; the last good frame stays on
stage while the next one loads, and is kept on errors so the card does not flash),
and a compact icon toolbar (Interrupt + Pop out, hover for the label).
- **Interrupt**: one click kills the in-flight bsk command of the focus session (same semantics
as the chat Stop button — the current action fails, the agent run may continue). Strip items
carry their own hover interrupt button.
- **Multi-session strip**: every session gets a tile (mini frame + id + status dot); focus
auto-follows the most recently active session; clicking a tile pins focus (pin badge, click
again to release); errored sessions get a red edge without stealing focus; sessions the daemon
lost are greyed out; prolonged daemon/browser outage shows "browser unavailable" and greys
the interrupt button until captures recover.
- **Drag & resize**: drag the header to move the card; drag any of the four
corners to resize (min 240×180, max 80% of the viewport; no visible grip).
Both are remembered for the page lifetime.
- **Pop out (PiP)**: upgrades the card into a native Document PiP window (requires a user
gesture, per browser rules), sized from the current card; closing the PiP falls back to the
in-page card with state intact. Browsers without Document PiP simply hide the button.
- **Wire**: the host serves `GET /bsk-observation/state`, `GET /bsk-observation/events` (SSE),
`POST /bsk-observation/interrupt`, and `GET /bsk-observation/thumbnail/<attachmentId>` over
the dsh `webServer` route seam (dsh 0.1's Typert Remote pipeline is closed to out-of-tree
packages). All commands for one session — tool calls and frame captures alike — run through a
per-session FIFO, because the daemon accepts only one unfinished command per session.
- **Trust model**: these routes expose live screenshots (and an interrupt write), so they
replicate the browser-trust fence dsh applies to its own `/api` routes: the request Host
must be a loopback authority (`localhost`, `127.0.0.0/8`, `[::1]`), a present Origin must
match the Host, `sec-fetch-site: cross-site` is refused, and POST requires an
`application/json` body (cross-site simple requests can never satisfy that). The channel is
therefore built for **loopback-only serving** — binding the dsh web server to `0.0.0.0` and
reaching it through a LAN address will (deliberately) fail the fence; do not put these
routes behind a non-loopback reverse proxy without adding your own authentication.
- Configure with `observationEnabled` / `thumbnailIntervalMs` / `idleIntervalMs`.
## Behavior notes
- **Cancellation**: aborting a tool call (`exec.signal`) kills the underlying bsk child process,
matching BrowserSkill's cooperative tool-cancellation model.
- **UI cards**: calls render as terminal cards (command line as title, output as the completed
card). Screenshots additionally attach the image itself when the host mounts an attachment store
and the active model route declares image input; otherwise the PNG path is returned.
- **Web UI toolview (browser half)**: the package is dual-face. `dsh.client` (platform `web`) ships
`lib/client.js`, which registers a keyed `tool.call.toolview` view for `browser_screenshot`. The
custom view keeps the terminal block (command + output) and, when the settled result carries an
image block, resolves the durable attachment through the client session's authorized
`readAttachment` RPC and renders it with the shared `MessageImage` thumbnail/lightbox atoms.
Every other `browser_*` tool keeps the stock terminal card. The bundle follows the dsh client
contract: a CJS closure factory handed to `window.__ModuleLoader__.load`, platform modules
(`react`, `dsh-client-ui-*`) external, everything else inlined, CSS Modules compiled by
lightningcss.
- **Errors**: non-zero bsk exits surface the CLI's JSON error envelope (`code`, `message`, `hint`)
so the model gets the daemon's actionable guidance.
- **Long-running work** (e.g. `bsk record`) is not backgrounded via `ctx.jobs` yet — tracked as a
follow-up.
## Development
```sh
pnpm install
pnpm --filter @wxg-prc-cpg/browser-skill-dsh-plugin typecheck
pnpm --filter @wxg-prc-cpg/browser-skill-dsh-plugin test # unit tests mock bsk; no browser needed
pnpm --filter @wxg-prc-cpg/browser-skill-dsh-plugin build # tsdown -> lib/
```
## Publishing
The GitHub Actions workflow **Release dsh plugin** publishes this package to npm
as `@wxg-prc-cpg/browser-skill-dsh-plugin`. The Cordis plugin id stays
`dsh-plugin-browserskill`.
Trigger it by pushing a tag that matches `package.json`'s `version`:
```sh
git tag dsh-plugin-v0.1.0
git push origin dsh-plugin-v0.1.0
```
Or run the workflow from the Actions tab (`workflow_dispatch`). The job reads the
`NPM_TOKEN` secret from the `npm-publish` GitHub Environment.
## License
MIT
@@ -0,0 +1,5 @@
- insert:
- id: browserskill
# Must be the published package name — the loader imports this specifier.
# Quoted because YAML reserves a leading '@' in plain scalars.
name: "@wxg-prc-cpg/browser-skill-dsh-plugin"
@@ -0,0 +1,130 @@
{
"name": "@wxg-prc-cpg/browser-skill-dsh-plugin",
"description": "DeepSeek Harness tool plugin that exposes BrowserSkill (bsk) browser automation to the model",
"version": "0.1.1",
"license": "MIT",
"publishConfig": {
"access": "public",
"registry": "https://registry.npmjs.org/"
},
"repository": {
"type": "git",
"url": "git+https://github.com/Tencent/BrowserSkill.git",
"directory": "packages/dsh-plugin-browserskill"
},
"homepage": "https://github.com/Tencent/BrowserSkill/tree/main/packages/dsh-plugin-browserskill#readme",
"bugs": {
"url": "https://github.com/Tencent/BrowserSkill/issues"
},
"keywords": [
"dsh-plugin",
"deepseek-harness",
"browserskill",
"browser-automation"
],
"type": "module",
"main": "lib/index.mjs",
"types": "lib/index.d.mts",
"exports": {
".": {
"types": "./lib/index.d.mts",
"default": "./lib/index.mjs"
},
"./client": {
"default": "./lib/client.cjs"
},
"./package.json": "./package.json"
},
"files": [
"lib",
"cordis.patch.yml",
"LICENSE"
],
"dsh": {
"bundle": {
"patch": "./cordis.patch.yml"
},
"client": {
"platform": "web",
"inject": [
"@deepseek-ai/dsh-client-runtime",
"@deepseek-ai/dsh-client-ui-tool",
"@deepseek-ai/dsh-client-ui-layout"
]
}
},
"scripts": {
"build": "node scripts/build-client-css.mjs && node scripts/build-skill-content.mjs && tsdown",
"typecheck": "pnpm run build:skill && tsc --noEmit",
"test": "pnpm run build:css && pnpm run build:skill && vitest run",
"build:css": "node scripts/build-client-css.mjs",
"build:skill": "node scripts/build-skill-content.mjs",
"prepack": "pnpm run build",
"lint:publint": "publint"
},
"peerDependencies": {
"@deepseek-ai/cordis": "^4.0.1",
"@deepseek-ai/dsh-attachment": "^0.1.0-rc.6",
"@deepseek-ai/dsh-client-runtime": "^0.1.0-rc.6",
"@deepseek-ai/dsh-client-ui-attachment": "^0.1.0-rc.6",
"@deepseek-ai/dsh-client-ui-primitives": "^0.1.0-rc.6",
"@deepseek-ai/dsh-client-ui-tool": "^0.1.0-rc.6",
"@deepseek-ai/dsh-llm": "^0.1.0-rc.6",
"@deepseek-ai/dsh-tools": "^0.1.0-rc.6",
"@deepseek-ai/schemastery": "^3.18.1",
"react": "^18.2.0"
},
"peerDependenciesMeta": {
"@deepseek-ai/dsh-attachment": {
"optional": true
},
"@deepseek-ai/dsh-client-runtime": {
"optional": true
},
"@deepseek-ai/dsh-client-ui-attachment": {
"optional": true
},
"@deepseek-ai/dsh-client-ui-primitives": {
"optional": true
},
"@deepseek-ai/dsh-client-ui-tool": {
"optional": true
},
"@deepseek-ai/dsh-llm": {
"optional": true
},
"react": {
"optional": true
}
},
"devDependencies": {
"@deepseek-ai/cordis": "^4.0.1",
"@deepseek-ai/dsh-agent": "^0.1.0-rc.6",
"@deepseek-ai/dsh-attachment": "^0.1.0-rc.6",
"@deepseek-ai/dsh-client-runtime": "^0.1.0-rc.6",
"@deepseek-ai/dsh-client-ui-attachment": "^0.1.0-rc.6",
"@deepseek-ai/dsh-client-ui-layout": "^0.1.0-rc.6",
"@deepseek-ai/dsh-client-ui-primitives": "^0.1.0-rc.6",
"@deepseek-ai/dsh-client-ui-tool": "^0.1.0-rc.6",
"@deepseek-ai/dsh-llm": "^0.1.0-rc.6",
"@deepseek-ai/dsh-tools": "^0.1.0-rc.6",
"@deepseek-ai/schemastery": "^3.18.1",
"@tailwindcss/cli": "^4.3.3",
"@testing-library/react": "^16.1.0",
"@types/node": "^22.10.0",
"@types/react": "~18.3.1",
"@types/react-dom": "^18.3.0",
"happy-dom": "^15.11.7",
"lightningcss": "^1.30.0",
"postcss": "^8.5.0",
"postcss-prefix-selector": "^2.1.1",
"react": "^18.2.0",
"react-dom": "^18.2.0",
"tailwindcss": "^4.3.3",
"tsdown": "^0.22.14",
"typescript": "^5.8.0",
"vitest": "^4.1.6",
"@browser-skill/ui": "workspace:*",
"@remixicon/react": "^4.9.0"
}
}
@@ -0,0 +1,72 @@
#!/usr/bin/env node
/**
* Build the scoped BrowserSkill-UI utility sheet for the dsh client bundle.
*
* The overlay reuses @browser-skill/ui components (shadcn-style, Tailwind v4
* utility classes). Those classes need a stylesheet — but a global utility
* sheet inside the dsh page would collide with the host UI, so the compiled
* output is scope-prefixed under `.bsk-obs` (the overlay root class):
* 1. tailwindcss compiles the utilities used by packages/ui + our client;
* 2. postcss-prefix-selector nests every rule under .bsk-obs.
* Token values themselves live in src/client/bsk-tokens.nomodule.css
* (hand-mirrored from packages/ui/src/styles/tailwind.css).
*
* Output: src/client/bsk-ui.nomodule.css (gitignored build artifact).
*/
import { execFileSync } from "node:child_process";
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import postcss from "postcss";
import prefixer from "postcss-prefix-selector";
const pkgRoot = dirname(dirname(fileURLToPath(import.meta.url)));
const input = join(pkgRoot, ".bsk-ui.input.css");
const rawOut = join(pkgRoot, ".bsk-ui.raw.css");
const finalOut = join(pkgRoot, "src/client/bsk-ui.nomodule.css");
// The canonical sheet can't be imported directly (its own `@import
// "tailwindcss"` must resolve from a package that has it), so we inline the
// canonical @theme block read from packages/ui at build time.
const canonical = readFileSync(join(pkgRoot, "..", "ui", "src", "styles", "tailwind.css"), "utf8");
const themeBlock = canonical.match(/@theme inline \{[\s\S]*?\n\}/)?.[0];
if (themeBlock === undefined) throw new Error("canonical @theme inline block not found");
writeFileSync(
input,
[
'@import "tailwindcss";',
"@custom-variant dark (&:is(.dark *));",
themeBlock,
'@source "../ui/src";',
'@source "./src/client";',
"",
].join("\n"),
);
execFileSync(
process.execPath,
[
join(pkgRoot, "node_modules", "@tailwindcss", "cli", "dist", "index.mjs"),
"-i",
input,
"-o",
rawOut,
],
{ cwd: pkgRoot, stdio: "inherit" },
);
const result = await postcss([
prefixer({
prefix: ".bsk-obs",
transform(prefix, selector, prefixedSelector) {
// Root-level anchors collapse onto the scope root itself.
if (selector === ":root" || selector === "html" || selector === "body") return prefix;
if (selector.startsWith(prefix)) return selector;
return prefixedSelector;
},
}),
]).process(readFileSync(rawOut, "utf8"), { from: rawOut });
mkdirSync(dirname(finalOut), { recursive: true });
writeFileSync(finalOut, result.css);
console.log(`bsk-ui sheet: ${result.css.length} bytes -> ${finalOut}`);
@@ -0,0 +1,44 @@
// Assemble the plugin's skill content: a dsh-specific routing prelude, the
// canonical CLI skill body verbatim (repo-root skill/SKILL.md — the same source
// crates/bsk-cli's build.rs mirrors), then a dsh routing reminder postlude. The
// original's YAML frontmatter is stripped: name/description are supplied by the
// registration.
// Output: src/skill-content.generated.ts (gitignored, built before tsdown and
// before vitest) so the markdown ships inlined in the bundle and registration
// is a pure in-memory read — never a per-step disk access.
import { readFileSync, writeFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
const here = dirname(fileURLToPath(import.meta.url));
const pkg = join(here, "..");
const prelude = readFileSync(join(pkg, "skill", "prelude.md"), "utf8");
const upstream = readFileSync(join(pkg, "..", "..", "skill", "SKILL.md"), "utf8");
const postlude = readFileSync(join(pkg, "skill", "postlude.md"), "utf8");
function stripFrontmatter(markdown) {
if (!markdown.startsWith("---\n")) return markdown;
const end = markdown.indexOf("\n---\n", 4);
return end === -1 ? markdown : markdown.slice(end + 5).replace(/^\s*\n/, "");
}
const NAME = "browser-skill";
const DESCRIPTION =
"Browser automation against the user's logged-in Chromium through this plugin's injected browser_* tools; never invoke bsk through bash or shell. " +
"tools (bsk engine): visit and read pages, fill forms, click through flows, scrape data, " +
"smoke-test UI changes. Use when a task needs a real browser; requires the bsk CLI and the " +
"browser-skill extension connected.";
const content = `${prelude.trim()}\n\n${stripFrontmatter(upstream).trim()}\n\n${postlude.trim()}\n`;
const banner =
"// GENERATED by scripts/build-skill-content.mjs — do not edit.\n" +
"// Sources: skill/prelude.md + ../../../skill/SKILL.md + skill/postlude.md.\n";
writeFileSync(
join(pkg, "src", "skill-content.generated.ts"),
`${banner}export const BSK_SKILL_NAME = ${JSON.stringify(NAME)};\n` +
`export const BSK_SKILL_DESCRIPTION = ${JSON.stringify(DESCRIPTION)};\n` +
`export const BSK_SKILL_MARKDOWN = ${JSON.stringify(content)};\n`,
);
console.log(`skill-content.generated.ts written (${content.length} chars)`);
@@ -0,0 +1,10 @@
---
## DSH routing reminder
The CLI-oriented reference above describes behavior, not the execution mechanism in
DeepSeek Harness. For every browser operation, call the injected `browser_*` model
tool. Do not translate these examples into `bash` commands and do not execute `bsk`
directly; the plugin invokes `bsk` internally. In particular, always begin with
`browser_session_start`, pass its returned session id to subsequent `browser_*`
tools, and finish with `browser_session_stop`.
@@ -0,0 +1,71 @@
# browser-skill (dsh edition)
## DSH tool-routing requirement (highest priority)
This skill is running inside DeepSeek Harness, where the plugin has injected
structured `browser_*` model tools. **Always call those injected `browser_*` tools
for browser work. Never use `bash`, `shell`, `exec`, or another tool to invoke the
`bsk` CLI directly.** The CLI command names shown below are only conceptual
mappings for understanding the tools; they are not instructions to run commands.
If a `browser_*` tool is available, using the CLI directly is incorrect because it
bypasses plugin session ownership, observation UI, tool cards, and cleanup.
Use the injected tool whose name matches the operation, for example:
`browser_session_start`, `browser_navigate`, `browser_observe`, `browser_snapshot`,
`browser_click`, `browser_fill`, `browser_press`, `browser_screenshot`, and
`browser_session_stop`. Pass arguments as the tool schema specifies (not CLI flags).
Drive the user's **real Chromium browser** (logins, cookies, Agent Window isolation)
through this plugin's `browser_*` tools. Each injected tool is implemented by the
plugin using the `bsk` engine internally — the model must not invoke that engine
itself. The workflow, judgment rules, and safety constraints remain identical to
the canonical CLI skill.
> **Tool availability**: invoking this skill (which you just did) makes the plugin's
> `browser_*` tools available for the rest of the session — they are injected on
> successful skill load. Until then they are deliberately hidden, so do not attempt
> a browser action before loading this skill.
## Tool ↔ CLI map
The tools below are thin, structured wrappers over the bsk CLI verbs of the same
name; `session` parameters take the 4-letter session id (no `--session` flag
spelling here):
| dsh tool | bsk equivalent |
|---|---|
| `browser_session_start` / `browser_session_stop` / `browser_session_list` | `bsk session start/stop/list` |
| `browser_navigate` | `bsk navigate` |
| `browser_snapshot` / `browser_observe` | `bsk snapshot` / `bsk observe` |
| `browser_click` / `browser_fill` / `browser_press` | `bsk click` / `bsk fill` / `bsk press` |
| `browser_screenshot` | `bsk screenshot` |
| `browser_emulate` | `bsk emulate` |
(`bsk console` / `bsk network` have no tool counterpart in this plugin yet.)
Plugin-specific semantics that override or narrow the CLI rules below:
- **Owned sessions only**: this plugin sees and acts on just the sessions IT
started via `browser_session_start`. `browser_session_list` shows only those
(with the current marker); passing a session id created by another program
(or person) sharing the bsk daemon is refused outright — never retry those
with the same id.
- **Current session**: omitted `session` parameters resolve to the most
recently started or used OWNED session. With several sessions in flight,
pass `session` explicitly to avoid surprises.
- **Screenshots**: when the model route accepts images, `browser_screenshot`
returns the PNG inline as an image attachment; otherwise it returns the
saved file path.
- **`browser_emulate mobile`**: requires `width`+`height` — the bsk daemon
refuses `--mobile` without viewport dimensions.
- **Live observation**: in the dsh Web UI, every owned session is watched by
the floating overlay (thumbnails, action, interrupt) — observational side
traffic is free and never steals your command turn; just keep acting.
The canonical CLI skill body follows verbatim (source of truth: `skill/SKILL.md`
in the BrowserSkill repo, mirrored by build into `crates/bsk-cli/skill/SKILL.md`).
Read `bsk <verb> --session <id>` as "the `browser_<verb>` tool with
`session: <id>`"; everything else — when to use, the stop-when-done rules, the
interaction loop, refs, sandbox rules — applies unchanged.
---
@@ -0,0 +1,370 @@
/* Observation overlay skin, BrowserSkill family: colors/radius ride the BSK
design tokens (--card/--border/--primary/--destructive/--ring/--radius,
defined on the .bsk-obs scope root by bsk-tokens.nomodule.css — values
mirrored from packages/ui/src/styles/tailwind.css). Layout-only here;
component chrome (buttons etc.) comes from @browser-skill/ui plus the
scope-prefixed utility sheet. The shell overlay layer is click-through, so
the card and capsule opt back into pointer events themselves. */
.card {
position: fixed;
z-index: 40;
display: flex;
flex-direction: column;
pointer-events: auto;
background: var(--card);
border: 1px solid var(--border);
border-radius: var(--radius);
box-shadow: 0 8px 28px rgb(0 0 0 / 18%);
overflow: hidden;
color: var(--card-foreground);
}
.body {
display: flex;
flex-direction: column;
height: 100%;
min-height: 0;
font-size: 12px;
background: var(--card);
color: var(--card-foreground);
}
.header {
display: flex;
align-items: center;
gap: 6px;
padding: 6px 8px;
cursor: move;
user-select: none;
touch-action: none;
border-bottom: 1px solid var(--border);
}
.status-text {
flex: 1;
min-width: 0;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
font-weight: 500;
}
.icon-button {
flex: none;
display: inline-flex;
align-items: center;
border: none;
background: none;
padding: 2px;
border-radius: calc(var(--radius) - 6px);
font: inherit;
color: var(--muted-foreground);
cursor: pointer;
}
.icon-button:hover {
color: var(--foreground);
background: var(--accent);
}
.stage {
position: relative;
flex: 1;
min-height: 0;
display: flex;
align-items: center;
justify-content: center;
background: var(--background);
overflow: hidden;
}
/* Fade only on first mount. The <img> is reused across frames (src swap,
no React key) so this animation must not replay on every breath. */
.thumb {
max-width: 100%;
max-height: 100%;
object-fit: contain;
animation: bsk-obs-fade-in 240ms ease-out;
}
@keyframes bsk-obs-fade-in {
from {
opacity: 0;
}
to {
opacity: 1;
}
}
.placeholder {
color: var(--muted-foreground);
font-size: 12px;
}
.badge {
position: absolute;
top: 6px;
right: 6px;
width: 16px;
height: 16px;
border-radius: 50%;
display: inline-flex;
align-items: center;
justify-content: center;
color: oklch(0.7 0.19 60);
border: 1px solid oklch(0.7 0.19 60);
background: var(--card);
}
.actions {
position: relative;
z-index: 3;
display: flex;
align-items: center;
justify-content: space-between;
gap: 4px;
/* A little inset so icons are not flush with the resize corners. */
padding: 4px 8px;
min-height: 32px;
border-top: 1px solid var(--border);
}
.tool-wrap {
position: relative;
display: inline-flex;
}
.tool-button {
flex: none;
display: inline-flex;
align-items: center;
justify-content: center;
width: 28px;
height: 28px;
padding: 0;
border: none;
border-radius: calc(var(--radius) - 4px);
background: none;
color: var(--muted-foreground);
cursor: pointer;
}
.tool-button:disabled {
opacity: 0.4;
cursor: default;
}
.tool-button:focus-visible {
outline: 2px solid var(--ring);
outline-offset: 2px;
}
.tool-danger {
color: var(--destructive);
}
/* Hover the wrap (not just :hover on the button) so the tooltip target and
the left-corner resize overlay still light the icon up. */
.tool-wrap:hover .tool-button:not(:disabled) {
color: var(--foreground);
background: var(--accent);
}
.tool-wrap:hover .tool-danger:not(:disabled) {
color: var(--destructive);
background: color-mix(in oklch, var(--destructive) 18%, var(--card));
}
.tool-wrap:hover .tool-button:disabled {
background: var(--accent);
}
.hint {
position: absolute;
bottom: calc(100% + 4px);
left: 0;
width: max-content;
max-width: 220px;
padding: 4px 6px;
border-radius: calc(var(--radius) - 4px);
background: var(--card);
border: 1px solid var(--border);
color: var(--muted-foreground);
font-size: 11px;
line-height: 1.35;
box-shadow: 0 4px 14px rgb(0 0 0 / 16%);
z-index: 41;
}
.hint[data-align="end"] {
left: auto;
right: 0;
}
/* Invisible hit targets — no grip glyph; all four corners resize. */
.resize-handle {
position: absolute;
width: 16px;
height: 16px;
touch-action: none;
z-index: 2;
}
.resize-handle[data-corner="nw"] {
top: 0;
left: 0;
cursor: nwse-resize;
}
.resize-handle[data-corner="ne"] {
top: 0;
right: 0;
cursor: nesw-resize;
}
.resize-handle[data-corner="sw"] {
bottom: 0;
left: 0;
cursor: nesw-resize;
}
.resize-handle[data-corner="se"] {
bottom: 0;
right: 0;
cursor: nwse-resize;
}
.capsule {
position: fixed;
right: 16px;
top: 64px;
z-index: 40;
display: inline-flex;
align-items: center;
gap: 6px;
padding: 6px 12px;
pointer-events: auto;
border-radius: 999px;
border: 1px solid var(--border);
background: var(--card);
box-shadow: 0 4px 14px rgb(0 0 0 / 16%);
font-size: 12px;
color: var(--muted-foreground);
cursor: pointer;
}
.capsule:hover {
background: var(--accent);
}
.strip {
display: flex;
gap: 6px;
padding: 4px 8px;
overflow-x: auto;
border-top: 1px solid var(--border);
}
.strip-item {
position: relative;
flex: none;
border-radius: calc(var(--radius) - 6px);
border: 1px solid var(--border);
}
.strip-item[data-focused] {
border-color: var(--primary);
}
.strip-item[data-state="error"] {
border-color: var(--destructive);
}
.strip-item[data-state="dead"] {
opacity: 0.45;
}
.strip-main {
display: flex;
align-items: center;
gap: 4px;
padding: 3px 6px;
border: none;
border-radius: calc(var(--radius) - 6px);
background: none;
font: inherit;
font-size: 11px;
color: var(--muted-foreground);
cursor: pointer;
}
.strip-main:hover {
background: var(--accent);
}
.strip-thumb {
width: 28px;
height: 20px;
overflow: hidden;
border-radius: 3px;
background: var(--background);
display: inline-flex;
}
.strip-thumb img {
width: 100%;
height: 100%;
object-fit: cover;
}
.strip-thumb-empty {
width: 100%;
height: 100%;
}
.strip-id {
max-width: 64px;
overflow: hidden;
text-overflow: ellipsis;
}
.pin-badge {
color: var(--primary);
}
.strip-interrupt {
position: absolute;
top: -7px;
right: -7px;
width: 16px;
height: 16px;
border-radius: 50%;
display: inline-flex;
align-items: center;
justify-content: center;
border: 1px solid var(--destructive);
background: var(--card);
color: var(--destructive);
padding: 0;
cursor: pointer;
}
.strip-interrupt:disabled {
opacity: 0.4;
cursor: default;
}
.icon-button:focus-visible,
.capsule:focus-visible,
.strip-main:focus-visible,
.strip-interrupt:focus-visible {
outline: 2px solid var(--ring);
outline-offset: 2px;
}
@media (prefers-reduced-motion: reduce) {
.thumb {
animation: none;
}
}
@@ -0,0 +1,612 @@
// Observation overlay: the default in-app carrier for live session watching.
// Expanded = draggable/resizable floating card (top-right, out of the
// composer's way); collapsed = a status capsule; "pop out" upgrades the same
// content to a native Document PiP window (user gesture required by the
// browser). Multi-session renders a meeting-style strip under the focus view.
// Visuals follow the BrowserSkill product family: @browser-skill/ui components
// and oklch tokens on a .bsk-obs scope root, so the card reads as BSK's own
// surface without leaking styles into (or inheriting themes from) the shell.
import { cn } from "@browser-skill/ui";
import {
RiArrowDownSLine,
RiErrorWarningLine,
RiPictureInPicture2Line,
RiPushpinFill,
RiStopCircleLine,
} from "@remixicon/react";
// remixicon's component types target @types/react 19 while the dsh shell
// runs react 18 — a compile-time-only recast keeps the 18 typecheck happy.
type IconComponent = (props: { size?: number | string; className?: string }) => ReactNode;
const asIcon = (component: unknown): IconComponent => component as IconComponent;
const IconStop = asIcon(RiStopCircleLine);
const IconPip = asIcon(RiPictureInPicture2Line);
const IconDown = asIcon(RiArrowDownSLine);
const IconWarn = asIcon(RiErrorWarningLine);
const IconPin = asIcon(RiPushpinFill);
import {
type ReactNode,
type PointerEvent as ReactPointerEvent,
useCallback,
useEffect,
useRef,
useState,
useSyncExternalStore,
} from "react";
import { createPortal } from "react-dom";
import type { SessionObservation } from "../observation";
import css from "./ObservationOverlay.module.css";
import type { ObservationClientStore } from "./observation-store";
/** Minimal Document PiP surface (TS lib.dom lacks it). */
interface DocumentPip {
requestWindow(options?: { width?: number; height?: number }): Promise<Window>;
}
interface Point {
x: number;
y: number;
}
interface Size {
w: number;
h: number;
}
const DEFAULT_SIZE: Size = { w: 320, h: 240 };
const MIN_SIZE: Size = { w: 240, h: 180 };
const EDGE_MARGIN = 16;
/** Default dock: top-right, just under the shell's top bar (no spacing tokens exist in dsh 0.1). */
const TOP_OFFSET = 64;
function pipApi(): DocumentPip | undefined {
if (typeof window === "undefined") return undefined;
return (window as unknown as { documentPictureInPicture?: DocumentPip }).documentPictureInPicture;
}
function clampSize(size: Size, viewport: Size): Size {
const maxW = viewport.w * 0.8;
const maxH = viewport.h * 0.8;
return {
w: Math.min(Math.max(size.w, MIN_SIZE.w), maxW),
h: Math.min(Math.max(size.h, MIN_SIZE.h), maxH),
};
}
function clampPos(pos: Point, size: Size, viewport: Size): Point {
return {
x: Math.min(Math.max(pos.x, 0), Math.max(0, viewport.w - size.w)),
y: Math.min(Math.max(pos.y, 0), Math.max(0, viewport.h - size.h)),
};
}
export type ResizeCorner = "nw" | "ne" | "sw" | "se";
/** Grow/shrink from one corner, keeping the opposite corner planted. */
export function applyResize(
base: Point & Size,
corner: ResizeCorner,
dx: number,
dy: number,
viewport: Size,
): { pos: Point; size: Size } {
const nextW = corner === "ne" || corner === "se" ? base.w + dx : base.w - dx;
const nextH = corner === "sw" || corner === "se" ? base.h + dy : base.h - dy;
const size = clampSize({ w: nextW, h: nextH }, viewport);
const x = corner === "nw" || corner === "sw" ? base.x + base.w - size.w : base.x;
const y = corner === "nw" || corner === "ne" ? base.y + base.h - size.h : base.y;
return { pos: clampPos({ x, y }, size, viewport), size };
}
const CORNER_LABEL: Record<ResizeCorner, string> = {
nw: "top left",
ne: "top right",
sw: "bottom left",
se: "bottom right",
};
function formatElapsed(sinceMs: number, nowMs: number): string {
const total = Math.max(0, Math.floor((nowMs - sinceMs) / 1000));
const mm = String(Math.floor(total / 60)).padStart(2, "0");
const ss = String(total % 60).padStart(2, "0");
return `${mm}:${ss}`;
}
/**
* Auto-follow focus: the most recently touched session, but never yank focus
* to a session whose latest action failed (errors flag the strip item, they
* do not steal the stage) or one already reported dead.
*/
export function focusOf(sessions: readonly SessionObservation[]): SessionObservation | undefined {
if (sessions.length === 0) return undefined;
const byRecency = [...sessions].sort((a, b) => b.since - a.since);
const healthy = byRecency.find((s) => s.lastError === undefined && s.dead !== true);
return healthy ?? byRecency[0];
}
export function statusOf(obs: SessionObservation): "active" | "idle" | "error" {
if (obs.lastError !== undefined && obs.action === "idle") return "error";
return obs.action === "idle" ? "idle" : "active";
}
/** Compact toolbar icon: no label, hover bubble for the name / semantics. */
function IconAction(props: {
label: string;
hint: string;
disabled?: boolean;
danger?: boolean;
align?: "start" | "end";
onClick: () => void;
children: ReactNode;
}) {
const [open, setOpen] = useState(false);
return (
<span
className={css["tool-wrap"]}
onPointerEnter={() => setOpen(true)}
onPointerLeave={() => setOpen(false)}
>
<button
type="button"
className={cn(css["tool-button"], props.danger === true && css["tool-danger"])}
disabled={props.disabled}
aria-label={props.label}
onClick={props.onClick}
>
{props.children}
</button>
{open ? (
<span className={css.hint} data-align={props.align} role="tooltip">
{props.hint}
</span>
) : null}
</span>
);
}
/** Flat status dot, specced after the BSK popup's ConnectionStatusIndicator. */
function StatusDot({ state }: { state: "active" | "idle" | "error" | "dead" }) {
const color =
state === "active"
? "bg-emerald-500"
: state === "error"
? "bg-red-500"
: state === "dead"
? "bg-amber-500"
: "bg-muted-foreground/40";
return (
<span
className={cn("size-2 shrink-0 rounded-full ring-2 ring-background", color)}
data-state={state}
aria-hidden
/>
);
}
/** One strip item: mini frame, id, status dot, hover interrupt, pin toggle. */
function StripItem(props: {
store: ObservationClientStore;
obs: SessionObservation;
pinned: boolean;
focused: boolean;
onTogglePin: (sessionId: string) => void;
}) {
const { store, obs, pinned, focused, onTogglePin } = props;
const [hover, setHover] = useState(false);
const thumbId = obs.thumbnailAttachmentId;
useEffect(() => {
store.ensureThumbnail(thumbId);
}, [store, thumbId]);
const thumb = store.getSnapshot().displayFrames[obs.sessionId];
const state = obs.dead === true ? "dead" : statusOf(obs);
return (
<div
className={css["strip-item"]}
data-state={state}
data-focused={focused || undefined}
data-pinned={pinned || undefined}
onPointerEnter={() => setHover(true)}
onPointerLeave={() => setHover(false)}
>
<button
type="button"
className={css["strip-main"]}
aria-label={`${pinned ? "Unpin" : "Pin"} session ${obs.sessionId}`}
aria-pressed={pinned}
onClick={() => onTogglePin(obs.sessionId)}
>
<span className={css["strip-thumb"]}>
{thumb?.url !== undefined ? (
<img src={thumb.url} alt="" />
) : (
<span className={css["strip-thumb-empty"]} />
)}
</span>
<span className={css["strip-id"]}>{obs.sessionId}</span>
<StatusDot state={state} />
{pinned ? <IconPin size={9} className={css["pin-badge"]} aria-label="pinned" /> : null}
</button>
{hover && !pinned && obs.dead !== true ? (
<button
type="button"
className={css["strip-interrupt"]}
aria-label={`Interrupt session ${obs.sessionId}`}
disabled={obs.action === "idle"}
onClick={(event) => {
event.stopPropagation();
void store.interrupt(obs.sessionId);
}}
>
<IconStop size={10} />
</button>
) : null}
</div>
);
}
/** The floating card / PiP shared content. */
function OverlayBody(props: {
store: ObservationClientStore;
focus: SessionObservation | undefined;
sessions: readonly SessionObservation[];
available: boolean;
pinnedId: string | null;
onTogglePin: (sessionId: string) => void;
now: number;
onPopOut?: (() => void) | undefined;
onCollapse?: (() => void) | undefined;
inPip: boolean;
onHeaderPointerDown?: (event: ReactPointerEvent<HTMLDivElement>) => void;
}) {
const {
store,
focus,
sessions,
available,
pinnedId,
onTogglePin,
now,
onPopOut,
onCollapse,
inPip,
onHeaderPointerDown,
} = props;
const [interrupting, setInterrupting] = useState(false);
const thumbId = focus?.thumbnailAttachmentId;
useEffect(() => {
store.ensureThumbnail(thumbId);
}, [store, thumbId]);
// Paint the last good frame while the next attachment decodes — swapping
// to a placeholder (and remounting <img> with a fade) is what made the
// card flash on every breath / action-end capture.
const thumb =
focus !== undefined ? store.getSnapshot().displayFrames[focus.sessionId] : undefined;
const displayUrl = thumb?.url;
const canInterrupt =
available &&
!interrupting &&
focus !== undefined &&
focus.action !== "idle" &&
focus.dead !== true;
const onInterrupt = (): void => {
if (!canInterrupt || focus === undefined) return;
setInterrupting(true);
void store.interrupt(focus.sessionId).finally(() => setInterrupting(false));
};
const statusText = !available
? "browser unavailable"
: focus === undefined
? "no session"
: `${focus.sessionId} · ${focus.action === "idle" ? "idle" : focus.action} · ${formatElapsed(focus.since, now)}`;
const state = !available ? "error" : focus !== undefined ? statusOf(focus) : "idle";
return (
<div className={cn(css.body, "bsk-obs")} data-state={state} data-in-pip={inPip || undefined}>
<div
className={css.header}
data-testid="obs-header"
onPointerDown={onHeaderPointerDown}
role="presentation"
>
<StatusDot state={state === "error" ? "error" : state === "active" ? "active" : "idle"} />
<span className={css["status-text"]}>{statusText}</span>
{onCollapse !== undefined ? (
<button
type="button"
className={css["icon-button"]}
aria-label="Collapse"
onClick={onCollapse}
>
<IconDown size={14} />
</button>
) : null}
</div>
<div className={css.stage}>
{displayUrl !== undefined ? (
<img
className={css.thumb}
src={displayUrl}
alt={`session ${focus?.sessionId ?? ""} view`}
/>
) : (
<div className={css.placeholder}>
{!available
? "last frame kept"
: thumb?.status === "error"
? "frame unavailable"
: "waiting for page"}
</div>
)}
{thumb?.status === "error" ? (
<span className={css.badge} aria-label="thumbnail failed">
<IconWarn size={12} />
</span>
) : null}
</div>
{sessions.length >= 2 ? (
<div className={css.strip} data-testid="obs-strip" role="list">
{sessions.map((obs) => (
<StripItem
key={obs.sessionId}
store={store}
obs={obs}
pinned={pinnedId === obs.sessionId}
focused={focus?.sessionId === obs.sessionId}
onTogglePin={onTogglePin}
/>
))}
</div>
) : null}
<div className={css.actions}>
<IconAction
label={interrupting ? "Interrupting…" : "Interrupt the current browser action"}
hint="Stop the current browser action."
disabled={!canInterrupt}
danger
onClick={onInterrupt}
>
<IconStop size={16} />
</IconAction>
{onPopOut !== undefined ? (
<IconAction
label="Pop out into a mini window"
hint="Pop out into a mini window"
align="end"
onClick={onPopOut}
>
<IconPip size={16} />
</IconAction>
) : null}
</div>
</div>
);
}
/** Clone the host document's style/link nodes into a PiP window. */
function cloneStylesInto(pipWindow: Window): void {
for (const node of document.querySelectorAll('link[rel="stylesheet"], style')) {
pipWindow.document.head.appendChild(node.cloneNode(true));
}
}
export function ObservationOverlay({ store }: { store: ObservationClientStore }) {
const snapshot = useSyncExternalStore(store.subscribe, store.getSnapshot);
useEffect(() => {
store.start();
return () => store.stop();
}, [store]);
const [collapsed, setCollapsed] = useState(false);
const [pos, setPos] = useState<Point | null>(null);
const [size, setSize] = useState<Size>(DEFAULT_SIZE);
const [pipWindow, setPipWindow] = useState<Window | null>(null);
const [pinnedId, setPinnedId] = useState<string | null>(null);
const dragRef = useRef<{
kind: "move" | "resize";
corner?: ResizeCorner;
startX: number;
startY: number;
base: Point & Size;
} | null>(null);
// Elapsed-time ticker: 1s while anything is active.
const anyActive = snapshot.sessions.some((s) => s.action !== "idle");
const [now, setNow] = useState(() => Date.now());
useEffect(() => {
if (!anyActive) return;
const timer = setInterval(() => setNow(Date.now()), 1000);
return () => clearInterval(timer);
}, [anyActive]);
// Focus: the pinned session wins while it still exists; otherwise auto-follow.
const pinned =
pinnedId !== null ? snapshot.sessions.find((s) => s.sessionId === pinnedId) : undefined;
const focus = pinned ?? focusOf(snapshot.sessions);
const onTogglePin = useCallback((sessionId: string) => {
setPinnedId((current) => (current === sessionId ? null : sessionId));
}, []);
const viewport = (): Size => ({ w: window.innerWidth, h: window.innerHeight });
const onPointerMove = useCallback((event: PointerEvent) => {
const drag = dragRef.current;
if (drag === null) return;
const dx = event.clientX - drag.startX;
const dy = event.clientY - drag.startY;
if (drag.kind === "move") {
setPos(
clampPos(
{ x: drag.base.x + dx, y: drag.base.y + dy },
{ w: drag.base.w, h: drag.base.h },
viewport(),
),
);
} else if (drag.corner !== undefined) {
const next = applyResize(drag.base, drag.corner, dx, dy, viewport());
setPos(next.pos);
setSize(next.size);
}
}, []);
const onPointerUp = useCallback(() => {
dragRef.current = null;
document.removeEventListener("pointermove", onPointerMove);
document.removeEventListener("pointerup", onPointerUp);
}, [onPointerMove]);
const cardOrigin = (): Point & Size => {
const vp = viewport();
return {
x: pos?.x ?? vp.w - size.w - EDGE_MARGIN,
y: pos?.y ?? TOP_OFFSET,
w: size.w,
h: size.h,
};
};
const beginMove = (event: ReactPointerEvent) => {
event.preventDefault();
const rect = (
event.currentTarget.closest("[data-obs-card]") as HTMLElement | null
)?.getBoundingClientRect();
const origin = cardOrigin();
const base = {
x: rect?.left ?? origin.x,
y: rect?.top ?? origin.y,
w: rect !== undefined && rect.width > 0 ? rect.width : origin.w,
h: rect !== undefined && rect.height > 0 ? rect.height : origin.h,
};
dragRef.current = { kind: "move", startX: event.clientX, startY: event.clientY, base };
setPos({ x: base.x, y: base.y });
document.addEventListener("pointermove", onPointerMove);
document.addEventListener("pointerup", onPointerUp);
};
const beginResize = (corner: ResizeCorner) => (event: ReactPointerEvent) => {
event.preventDefault();
event.stopPropagation();
const rect = (
event.currentTarget.closest("[data-obs-card]") as HTMLElement | null
)?.getBoundingClientRect();
const origin = cardOrigin();
const base = {
x: rect?.left ?? origin.x,
y: rect?.top ?? origin.y,
w: rect !== undefined && rect.width > 0 ? rect.width : origin.w,
h: rect !== undefined && rect.height > 0 ? rect.height : origin.h,
};
dragRef.current = {
kind: "resize",
corner,
startX: event.clientX,
startY: event.clientY,
base,
};
setPos({ x: base.x, y: base.y });
document.addEventListener("pointermove", onPointerMove);
document.addEventListener("pointerup", onPointerUp);
};
const popOut = async (): Promise<void> => {
const pip = pipApi();
if (pip === undefined) return;
try {
const win = await pip.requestWindow({ width: size.w, height: size.h });
cloneStylesInto(win);
win.addEventListener("pagehide", () => setPipWindow(null));
setPipWindow(win);
} catch {
// requestWindow rejects without a user gesture (or when the window was
// closed mid-request): stay on the in-page card, no state change.
}
};
// Hidden while no owned session exists (and no PiP is up).
if (snapshot.sessions.length === 0 && pipWindow === null) return null;
const body = (
<OverlayBody
store={store}
focus={focus}
sessions={snapshot.sessions}
available={snapshot.available}
pinnedId={pinned !== undefined ? pinnedId : null}
onTogglePin={onTogglePin}
now={now}
inPip={pipWindow !== null}
onPopOut={pipWindow === null && pipApi() !== undefined ? () => void popOut() : undefined}
onCollapse={pipWindow === null ? () => setCollapsed(true) : undefined}
onHeaderPointerDown={pipWindow === null ? beginMove : undefined}
/>
);
if (pipWindow !== null) {
return createPortal(body, pipWindow.document.body);
}
if (collapsed) {
const state = focus !== undefined ? statusOf(focus) : "idle";
return (
<button
type="button"
className={cn(css.capsule, "bsk-obs")}
data-state={state}
aria-label="Expand browser observation overlay"
onClick={() => setCollapsed(false)}
>
<StatusDot state={state} />
<span className={css["capsule-text"]}>
{snapshot.sessions.length} session{snapshot.sessions.length === 1 ? "" : "s"}
{focus !== undefined && focus.action !== "idle"
? ` · ${focus.action} · ${formatElapsed(focus.since, now)}`
: ""}
</span>
</button>
);
}
const style: React.CSSProperties =
pos !== null
? { left: pos.x, top: pos.y, width: size.w, height: size.h }
: { right: EDGE_MARGIN, top: TOP_OFFSET, width: size.w, height: size.h };
return (
<div className={cn(css.card, "bsk-obs")} style={style} data-obs-card data-testid="obs-card">
{body}
{(["nw", "ne", "sw", "se"] as const).map((corner) => (
<div
key={corner}
className={css["resize-handle"]}
data-corner={corner}
data-testid={`obs-resize-${corner}`}
aria-label={`Resize overlay from the ${CORNER_LABEL[corner]}`}
role="separator"
aria-valuenow={size.w}
aria-valuetext={`${Math.round(size.w)} by ${Math.round(size.h)} pixels`}
aria-valuemin={MIN_SIZE.w}
aria-valuemax={Math.round(viewport().w * 0.8)}
tabIndex={corner === "se" ? 0 : -1}
onPointerDown={beginResize(corner)}
onKeyDown={
corner === "se"
? (event) => {
const step = 16;
const dx =
event.key === "ArrowRight" ? step : event.key === "ArrowLeft" ? -step : 0;
const dy = event.key === "ArrowDown" ? step : event.key === "ArrowUp" ? -step : 0;
if (dx === 0 && dy === 0) return;
event.preventDefault();
const next = applyResize(cardOrigin(), "se", dx, dy, viewport());
setPos(next.pos);
setSize(next.size);
}
: undefined
}
/>
))}
</div>
);
}
@@ -0,0 +1,57 @@
/* browser_screenshot toolview: disclosure row over a terminal block plus the
screenshot image. Semantic tokens only (--dsw-alias-*), no literal colors. */
.card {
display: flex;
flex-direction: column;
}
.summary {
min-width: 0;
margin-left: 6px;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
color: var(--dsw-alias-label-secondary);
}
.card[data-state="error"] .summary {
color: var(--dsw-alias-label-error);
}
.body {
display: flex;
flex-direction: column;
gap: 8px;
padding: 4px 0 4px 22px;
}
.image-wrap {
display: flex;
}
.inspect-button {
align-self: flex-start;
border: none;
background: none;
padding: 0;
font: inherit;
font-size: 12px;
color: var(--dsw-alias-label-tertiary);
cursor: pointer;
}
.inspect-button:hover {
color: var(--dsw-alias-label-secondary);
}
.inspect-button:focus-visible {
outline: 2px solid var(--dsw-alias-border-focus);
outline-offset: 2px;
border-radius: 2px;
}
@media (prefers-reduced-motion: reduce) {
.inspect-button {
transition: none;
}
}
@@ -0,0 +1,161 @@
// browser_screenshot keyed toolview: the terminal card's command line and
// output stay exactly as the host renders them, and a result carrying an
// image block additionally renders the screenshot itself (thumbnail with the
// shared click-to-open lightbox). Pure function of the frozen call/result
// slice — live and replay paths render identically.
import type { ImageAttachmentRef } from "@deepseek-ai/dsh-attachment";
import {
type ImageLoader,
MessageImage,
type MessageImageLabels,
} from "@deepseek-ai/dsh-client-ui-attachment";
import {
DisclosureRow,
StateDot,
type StateDotState,
TerminalBlock,
type TerminalBlockLabels,
} from "@deepseek-ai/dsh-client-ui-primitives";
import type { ToolCallViewProps } from "@deepseek-ai/dsh-client-ui-tool/client";
import { useState } from "react";
import css from "./ScreenshotToolView.module.css";
/** Resolved loader bound to the owning session at registration time. */
export type ScreenshotImageLoader = (attachment: ImageAttachmentRef) => Promise<string>;
export type ScreenshotToolViewProps = ToolCallViewProps & {
loadImage: ScreenshotImageLoader;
};
type ViewState = "running" | "ok" | "error";
interface ViewModel {
readonly state: ViewState;
readonly command: string;
readonly output: string | null;
readonly image: ImageAttachmentRef | null;
readonly summary: string;
}
const TERMINAL_LABELS: Partial<TerminalBlockLabels> = {
running: "Running",
failed: "Failed",
done: "Done",
copy: "Copy",
copied: "Copied",
noOutput: "No output",
collapseAria: "Collapse output",
collapse: "Collapse",
expandAria: (hidden) => `Expand the remaining ${hidden} output lines`,
expand: (hidden) => `… ${hidden} more lines`,
};
const IMAGE_LABELS: MessageImageLabels = {
image: "screenshot",
open: "Open the original screenshot",
openNamed: (label) => `Open screenshot ${label}`,
loading: "Loading…",
loadFailed: "Load failed — retry",
lightbox: { dialog: "Screenshot preview", close: "Close preview" },
};
function firstLine(text: string): string {
const newline = text.indexOf("\n");
return newline === -1 ? text : text.slice(0, newline);
}
/** Rebuild the command line from the logged arguments (mirrors the host presenter). */
function commandOf(argsRaw: string, callId: string): string {
try {
const args = JSON.parse(argsRaw) as { session?: unknown; ref?: unknown };
const parts = ["bsk", "screenshot", "--session"];
parts.push(
typeof args.session === "string" && args.session !== "" ? args.session : "(current)",
);
if (typeof args.ref === "string" && args.ref !== "") parts.push("--ref", args.ref);
return parts.join(" ");
} catch {
// Streaming can expose a truncated JSON prefix; it is still the best label.
return argsRaw === "" ? `bsk screenshot (${callId})` : `bsk screenshot ${firstLine(argsRaw)}`;
}
}
/** Derive the display model from the frozen block only. */
export function viewModelOf(block: ToolCallViewProps["block"]): ViewModel {
const settled = "kind" in block;
const argsRaw = (settled ? block.call?.argsRaw : block.argsRaw) ?? "";
const command = commandOf(argsRaw, block.callId);
if (!settled) {
return { state: "running", command, output: null, image: null, summary: command };
}
const textBlock = block.content.find((item) => item.type === "text");
const imageBlock = block.content.find((item) => item.type === "image");
const output = textBlock !== undefined && textBlock.type === "text" ? textBlock.text : null;
const image =
imageBlock !== undefined && imageBlock.type === "image" ? imageBlock.attachment : null;
const state: ViewState = block.isError ? "error" : "ok";
const summary = output !== null ? firstLine(output) : command;
return { state, command, output, image, summary };
}
function dotState(state: ViewState): StateDotState {
switch (state) {
case "running":
return "ongoing";
case "error":
return "error";
default:
return "done";
}
}
/**
* Render one browser_screenshot call: a disclosure row over a terminal block
* (command + output) plus the screenshot image when the result carries one.
*/
export function ScreenshotToolView({ block, cwd, inspect, loadImage }: ScreenshotToolViewProps) {
const model = viewModelOf(block);
const [expanded, setExpanded] = useState(false);
const expandable = model.state === "running" || model.output !== null || model.image !== null;
const open = expanded && expandable;
return (
<div className={css.card} data-tool="browser_screenshot" data-state={model.state}>
<DisclosureRow
icon={<StateDot state={dotState(model.state)} />}
title="Screenshot"
open={open}
expandable={expandable}
onToggle={() => setExpanded((value) => !value)}
expandOnRowClick
previewChevron
collapsedContent={<span className={css.summary}>{model.summary}</span>}
>
<div className={css.body}>
<TerminalBlock
command={model.command}
cwd={cwd ?? undefined}
output={model.output ?? undefined}
running={model.state === "running"}
labels={TERMINAL_LABELS}
/>
{model.image !== null ? (
<div className={css["image-wrap"]}>
<MessageImage
attachment={model.image}
load={loadImage}
variant="single"
labels={IMAGE_LABELS}
/>
</div>
) : null}
{inspect !== undefined ? (
<button type="button" className={css["inspect-button"]} onClick={inspect}>
Inspect
</button>
) : null}
</div>
</DisclosureRow>
</div>
);
}
@@ -0,0 +1,49 @@
/*
* BrowserSkill UI design tokens, scoped to the observation overlay root.
* VALUES MIRROR packages/ui/src/styles/tailwind.css (:root and .dark) — keep
* in sync with that file; it is the canonical source. Scoped here (instead of
* :root) so the dsh host theme can neither leak in nor be polluted by us.
* Dark follows prefers-color-scheme, same as the extension popup.
*/
.bsk-obs {
color-scheme: light dark;
--background: oklch(0.99 0.005 50);
--foreground: oklch(0.14 0.03 50);
--card: oklch(0.995 0.003 50 / 0.82);
--card-foreground: oklch(0.14 0.03 50);
--primary: oklch(0.62 0.22 45);
--primary-foreground: oklch(0.99 0.01 50);
--secondary: oklch(0.96 0.04 65);
--secondary-foreground: oklch(0.25 0.08 65);
--muted: oklch(0.96 0.01 50);
--muted-foreground: oklch(0.55 0.04 50);
--accent: oklch(0.96 0.04 65);
--accent-foreground: oklch(0.25 0.08 65);
--destructive: oklch(0.6 0.25 25);
--destructive-foreground: oklch(0.99 0.01 50);
--border: oklch(0.9 0.025 60);
--input: oklch(0.9 0.025 60);
--ring: oklch(0.62 0.22 45);
--radius: 0.75rem;
}
@media (prefers-color-scheme: dark) {
.bsk-obs {
--background: oklch(0.2 0.02 50);
--foreground: oklch(0.98 0.01 50);
--card: oklch(0.24 0.02 50 / 0.82);
--card-foreground: oklch(0.98 0.01 50);
--primary: oklch(0.7 0.18 45);
--primary-foreground: oklch(0.14 0.03 50);
--secondary: oklch(0.32 0.03 60);
--secondary-foreground: oklch(0.98 0.01 50);
--muted: oklch(0.3 0.02 50);
--muted-foreground: oklch(0.75 0.03 50);
--accent: oklch(0.32 0.03 60);
--accent-foreground: oklch(0.98 0.01 50);
--destructive: oklch(0.45 0.2 25);
--destructive-foreground: oklch(0.98 0.01 50);
--border: oklch(0.35 0.02 60);
--input: oklch(0.35 0.02 60);
--ring: oklch(0.7 0.18 45);
}
}
@@ -0,0 +1,84 @@
/**
* dsh-plugin-browserskill browser half: the `browser_screenshot` keyed
* toolview plus the observation overlay (live thumbnails + interrupt) floating
* over the shell via the `shell.overlay` seat.
*/
import type { ImageAttachmentRef } from "@deepseek-ai/dsh-attachment";
import type { ClientContext, ISessions, SessionId } from "@deepseek-ai/dsh-client-runtime/client";
// Type-only: pulls the 'shell.overlay' SlotMap merge into scope.
import type {} from "@deepseek-ai/dsh-client-ui-layout/client";
import type { ToolCallViewProps } from "@deepseek-ai/dsh-client-ui-tool/client";
import { createElement } from "react";
// Scope-prefixed BSK design tokens and utility sheet (injected verbatim as
// <style> tags; selectors stay unhashed so `cn(..., "bsk-obs")` roots match).
import "./bsk-tokens.nomodule.css";
import "./bsk-ui.nomodule.css";
import { ObservationOverlay } from "./ObservationOverlay";
import { type EventSourceLike, ObservationClientStore } from "./observation-store";
import { ScreenshotToolView } from "./ScreenshotToolView";
/** Required services: slots, session-scoped attachment reads, and the overlay seat. */
export const inject = ["slots", "sessions"];
/** Resolve one durable image attachment into a browser blob URL. */
async function loadSessionImage(
sessions: ISessions,
sessionId: SessionId,
attachment: ImageAttachmentRef,
): Promise<string> {
const session = sessions.binding(sessionId)?.session;
if (session === undefined) {
throw new Error(`screenshot toolview: session "${String(sessionId)}" is not bound`);
}
const result = await session.readAttachment(attachment.attachmentId);
if (!result.ok) {
throw new Error(
`screenshot toolview: readAttachment failed: ${result.error.code}: ${result.error.message}`,
);
}
const bytes = Uint8Array.from(result.value.data);
return URL.createObjectURL(
new Blob([bytes.buffer as ArrayBuffer], { type: result.value.attachment.mediaType }),
);
}
/**
* Thumbnail loader for the overlay: frames are plugin-owned runtime data
* (never referenced by a session log, so the session-authorized RPC refuses
* them), served by the host over the plugin's own route.
*/
async function overlayImageLoader(attachmentId: string): Promise<string> {
const res = await fetch(`/bsk-observation/thumbnail/${encodeURIComponent(attachmentId)}`);
if (!res.ok) throw new Error(`thumbnail fetch failed: ${res.status}`);
return URL.createObjectURL(await res.blob());
}
/**
* Client plugin body: register the keyed toolview and the observation overlay.
* @param ctx - client root context.
*/
export function apply(ctx: ClientContext): void {
const sessions = ctx.get("sessions") as unknown as ISessions;
ctx.slots.inject("tool.call.toolview", () =>
ctx.slots.register(
{ name: "tool.call.toolview", key: "browser_screenshot" },
(props: ToolCallViewProps) =>
createElement(ScreenshotToolView, {
...props,
loadImage: (attachment: ImageAttachmentRef) =>
loadSessionImage(sessions, props.sessionId, attachment),
}),
),
);
const store = new ObservationClientStore({
fetchFn: (url, init) => fetch(url, init),
eventSourceFactory: (url) => new EventSource(url) as unknown as EventSourceLike,
loadImage: overlayImageLoader,
});
ctx.slots.inject("shell.overlay", () =>
ctx.slots.register({ name: "shell.overlay", id: "bsk-observation" }, () =>
createElement(ObservationOverlay, { store }),
),
);
}
@@ -0,0 +1,295 @@
/**
* Client-side data layer for the observation overlay: initial state fetch,
* SSE increment stream (with a state refetch on every (re)open — events that
* fired while the stream was down are otherwise lost forever), on-demand
* thumbnail blob loading through the session-authorized readAttachment path,
* and the interrupt call. The last ready blob per session is held until its
* replacement decodes so the overlay can swap frames in place. All I/O is
* injected so tests never touch a network.
*/
import type { ObservationEvent, SessionObservation } from "../observation";
export interface EventSourceLike {
onmessage: ((event: { data: string }) => void) | null;
/** Fires on the initial connect AND on every automatic reconnect. */
onopen?: (() => void) | null;
close(): void;
}
export interface ObservationClientDeps {
fetchFn: (
url: string,
init?: { method?: string; body?: string; headers?: Record<string, string> },
) => Promise<{
ok: boolean;
json(): Promise<unknown>;
}>;
eventSourceFactory: (url: string) => EventSourceLike;
/** Resolve one attachment id to a displayable blob URL. */
loadImage: (attachmentId: string) => Promise<string>;
}
export interface ThumbnailState {
status: "loading" | "ready" | "error";
url?: string;
}
export interface OverlaySnapshot {
readonly sessions: readonly SessionObservation[];
/** Whether the SSE increment stream is currently subscribed. */
readonly subscribed: boolean;
readonly thumbnails: Readonly<Record<string, ThumbnailState>>;
/**
* sessionId → the frame the overlay should paint. While a newer attachment
* is still loading (or failed), this keeps the last good URL so the card
* never drops to a placeholder between breaths.
*/
readonly displayFrames: Readonly<Record<string, ThumbnailState>>;
/** False when the host reports the browser/daemon as unreachable. */
readonly available: boolean;
}
const STATE_URL = "/bsk-observation/state";
const EVENTS_URL = "/bsk-observation/events";
const INTERRUPT_URL = "/bsk-observation/interrupt";
function revoke(url: string | undefined): void {
if (url !== undefined && typeof URL.revokeObjectURL === "function") {
URL.revokeObjectURL(url);
}
}
export class ObservationClientStore {
private sessions = new Map<string, SessionObservation>();
private thumbs = new Map<string, ThumbnailState>();
/** sessionId → the attachment id currently advertised for it. */
private readonly thumbBySession = new Map<string, string>();
/** sessionId → last successfully decoded frame (held across the next load). */
private readonly lastReady = new Map<string, { attachmentId: string; url: string }>();
private listeners = new Set<() => void>();
private events: EventSourceLike | undefined;
private snapshot: OverlaySnapshot = {
sessions: [],
subscribed: false,
thumbnails: {},
displayFrames: {},
available: true,
};
private available = true;
private started = false;
constructor(private readonly deps: ObservationClientDeps) {}
subscribe = (listener: () => void): (() => void) => {
this.listeners.add(listener);
return () => this.listeners.delete(listener);
};
getSnapshot = (): OverlaySnapshot => this.snapshot;
private publish(): void {
this.snapshot = {
sessions: [...this.sessions.values()],
subscribed: this.events !== undefined,
thumbnails: Object.fromEntries(this.thumbs),
displayFrames: this.buildDisplayFrames(),
available: this.available,
};
for (const listener of [...this.listeners]) listener();
}
/** Frame the overlay should paint for one session (last good while next loads). */
private frameFor(sessionId: string): ThumbnailState | undefined {
const attachmentId = this.thumbBySession.get(sessionId);
const current = attachmentId !== undefined ? this.thumbs.get(attachmentId) : undefined;
if (current?.status === "ready" && current.url !== undefined) return current;
const held = this.lastReady.get(sessionId);
if (held !== undefined) {
return {
status: current?.status === "error" ? "error" : "ready",
url: held.url,
};
}
return current;
}
private buildDisplayFrames(): Record<string, ThumbnailState> {
const frames: Record<string, ThumbnailState> = {};
for (const sessionId of this.sessions.keys()) {
const frame = this.frameFor(sessionId);
if (frame !== undefined) frames[sessionId] = frame;
}
return frames;
}
private sessionOf(attachmentId: string): string | undefined {
for (const [sessionId, id] of this.thumbBySession) {
if (id === attachmentId) return sessionId;
}
return undefined;
}
/** (Re)pull the full state: initial load and every SSE (re)open. */
private refreshState(): void {
void this.deps
.fetchFn(STATE_URL)
.then(async (res) => {
if (!res.ok) return;
const body = (await res.json()) as {
sessions?: SessionObservation[];
available?: boolean;
};
const sessions = body.sessions ?? [];
this.sessions = new Map(sessions.map((s) => [s.sessionId, s]));
// Prune thumbnails of sessions that vanished while we were away.
const alive = new Set(sessions.map((s) => s.sessionId));
for (const sessionId of [...this.thumbBySession.keys()]) {
if (!alive.has(sessionId)) this.dropThumb(sessionId);
}
for (const s of sessions) this.trackThumb(s.sessionId, s.thumbnailAttachmentId);
if (typeof body.available === "boolean") this.available = body.available;
this.publish();
})
.catch(() => {});
}
/** Initial fetch + SSE subscription. Idempotent. */
start(): void {
if (this.started) return;
this.started = true;
this.refreshState();
const events = this.deps.eventSourceFactory(EVENTS_URL);
events.onmessage = (message) => {
let event: ObservationEvent;
try {
event = JSON.parse(message.data) as ObservationEvent;
} catch {
return;
}
this.apply(event);
};
events.onopen = () => {
// Reconnects lose every event fired during the outage — resync.
if (this.events === events) this.refreshState();
};
this.events = events;
this.publish();
}
stop(): void {
this.events?.close();
this.events = undefined;
this.started = false;
for (const thumb of this.thumbs.values()) revoke(thumb.url);
this.thumbs.clear();
this.thumbBySession.clear();
this.lastReady.clear();
this.sessions.clear();
this.publish();
}
/** Forget one session's tracked + held frames, revoking their blob URLs. */
private dropThumb(sessionId: string): void {
const attachmentId = this.thumbBySession.get(sessionId);
const held = this.lastReady.get(sessionId);
this.thumbBySession.delete(sessionId);
this.lastReady.delete(sessionId);
if (attachmentId !== undefined) {
revoke(this.thumbs.get(attachmentId)?.url);
this.thumbs.delete(attachmentId);
}
if (held !== undefined && held.attachmentId !== attachmentId) {
revoke(held.url);
this.thumbs.delete(held.attachmentId);
}
}
/**
* Track the frame a session currently advertises. The last *ready* blob is
* kept until the replacement decodes — dropping it on the upsert is what
* made the overlay flash a placeholder between breaths.
*/
private trackThumb(sessionId: string, attachmentId: string | undefined): void {
const previous = this.thumbBySession.get(sessionId);
if (previous === attachmentId) return;
if (attachmentId !== undefined) this.thumbBySession.set(sessionId, attachmentId);
else this.thumbBySession.delete(sessionId);
const heldId = this.lastReady.get(sessionId)?.attachmentId;
if (previous !== undefined && previous !== heldId) {
revoke(this.thumbs.get(previous)?.url);
this.thumbs.delete(previous);
}
}
private apply(event: ObservationEvent): void {
if (event.type === "reset") {
this.sessions.clear();
for (const thumb of this.thumbs.values()) revoke(thumb.url);
this.thumbs.clear();
this.thumbBySession.clear();
this.lastReady.clear();
} else if (event.type === "remove" && event.session !== undefined) {
this.sessions.delete(event.session.sessionId);
this.dropThumb(event.session.sessionId);
} else if (event.type === "upsert" && event.session !== undefined) {
this.sessions.set(event.session.sessionId, event.session);
this.trackThumb(event.session.sessionId, event.session.thumbnailAttachmentId);
} else if (event.type === "availability") {
this.available = event.available;
}
this.publish();
}
/**
* Ensure a thumbnail load is in flight for one attachment reference. New
* frames replace the old URL only after they decode; failures keep the last
* good frame (the caller renders `status: 'error'` as a small badge over
* the old image).
*/
ensureThumbnail(attachmentId: string | undefined): void {
if (attachmentId === undefined || this.thumbs.has(attachmentId)) return;
this.thumbs.set(attachmentId, { status: "loading" });
this.publish();
this.deps.loadImage(attachmentId).then(
(url) => {
if (!this.thumbs.has(attachmentId)) {
// Replaced or removed while loading: never resurrect (or leak) it.
revoke(url);
return;
}
this.thumbs.set(attachmentId, { status: "ready", url });
const sessionId = this.sessionOf(attachmentId);
if (sessionId !== undefined) {
const previous = this.lastReady.get(sessionId);
if (previous !== undefined && previous.attachmentId !== attachmentId) {
revoke(previous.url);
this.thumbs.delete(previous.attachmentId);
}
this.lastReady.set(sessionId, { attachmentId, url });
}
this.publish();
},
() => {
this.thumbs.set(attachmentId, { status: "error" });
this.publish();
},
);
}
/** Interrupt the current or named session's in-flight call. */
async interrupt(sessionId?: string): Promise<boolean> {
try {
const res = await this.deps.fetchFn(INTERRUPT_URL, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(sessionId === undefined ? {} : { sessionId }),
});
if (!res.ok) return false;
const body = (await res.json()) as { interrupted?: boolean };
return body.interrupted === true;
} catch {
return false;
}
}
}
+6
View File
@@ -0,0 +1,6 @@
declare module "*.module.css" {
const classes: Record<string, string>;
export default classes;
}
declare module "*.css";
@@ -0,0 +1,83 @@
/**
* Screenshot delivery. The canonical tool value stays plain JSON (a PNG file
* path plus pixel metadata); when the host mounts a durable attachment store
* AND the calling route declares image input, the PNG bytes are additionally
* committed through `ctx.attachments` so the render step can attach the image
* itself to the tool result. Any uncertainty (no store, unknown route,
* text-only model) falls back to the path-only form.
*/
import type { Context } from "@deepseek-ai/cordis";
import type { ImageAttachmentRef } from "@deepseek-ai/dsh-attachment";
import type { ToolExecution } from "@deepseek-ai/dsh-tools";
/** Structural view of the attachment seam (avoids a runtime dependency). */
interface AttachmentLike {
imageLimits: {
mediaTypes: readonly string[];
maxImageBytes: number;
maxMessageImageBytes: number;
};
saveImage(input: {
data: Uint8Array;
mediaType: string;
name?: string;
}): Promise<ImageAttachmentRef>;
}
interface LlmLike {
resolveModelInfo(
provider: string,
model: string,
signal?: AbortSignal,
): Promise<{ inputModalities?: readonly string[] }>;
}
/**
* Commit screenshot bytes to the host attachment store when the composition
* supports durable images on the current model route.
* @returns the durable reference, or undefined to stay in path-only mode.
*/
export async function trySaveScreenshot(
ctx: Context,
exec: ToolExecution,
data: Uint8Array,
name: string,
): Promise<ImageAttachmentRef | undefined> {
const attachments = ctx.get("attachments") as AttachmentLike | undefined;
if (attachments === undefined) return undefined;
if (!attachments.imageLimits.mediaTypes.includes("image/png")) return undefined;
if (
data.byteLength >
Math.min(attachments.imageLimits.maxImageBytes, attachments.imageLimits.maxMessageImageBytes)
) {
return undefined;
}
if (!(await isImageCapableRoute(ctx, exec))) return undefined;
try {
return await attachments.saveImage({ data, mediaType: "image/png", name });
} catch {
// A store hiccup must not fail the tool call; the PNG path still works.
return undefined;
}
}
/**
* Best-effort check that the calling route's model accepts image input,
* mirroring dsh-tool-fs `read_image`. Unknown route / missing llm service
* answers false (refuse to attach) so history never gains an image block a
* text-only adapter cannot replay.
*/
async function isImageCapableRoute(ctx: Context, exec: ToolExecution): Promise<boolean> {
const llm = ctx.get("llm") as LlmLike | undefined;
const provider =
exec.agent?.session.requestHeader()?.config.provider ?? exec.agent?.options.provider;
const model = exec.agent?.session.requestHeader()?.config.model ?? exec.agent?.options.model;
if (llm === undefined || provider === undefined || model === undefined) return false;
try {
const info = await llm.resolveModelInfo(provider, model, exec.signal);
return info.inputModalities?.includes("image") ?? false;
} catch {
return false;
}
}

Some files were not shown because too many files have changed in this diff Show More