Compare commits

...
94 Commits
Author SHA1 Message Date
BB-fat b4450fbb8c chore(release): bsk 0.1.10 (#73) 2026-08-08 23:22:03 +08:00
BB-fat 1268761611 fix(screenshot): hide overlays during capture so shots only contain page content (#64) 2026-08-08 22:48:50 +08:00
Zhang GH 37d3e2b22f Merge pull request #72 from Tencent/issue-69-cws-publishing
CI: add automate Chrome Web Store publishing with service account
2026-08-07 19:24:30 +08:00
drakezhang ce8bc6c870 add automate Chrome Web Store publishing with service account 2026-08-07 14:30:59 +08:00
Zhang GH 93df62a356 Merge pull request #42 from shnpd/feat/popup-function-icon
feat(extension): use function launcher icon
2026-08-06 16:47:52 +08:00
Zhang GH 20002e1ff2 Merge pull request #63 from Tencent/feat/record-overlay-timer
feat(extension): compact recording overlay with live timer
2026-08-06 16:47:11 +08:00
“haonan” 674c53339f fix(extension): keep record overlay timer across navigations 2026-08-06 16:28:21 +08:00
BB-fat d0bee1a09b feat(install-skill): add Kimi Code harness support (#65)
* feat(install-skill): add Kimi Code harness support

Add kimi-code to the install-skill harness list: installs the bundled SKILL.md into ~/.kimi/skills (Kimi Code CLI's brand-group user skills directory, highest priority per https://github.com/MoonshotAI/kimi-cli/blob/main/docs/en/customization/skills.md). Detection signals: ~/.kimi directory and kimi/kimi-cli on PATH. Aliases: kimi, kimi-code, kimi-cli, kimicode, kimi_cli.

* fix(install-skill): point kimi-code harness at ~/.kimi-code and honor KIMI_CODE_HOME

Kimi Code CLI's canonical repo is MoonshotAI/kimi-code: user-level skills live in $KIMI_CODE_HOME/skills (default ~/.kimi-code/skills), not kimi-cli's ~/.kimi/skills. Detection checks the kimi-code home dir (env-aware label) and the kimi binary on PATH. Serialize env-mutating kimi tests with a shared mutex to avoid a parallel-test race on KIMI_CODE_HOME.
2026-08-06 13:25:18 +08:00
“haonan” 66684c4ce3 feat(extension): compact recording overlay with live timer
Show the recording message for 10s, then switch to a fixed-width MM:SS
timer with the icon-only finish button.
2026-08-05 20:26:43 +08:00
BB-fat 0f264c5df1 feat(emulate): add mobile device emulation via CDP Emulation domain (#60)
* feat(emulate): add mobile device emulation via CDP Emulation domain

Add `bsk emulate` so agents can debug mobile page behaviour by
emulating a device environment on a session tab:

- Viewport metrics (Emulation.setDeviceMetricsOverride), User-Agent
  (Emulation.setUserAgentOverride, optional accept-language and UA
  client-hints metadata), and touch (Emulation.setTouchEmulationEnabled
  with maxTouchPoints).
- Seven CLI-side device presets (iphone-14, iphone-14-pro-max,
  iphone-se, pixel-7, galaxy-s23, ipad-mini, galaxy-tab-s8) via
  --device; manual flags (--width/--height/--dpr/--mobile/--ua/
  --accept-language/--touch/--max-touch-points) work standalone or
  override individual preset fields.
- `bsk emulate --off` clears every override (clearDeviceMetricsOverride
  + touch disabled + empty UA override) and restores the real
  environment.
- Overrides are per-tab (CDP per-target) and not inherited by new tabs;
  the result JSON carries a note reminding agents to re-apply.

Wire: new tool.emulate method end-to-end (bsk-protocol types + JSON
schemas + Method::ToolEmulate classified BrowserMutation, daemon
forwarding, CLI subcommand, extension tool handler + ChromiumCdp
Emulation wrappers, dispatcher registration), docs in both SKILL.md
copies, and unit tests on every layer.

* refine(emulate): merge per-tab state and align preset override semantics

- extension: remember the applied emulation state per tab and merge each new request onto it field by field before applying, so a later --width/--height no longer resets an earlier preset's dpr/mobile; --off clears the stored state along with the CDP overrides
- extension: cdp_failed errors now note that already-applied overrides were not rolled back and can be reset with --off
- cli: validate width/height pairing and metric requirements on the merged preset+flags result, so --device iphone-14 --width 500 keeps the preset height
- cli: add --no-mobile/--no-touch (conflicting with --mobile/--touch) to turn preset mobile/touch fields back off
- docs: describe the merge behaviour, --no-mobile/--no-touch and --off state clearing in both SKILL.md copies
2026-08-05 18:10:39 +08:00
BB-fat 49378d70de feat: support Agent Window sizing (session start --width/--height, window resize) (#52)
- `bsk session start --width <px> --height <px>` opens the Agent Window
  with the given outer size (chrome.windows.create)
- new `bsk window resize --session <id> --width <px> --height <px>`
  resizes the Agent Window at runtime (chrome.windows.update)
- both validated on CLI, daemon and extension sides (100..=7680 CSS px)
- protocol: new tool.window_resize method (BrowserMutation) and
  optional width/height on tool.session_start params; schemas regenerated
2026-08-05 11:19:10 +08:00
BB-fat d7f0f1e78c fix(install-skill): restrict --all to detected harnesses (#59)
`bsk install-skill --all` installed the bundled skill into every supported harness directory (detected or not), creating junk paths like ~/.claude/skills on machines without those harnesses. --all now targets detected harnesses only and errors with guidance when none are found; --harness remains the explicit override. The --yes non-interactive branch shares the same detected-only logic.
2026-08-04 21:35:01 +08:00
BB-fat 71255ae142 feat(request-help): add BSK_REQUEST_HELP=off to disable blocking help requests (#53)
* feat(request-help): add BSK_REQUEST_HELP=off to disable blocking help requests

* style(extension): format HelpOutcome union per biome

* docs(skill): condense BSK_REQUEST_HELP section
2026-07-31 23:54:30 +08:00
Ljy-0827 3f091318d0 chore(release): bsk 0.1.9 + extension 0.1.5 2026-07-29 15:01:59 +08:00
Zhang GH c6a2b3ec1e Merge pull request #45 from Tencent/vom
This PR brings the new VOM feature and greatly improves BrowserSkill. Some strategies may need user feedback to catch edge cases, so let's merge it now as a new version and continue polishing it iteratively.
2026-07-29 13:42:00 +08:00
Ljy-0827 d67c1a71b0 fix(vom): CI fix 2026-07-28 18:19:47 +08:00
Ljy-0827 03d2adc035 fix(vom): clean up pr 2026-07-28 17:36:41 +08:00
Ljy-0827 8f5df59b79 fix(vom): overlay 2026-07-28 17:31:33 +08:00
Ljy-0827 000fbf5540 fix(vom): bug fix 2026-07-28 16:45:40 +08:00
Ljy-0827 a52c2d6bf2 feat(vom): split function fix 2026-07-28 11:19:19 +08:00
Ljy-0827 c002c394e6 feat(vom): vom fix 2026-07-27 14:53:10 +08:00
Ljy-0827 4fb2e331a6 feat(vom): update vom algorithm, fix overlay problem with background state machine 2026-07-27 13:46:33 +08:00
Ljy-0827 e14ae9cea0 feat(vom): vom method bug fix 2026-07-24 17:25:57 +08:00
Ljy-0827 f319672721 feat(vom): method update, solve context over exposed problem 2026-07-24 17:25:57 +08:00
Ljy-0827 5e0d26ccb2 feat(vom): method update 2026-07-24 17:25:57 +08:00
Ljy-0827 9dbc553e33 feat(vom): vom base algorithm and infrastructure 2026-07-24 17:25:57 +08:00
Zhang GH b657ab574b Merge pull request #43 from Tencent/fix/early-control-return
fix(extension): fix early control return and preserve request-help handoff across tabs
LGTM
2026-07-24 15:27:41 +08:00
Ljy-0827 959bfac1ae fix(extension): human-loop fixed 2026-07-24 14:07:33 +08:00
Ljy-0827 c095128218 fix(extension): CI problem fixed, tab logic fixed 2026-07-24 14:07:33 +08:00
Ljy-0827 f73a0e9fb3 fix(runtime): incorrectly takeover the browser when user open up different tab within the same window 2026-07-24 14:07:33 +08:00
Ljy-0827 4e44fb2823 fix(runtime): fix early control return and update lifecycle 2026-07-24 14:07:33 +08:00
Zhang GH c8963e3a68 Merge pull request #37 from Tencent/fix-browser-keepalive-reconnect
Fix browser keepalive reconnect
2026-07-23 21:45:17 +08:00
“haonan”andCursor aab3b7a8e5 feat(extension): use function launcher icon
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-23 15:44:11 +08:00
drakezhang aa63465050 Merge remote-tracking branch 'origin/main' into fix-browser-keepalive-reconnect
Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	apps/extension/wxt.config.ts
#	crates/bsk-cli/src/cli/session.rs
2026-07-22 15:59:12 +08:00
“haonan” ff74db48c8 chore(release): bsk 0.1.8 + extension 0.1.4 2026-07-22 15:29:02 +08:00
polarday bc8d6fa190 Merge pull request #28 from shnpd/feat/semantic-record-v3
feat: add semantic browser recording via bsk record
2026-07-22 15:17:56 +08:00
“haonan” 19981526c7 fix: record prompt copy 2026-07-22 15:05:54 +08:00
“haonan” d1de850b42 fix(record): use example.com as the default record start URL 2026-07-22 14:15:53 +08:00
“haonan” 585308df45 feat(record): add popup quick-actions launcher and make record --url optional
Add a popup launcher + action-recording panel that builds a copyable
`bsk record start` prompt, add `--browser` targeting, and default `--url`
to https://example.com/ when omitted. Drop the popup browser-alias field.
2026-07-22 13:27:30 +08:00
drakezhangandCursor 29ff239726 Make liveness thresholds configurable and cover the reaper end-to-end
Move the reaper's silence timeout and scan cadence into DaemonConfig
(defaulting to the existing 60s/15s constants) with a
`with_browser_liveness` builder, so tests can drive the task in
sub-second windows.

Add an integration test that runs a real daemon and asserts a silent
heartbeat-capable browser — and its sessions — get reaped, while a
legacy browser that never heartbeated is left untouched.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-22 13:06:17 +08:00
drakezhangandCursor 31d2a1b3dc Gate liveness reaping on heartbeat capability
Only reap browsers that have sent at least one `system.heartbeat`.
A pre-heartbeat extension never opts in, so it is never dropped on
silence and keeps relying on socket-close detection exactly as before.
This removes a backward-compat regression where a new daemon paired
with an old extension would wrongly reap a live-but-idle connection
and then report `no_browser_connected` on the next command.

- Add a `heartbeat_seen` flag to `BrowserClient`, set when a heartbeat
  event arrives, and exclude never-heartbeated clients from
  `stale_browsers`.
- Cover both the opt-in and the legacy-survivor cases in unit tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-22 13:06:17 +08:00
drakezhangandCursor ed92e54dff Bump extension to 0.1.4
The manifest now declares a minimum Chrome version and requests the
`idle` permission, and the runtime gains the WebSocket heartbeat and
wake-driven reconnect. Publishing that update requires a new version.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-22 12:30:46 +08:00
drakezhangandCursor cb11cecdf6 Lock system.heartbeat wire name with a contract test
The extension emits the literal event string "system.heartbeat";
assert the daemon-side serde name serialises to the same value and
round-trips from the exact frame shape the extension sends, so a
future rename cannot silently break keepalive and liveness.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-22 12:30:46 +08:00
drakezhangandCursor ea03f36f3a Widen session.start browser-connect wait for cold reconnects
After a long idle the service worker is only revived on its next
keepalive-alarm tick (up to 30s away), so the previous 5s wait made
the first command after idle fail with `no_browser_connected` even
though the extension was about to reconnect.

- Raise the daemon's `EXTENSION_CONNECT_WAIT` (used by `session.start`)
  to 35s so it comfortably covers one alarm period plus handshake slack.
- Bump the CLI `session start` IPC timeout to that window plus slack so
  it never gives up before the daemon answers, and print a one-line
  "waiting for browser extension to connect…" hint when it actually
  waits.
- Keep `bsk status` / `browsers` on their own short 5s wait so quick
  informational commands are not slowed down.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-22 12:17:15 +08:00
drakezhangandCursor 234e9101ed Keep service worker alive with WS heartbeat and wake reconnect
Harden the extension's connection so the daemon link survives normal
idle periods and OS sleep instead of silently dropping.

- Send a `system.heartbeat` event every 20s while the post-handshake
  link is live. Since Chrome 116 WebSocket activity resets the MV3
  service-worker idle timer, so this keeps the worker (and the socket)
  alive during use rather than relying on the 30s keepalive alarm that
  sits right on the eviction boundary. Declare `minimum_chrome_version:
  "116"` accordingly.
- Reconnect on `runtime.onStartup` and `idle.onStateChanged` ("active"),
  which revive the worker after a browser restart or machine wake and
  let it reconnect immediately instead of waiting for the next alarm
  tick. Add the `idle` permission and document it.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-22 12:17:06 +08:00
drakezhangandCursor 5e91178e03 Add heartbeat event and daemon browser-liveness reaper
Introduce a `system.heartbeat` protocol event and use it as a
liveness signal on the daemon side.

- Track `last_seen` per connected browser, bumped on every inbound
  WS frame (tool response, event, ping/pong, or heartbeat).
- Add a reaper task that drops any browser silent past a 60s window
  and purges its sessions, mirroring the WS disconnect cleanup. This
  releases half-open connections left behind when the OS resumes from
  sleep and kills the service worker without delivering a close frame,
  so the daemon no longer pins a phantom "connected" browser and can
  idle-exit normally.
- Wire the reaper into both the foreground daemon and the test/embed
  entry point, aborting it on shutdown.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-22 12:16:56 +08:00
“haonan” ac028a7a18 feat: add bsk record for semantic user-action traces
Capture user actions in the Agent Window and export a record-only
trace (pages[] + navigate/click/fill/select/press steps with target
descriptors). Adds the record overlay UI aligned with the control
overlay, and wires record start/stop/await through CLI, daemon,
protocol, and extension.
2026-07-22 11:27:30 +08:00
BB-fat ccf3c32140 Merge pull request #8 from hjxccc/feat/console-network-commands
feat: add bsk network read-only command
2026-07-18 22:53:20 +08:00
BB-fatandCursor 9799b98937 fix: address network command review feedback
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-18 22:52:53 +08:00
hjxccc 259b4826b7 feat: add bsk network read-only command
Add `bsk network` as a sibling of the `console` command from #7,
reading the buffered network responses / failures for a tab. Fills the
remaining half of the request in #2 (console landed in #7; network did
not).

Mirrors the console command's conventions exactly:

  bsk network --session <id> [--tab-id N] [--since C]
                             [--limit N] [--max-text-chars N]

The extension enables `Network` on attach (best-effort) and buffers
`responseReceived` / `loadingFailed` events per tab with a monotonic
sequence, correlating them to `requestWillBeSent` for method/URL. Reads
are cursor-paginated (`since` -> `next_since`) and bounded (`limit`,
`max_text_chars`) for agent-context safety, same as console. Entries
carry the real HTTP status code (response) or the CDP failure reason
(failure) — things `performance.getEntriesByType('resource')` can't
give.

Classified read-only (pass-through under the pending-interrupt gate);
defaults to the Agent Window's active tab.

Wires bsk-protocol (Method + params/result/entry structs + schema),
bsk-cli (network subcommand + daemon dispatch allow-list) and the
extension (handler + dispatcher route), with unit / parse / IPC tests
paralleling the console command.

Refs #2.
2026-07-18 22:37:57 +08:00
BB-fat 3d6c95c22a Merge pull request #16 from NianJiuZst/codex/fail-closed-tab-borrow
fix(extension): fail closed on borrow confirmation
2026-07-18 22:34:31 +08:00
BB-fatandCursor 11ec51dfb8 fix(extension): dismiss overlay on borrow timeout
Send borrow-cancel when the confirmation timeout denies, and add a
timer fallback so auto-deny does not depend only on CSS transitionend.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-18 22:31:27 +08:00
BB-fat e16b37aaa0 Merge pull request #22 from NianJiuZst/codex/enforce-session-idle-timeout
fix(daemon): enforce session idle timeout
2026-07-17 16:53:21 +08:00
BB-fat d27ef1f5e0 Merge pull request #23 from NianJiuZst/codex/doctor-fails-nonzero
fix(doctor): exit nonzero on failed checks
2026-07-17 16:50:19 +08:00
BB-fat e55ce8ad99 Merge pull request #29 from klren0312/fixbug/errortips
Correct the incorrect command examples in the error message that does not indicate the mistake.
2026-07-17 16:13:49 +08:00
klkl 7bee906ffb fix(bsk-cli): Correct the incorrect command examples in the error message that does not indicate the mistake. 2026-07-17 15:42:52 +08:00
NianJiuZst bf6c384b2e test: avoid probe race in idle daemon helper 2026-07-10 23:07:43 +08:00
NianJiuZst ec2cde4467 test: bind daemon ports without probe races 2026-07-10 23:06:51 +08:00
NianJiuZst c5e83eac4c test: bind daemon ports without probe races 2026-07-10 23:06:46 +08:00
NianJiuZst 9f82d89788 test: bind daemon ports without probe races 2026-07-10 23:06:02 +08:00
NianJiuZst 2964f6219d ci: retry stalled Rust job 2026-07-10 22:59:01 +08:00
NianJiuZst f8907e5f5a ci: retry stalled Rust job 2026-07-10 22:58:57 +08:00
NianJiuZst d422f655d4 ci: retry stalled Rust job 2026-07-10 22:50:15 +08:00
NianJiuZst d4d4c3beeb ci: retry stalled Rust job 2026-07-10 22:50:01 +08:00
NianJiuZst 3c7a22e7c7 chore(ci): match Biome 2.4 formatting 2026-07-10 22:40:27 +08:00
NianJiuZst dca2b58cbd fix(doctor): exit nonzero on failed checks 2026-07-10 22:39:41 +08:00
NianJiuZst 13279a7f56 fix(daemon): enforce session idle timeout 2026-07-10 22:36:13 +08:00
NianJiuZst ed6d14eab3 fix(extension): fail closed on borrow confirmation 2026-07-10 22:01:46 +08:00
BB-fat 913a3f80a4 Merge pull request #13 from zjxzjw/fix2
config(extension): 配置WebSocket连接URL为构建时变量
2026-07-10 00:30:40 +08:00
BB-fat f60d1e65a7 Merge pull request #12 from NianJiuZst/codex/issue-10-pr-ci
Add PR CI workflow
2026-07-10 00:23:36 +08:00
BB-fat d4a89e89f1 Merge pull request #14 from Tencent/chore/fix-pnpm-lint
Fix pnpm lint formatting
2026-07-10 00:21:36 +08:00
BB-fatandCursor 3c936982a3 Fix Biome and Stylelint formatting so pnpm lint passes.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-10 00:21:19 +08:00
engineer 37056b3c6e config(extension): 配置WebSocket连接URL为构建时变量
- 移除DEFAULT_DAEMON_PORT常量定义
- 使用__BSK_DAEMON_WS_URL__构建时变量替代硬编码端口
- 在wxt.config.ts和vitest.config.ts中添加环境变量配置
- 为__BSK_DAEMON_WS_URL__添加类型声明和文档注释
- 默认值保持为ws://127.0.0.1:52800,可通过BSK_DAEMON_WS_URL环境变量覆盖
2026-07-09 13:44:48 +08:00
NianJiuZst fa3133a094 Add PR CI workflow 2026-07-09 12:42:42 +08:00
BB-fat d395faa0ab Merge pull request #9 from Tencent/feat/bsk-cli-auto-update
Add bsk CLI auto update support
2026-07-08 14:20:31 +08:00
BB-fatandCursor a7c48275f9 feat: add bsk CLI auto update
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-08 14:14:00 +08:00
BB-fatandCursor 647ae659b7 feat: add placeholder for bsk CLI auto update
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-07 17:20:59 +08:00
BB-fatandCursor eda8b22c43 chore: bump CLI to 0.1.7 and extension to 0.1.3
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-07 17:11:22 +08:00
BB-fat c987e42da9 Merge pull request #7 from Tencent/feat/console-command
feat: add bsk console command
2026-07-07 11:27:09 +08:00
BB-fatandCursor 7f5093d97a feat: add console observation command
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-07 11:08:12 +08:00
BB-fat ce29ddaa53 Merge pull request #5 from hobostay/fix/borrow-confirmation-tabs-get-fail-open
fix(extension): don't fail-open the borrow gate on a transient tabs.get error
2026-07-06 11:46:39 +08:00
BB-fat 2e7998477d Merge pull request #4 from hobostay/fix/ws-event-ownership
fix(daemon): reject cross-browser session events from non-owners
2026-07-06 11:44:19 +08:00
BB-fat c9e00bb641 Merge pull request #3 from hobostay/fix/tab-return-fallback-into-agent-window
fix(extension): never return a borrowed tab into another session's Agent Window
2026-07-06 11:33:38 +08:00
Test UserandClaude ad2a12e57e fix(extension): don't fail-open the borrow gate on a transient tabs.get error
requestBorrowConfirmation returned `true` (approve the borrow) when
`chrome.tabs.get(tabId)` threw. That fail-open was not one of the
documented fail-open cases in the function's own doc comment (no
candidate window; abort signal; timeout), and the tab was already
fetched successfully moments earlier by the borrow pipeline
(validateBorrowTarget), so a failure here is a transient SW/page hiccup
— not a missing tab. Silently approving a borrow with no confirmation
UI shown defeats the entire human-in-the-loop gate for seizing a user's
logged-in tab.

On a tabs.get failure, fall back to a generic title (the numeric tab id)
and still run the overlay + OS-notification confirmation flow so the
user gets a real chance to approve or deny.

Adds two regression tests (both fail on the unpatched code). Full
extension suite green (361 tests).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-02 21:02:54 +08:00
Test UserandClaude 61096110c0 fix(daemon): reject cross-browser session events from non-owners
The daemon dispatched inbound `session.window_closed` and
`session.user_interrupt` events using only the payload — the sending
browser's identity was never checked against the session's owner.

Any connected extension could therefore emit window_closed with
another browser's session_id to tear down its session (forgetting it,
dropping its tool queue + interrupt marker), or emit user_interrupt
to cancel every inflight tool of another browser's session and arm
its interrupt marker. This is a cross-tenant integrity/DoS hole that
is exploitable even between two legitimate extensions, independent of
the (separately tracked) WS Origin allow-list gap.

Thread the sender's BrowserId (already in scope as `client.id` in
handle_inbound_text) into both handlers and ignore the event when the
named session is live and owned by a different browser. A session
that is already gone is left to the existing no-op forget path.

Adds four ownership tests (owner honoured / non-owner ignored for each
event); updated two existing tests for the new signature. Full bsk
suite green (165 lib + integration), no new clippy warnings.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-02 20:47:42 +08:00
Test UserandClaude 8e8dd3840b fix(extension): never return a borrowed tab into another session's Agent Window
When a borrowed tab's original window is gone, tab_return's fallback
picker called getLastFocused({windowTypes:["normal"]}) and only
excluded the *current* session's Agent Window. Agent Windows are
created with type:"normal", so the picker could move a returned user
tab into *another* session's Agent Window. That session could then
write to the tab (it passes enforceAgentWindow), and the tab would be
destroyed when that session stops and closes its window — a
cross-session sandbox break plus silent user-data loss.

Thread an isAgentWindowId predicate (derived from SessionManager in
the production handlers, mirroring the existing filter in
borrow-confirmation's listConfirmationCandidates) through
returnBorrowedTab -> chooseFallbackWindow so every live Agent Window
is skipped as a fallback target.

Adds two regression tests (both fail on the unpatched code).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-02 20:23:02 +08:00
BB-fatandCursor 9c229b2489 chore: bump CLI version to 0.1.6
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-30 18:14:26 +08:00
BB-fatandCursor 26e5f98c16 docs: open extension install page from agent guide
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-25 15:55:20 +08:00
BB-fatandCursor dc8a6d38b3 Fix browser wait during daemon startup
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-23 15:04:20 +08:00
BB-fat ab6aa16dfc feat: add Git Bash PATH support to installer
- Add Add-ToBashProfile helper to append install dir to ~/.bashrc
- Converts Windows paths to Unix-style for Git Bash compatibility
- Integrate into Main install flow alongside Add-ToUserPath
- Update completion message to mention Git Bash
2026-06-23 11:00:47 +08:00
BB-fat 45c85fa64a feat: add PowerShell install script for Windows
- Add install.ps1 mirrored from install.sh patterns
  - Same env var overrides (BSK_REPO, BSK_VERSION, BSK_INSTALL_DIR)
  - Default install dir ~/.local/bin
  - Fetches latest version from version.json via Invoke-RestMethod
  - Downloads, extracts zip, installs bsk.exe
  - Adds to user PATH persistently with idempotent check
  - Supports x64 and ARM64 architecture detection
- Update README.md Windows section with irm | iex one-liner
- Update AGENT_INSTALL.md with Windows PowerShell instructions
2026-06-23 10:45:29 +08:00
BB-fat b8c4be4362 docs: move video link after intro paragraph in both READMEs 2026-06-22 17:55:42 +08:00
BB-fat 335391c2bf docs: sync video link to Chinese README 2026-06-22 17:51:20 +08:00
BB-fat 3cf81d509b chore: add demo video to README 2026-06-22 17:49:52 +08:00
204 changed files with 28230 additions and 1465 deletions
+84
View File
@@ -0,0 +1,84 @@
name: CI
on:
pull_request:
types:
- opened
- synchronize
- reopened
- ready_for_review
push:
branches:
- main
permissions:
contents: read
concurrency:
group: ci-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
rust:
name: Rust fmt, clippy, tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: dtolnay/rust-toolchain@stable
- name: Check Rust formatting
run: cargo fmt --all -- --check
- name: Run clippy
run: cargo clippy --workspace --all-targets --locked -- -D warnings
- name: Run Rust tests
run: cargo test --workspace --locked
frontend:
name: Frontend lint, typecheck, tests, build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: pnpm/action-setup@v6
- uses: actions/setup-node@v6
with:
node-version: 22
cache: pnpm
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Prepare extension types
run: pnpm --filter @browser-skill/extension exec wxt prepare
- name: Run lint checks
run: pnpm lint
- name: Compile extension TypeScript
run: pnpm --filter @browser-skill/extension compile
- name: Run extension tests
run: pnpm ext:test
- name: Build extension
run: pnpm ext:build
node-scripts:
name: Node script tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: actions/setup-node@v6
with:
node-version: 22
- name: Run script tests
run: node --test scripts/*.test.mjs
+1
View File
@@ -139,6 +139,7 @@ jobs:
--repo "${{ github.repository }}" \
--server-url "${{ github.server_url }}" \
--branch "${{ github.event.repository.default_branch }}" \
--dist dist \
--out dist/version.json
- name: Publish GitHub Release
+199
View File
@@ -126,3 +126,202 @@ jobs:
make_latest: false
generate_release_notes: true
files: dist/*.zip
publish-cws:
needs: [resolve, build-extension]
runs-on: ubuntu-latest
environment: chrome-web-store
permissions:
contents: read
steps:
- name: Download build artifact
uses: actions/download-artifact@v8
with:
name: extension-zip
path: dist
- name: Validate Chrome Web Store configuration
shell: bash
env:
PUBLISHER_ID: ${{ vars.CHROME_WEBSTORE_PUBLISHER_ID }}
EXTENSION_ID: ${{ vars.CHROME_WEBSTORE_EXTENSION_ID }}
run: |
set -euo pipefail
if [ -z "$PUBLISHER_ID" ]; then
echo "::error::Repository variable CHROME_WEBSTORE_PUBLISHER_ID is not configured"
exit 1
fi
if [ -z "$EXTENSION_ID" ]; then
echo "::error::Repository variable CHROME_WEBSTORE_EXTENSION_ID is not configured"
exit 1
fi
ARCHIVE="$(find dist -maxdepth 1 -name '*.zip' -type f -print -quit)"
if [ -z "$ARCHIVE" ]; then
echo "::error::No extension zip found in the downloaded artifact"
exit 1
fi
echo "ARCHIVE=$ARCHIVE" >> "$GITHUB_ENV"
- name: Authenticate with service account
id: cws-auth
env:
SERVICE_ACCOUNT_JSON: ${{ secrets.CHROME_WEBSTORE_SERVICE_ACCOUNT_JSON }}
shell: bash
run: |
set -euo pipefail
node --input-type=module <<'NODE'
import { sign } from "node:crypto";
import { appendFileSync } from "node:fs";
const rawCredentials = process.env.SERVICE_ACCOUNT_JSON;
if (!rawCredentials) {
throw new Error(
"Repository secret CHROME_WEBSTORE_SERVICE_ACCOUNT_JSON is not configured",
);
}
let credentials;
try {
credentials = JSON.parse(rawCredentials);
} catch {
throw new Error(
"Repository secret CHROME_WEBSTORE_SERVICE_ACCOUNT_JSON is not valid JSON",
);
}
if (!credentials.client_email || !credentials.private_key) {
throw new Error(
"Service account JSON must contain client_email and private_key",
);
}
const encode = (value) =>
Buffer.from(JSON.stringify(value)).toString("base64url");
const issuedAt = Math.floor(Date.now() / 1000);
const header = {
alg: "RS256",
typ: "JWT",
kid: credentials.private_key_id,
};
const claim = {
iss: credentials.client_email,
scope: "https://www.googleapis.com/auth/chromewebstore",
aud: "https://oauth2.googleapis.com/token",
iat: issuedAt,
exp: issuedAt + 3600,
};
const unsignedJwt = `${encode(header)}.${encode(claim)}`;
const signature = sign(
"RSA-SHA256",
Buffer.from(unsignedJwt),
credentials.private_key,
).toString("base64url");
const assertion = `${unsignedJwt}.${signature}`;
const response = await fetch("https://oauth2.googleapis.com/token", {
method: "POST",
headers: { "Content-Type": "application/x-www-form-urlencoded" },
body: new URLSearchParams({
grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer",
assertion,
}),
});
const responseText = await response.text();
if (!response.ok) {
throw new Error(
`Service account token exchange failed (${response.status}): ${responseText}`,
);
}
const tokenResponse = JSON.parse(responseText);
if (!tokenResponse.access_token) {
throw new Error("Service account token response has no access_token");
}
console.log(`::add-mask::${tokenResponse.access_token}`);
appendFileSync(
process.env.GITHUB_OUTPUT,
`access_token=${tokenResponse.access_token}\n`,
);
NODE
- name: Upload and publish Chrome Web Store extension
shell: bash
env:
ACCESS_TOKEN: ${{ steps.cws-auth.outputs.access_token }}
PUBLISHER_ID: ${{ vars.CHROME_WEBSTORE_PUBLISHER_ID }}
EXTENSION_ID: ${{ vars.CHROME_WEBSTORE_EXTENSION_ID }}
run: |
set -euo pipefail
ITEM_NAME="publishers/${PUBLISHER_ID}/items/${EXTENSION_ID}"
API_ROOT="https://chromewebstore.googleapis.com"
UPLOAD_RESPONSE="$RUNNER_TEMP/cws-upload-response.json"
STATUS_RESPONSE="$RUNNER_TEMP/cws-status-response.json"
PUBLISH_RESPONSE="$RUNNER_TEMP/cws-publish-response.json"
HTTP_STATUS="$(curl --silent --show-error \
--output "$UPLOAD_RESPONSE" \
--write-out '%{http_code}' \
--request POST \
--header "Authorization: Bearer $ACCESS_TOKEN" \
--header "Content-Type: application/zip" \
--data-binary "@$ARCHIVE" \
"$API_ROOT/upload/v2/$ITEM_NAME:upload")"
if [ "$HTTP_STATUS" -lt 200 ] || [ "$HTTP_STATUS" -ge 300 ]; then
echo "::error::Chrome Web Store upload failed with HTTP $HTTP_STATUS"
jq . "$UPLOAD_RESPONSE" || cat "$UPLOAD_RESPONSE"
exit 1
fi
UPLOAD_STATE="$(jq -r '.uploadState // empty' "$UPLOAD_RESPONSE")"
echo "Chrome Web Store upload state: $UPLOAD_STATE"
if [ "$UPLOAD_STATE" = "IN_PROGRESS" ] || [ "$UPLOAD_STATE" = "UPLOAD_IN_PROGRESS" ]; then
for attempt in $(seq 1 30); do
sleep 5
HTTP_STATUS="$(curl --silent --show-error \
--output "$STATUS_RESPONSE" \
--write-out '%{http_code}' \
--header "Authorization: Bearer $ACCESS_TOKEN" \
"$API_ROOT/v2/$ITEM_NAME:fetchStatus")"
if [ "$HTTP_STATUS" -lt 200 ] || [ "$HTTP_STATUS" -ge 300 ]; then
echo "::error::Chrome Web Store status check failed with HTTP $HTTP_STATUS"
jq . "$STATUS_RESPONSE" || cat "$STATUS_RESPONSE"
exit 1
fi
UPLOAD_STATE="$(jq -r '.lastAsyncUploadState // empty' "$STATUS_RESPONSE")"
echo "Upload status check $attempt/30: $UPLOAD_STATE"
if [ "$UPLOAD_STATE" = "SUCCEEDED" ]; then
break
fi
if [ "$UPLOAD_STATE" != "IN_PROGRESS" ] && [ "$UPLOAD_STATE" != "UPLOAD_IN_PROGRESS" ]; then
echo "::error::Chrome Web Store upload did not succeed: $UPLOAD_STATE"
jq . "$STATUS_RESPONSE"
exit 1
fi
done
fi
if [ "$UPLOAD_STATE" != "SUCCEEDED" ]; then
echo "::error::Chrome Web Store upload did not complete: $UPLOAD_STATE"
jq . "$UPLOAD_RESPONSE"
exit 1
fi
HTTP_STATUS="$(curl --silent --show-error \
--output "$PUBLISH_RESPONSE" \
--write-out '%{http_code}' \
--request POST \
--header "Authorization: Bearer $ACCESS_TOKEN" \
--header "Content-Type: application/json" \
--data '{"publishType":"DEFAULT_PUBLISH"}' \
"$API_ROOT/v2/$ITEM_NAME:publish")"
if [ "$HTTP_STATUS" -lt 200 ] || [ "$HTTP_STATUS" -ge 300 ]; then
echo "::error::Chrome Web Store publish failed with HTTP $HTTP_STATUS"
jq . "$PUBLISH_RESPONSE" || cat "$PUBLISH_RESPONSE"
exit 1
fi
echo "Chrome Web Store submission created successfully:"
jq '{itemId, state, warningInfo}' "$PUBLISH_RESPONSE"
+17 -4
View File
@@ -6,13 +6,22 @@ extension yourself (Step 3 is the user's job).
## 1. Install the CLI and skill
Run the install command from [`README.md`](README.md) (macOS/Linux):
Run the install command from [`README.md`](README.md):
**macOS / Linux:**
```bash
curl -fsSL https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.sh | sh
bsk install-skill --yes
```
**Windows (PowerShell):**
```powershell
irm https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.ps1 | iex
bsk install-skill --yes
```
## 2. Run `bsk doctor`
```bash
@@ -22,12 +31,16 @@ bsk doctor
Each `fail` row prints a `hint` — follow it and re-run once. A fresh install
where only `extension connected` fails is expected; go to Step 3.
## 3. Ask the user to install the extension
## 3. Open the extension install page
If `extension connected` is `FAIL` (`0 browsers connected`), tell the user:
If `extension connected` is `FAIL` (`0 browsers connected`), the user likely
has not installed or enabled the browser extension yet. Open the
[Chrome Web Store install page](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi)
in the user's browser, then tell the user:
> Install **BrowserSkill** from the
> [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi),
> open the popup, and wait until it turns green. Reply when done.
Then run `bsk doctor` once more. All `ok`/`na` → tell the user it's ready.
If opening the page fails, give the user the same link. Then run `bsk doctor`
once more. All `ok`/`na` → tell the user it's ready.
Generated
+1220 -10
View File
File diff suppressed because it is too large Load Diff
+7 -2
View File
@@ -7,7 +7,7 @@ members = [
default-members = ["crates/bsk-cli"]
[workspace.package]
version = "0.1.5"
version = "0.1.10"
edition = "2024"
rust-version = "1.85"
license = "MIT"
@@ -31,13 +31,18 @@ console = "0.15"
dialoguer = "0.11"
dirs = "5"
fs2 = "0.4"
reqwest = { version = "0.13.4", default-features = false, features = ["blocking", "rustls"] }
flate2 = "1.1.9"
tar = "0.4.46"
zip = { version = "7.2.0", default-features = false, features = ["deflate"] }
sha2 = "0.11.0"
tempfile = "3"
nix = { version = "0.29", features = ["signal", "process"] }
rand = "0.8"
uuid = { version = "1", features = ["v4", "serde"] }
# Workspace crates (version required for `cargo publish` of dependents)
bsk-protocol = { version = "0.1.5", path = "crates/bsk-protocol" }
bsk-protocol = { version = "0.1.9", path = "crates/bsk-protocol" }
[profile.release]
strip = "symbols"
+7 -3
View File
@@ -20,6 +20,8 @@ Need the agent to touch a tab you already have open? It must borrow that tab
explicitly, return it when the task is done, and leave the rest of your browser
alone.
https://github.com/user-attachments/assets/db782c92-b1d4-4aae-a255-039675937a90
## BrowserSkill Advantages
- **Reuse real login state**: Agents can work with sites you are already signed
@@ -75,9 +77,11 @@ Install the CLI, then install the extension from the [Chrome Web Store](https://
curl -fsSL https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.sh | sh
```
**Windows**: download `bsk-v<version>-x86_64-pc-windows-msvc.zip` from the
[latest CLI release](https://github.com/Tencent/BrowserSkill/releases/latest)
and add the extracted `bsk.exe` to your `PATH`.
**Windows** (PowerShell — installs to `~/.local/bin`):
```powershell
irm https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.ps1 | iex
```
Verify the binary:
+2
View File
@@ -16,6 +16,8 @@
需要 Agent 操作你已打开的标签页?必须显式借用该标签,任务结束后归还,其余浏览器窗口不受影响。
https://github.com/user-attachments/assets/db782c92-b1d4-4aae-a255-039675937a90
## BrowserSkill 的优势
- **复用真实登录态**:Agent 可以操作你已经登录的网站,不需要额外测试账号。
+1
View File
@@ -48,6 +48,7 @@ The Extension requests the following Chrome permissions. Each is used solely for
- **`tabs`** — Inspect, create, and close tabs in the Agent Window; query tab metadata.
- **`windows`** — Create and manage the dedicated Agent Window that isolates agent activity from the user's normal browsing.
- **`alarms`** — Periodically wake the service worker to keep the local WebSocket connection alive.
- **`idle`** — Detect when the device returns from idle/locked so the Extension can promptly re-establish the local WebSocket connection after the machine wakes. No idle data is stored or transmitted.
- **`notifications`** — Show a system notification to obtain user approval before the agent borrows a user-owned tab.
- **`storage`** — Persist a random instance ID and optional label in `chrome.storage.local`.
- **Host permission `<all_urls>`** — Inject a small status overlay (showing "Agent Active") on pages controlled by the agent, and enable automation across whatever sites the user directs the agent to. The Extension does **not** read or transmit page content from sites the agent is not actively driving.
+4
View File
@@ -0,0 +1,4 @@
<svg width="24" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M6.30589 17.9712C6.21652 18.4603 6.63298 18.8722 7.12749 18.7838C7.88158 18.6491 8.86635 18.4075 9.25454 18.0236C9.49544 17.7853 9.61164 17.4432 9.57757 17.0726C9.5435 16.702 9.36195 16.3331 9.07287 16.0472C8.78378 15.7613 8.41084 15.5818 8.03608 15.5481C7.66133 15.5144 7.31545 15.6293 7.07455 15.8676C6.6865 16.2513 6.4422 17.2253 6.30589 17.9712Z" fill="#E94500"/>
<path d="M7.77425 8.64281C10.7374 4.12636 15.4876 3.7066 17.9364 4.13499C18.3335 4.20461 18.6474 4.48261 18.7821 4.84105C20.6715 9.87191 18.4167 13.7056 16.5008 15.5334C16.1262 15.8909 15.9395 16.3745 16.0282 16.8215L16.12 17.2834C16.24 17.887 15.871 18.4819 15.2723 18.6496L12.7811 19.3469C12.1057 19.536 11.4124 19.1112 11.285 18.4289L11.036 17.0959C10.9844 16.82 10.8353 16.5712 10.6151 16.3948L8.12777 14.4035C7.90665 14.2266 7.62818 14.1345 7.34359 14.1428L5.72835 14.1897C4.9877 14.2112 4.40832 13.5657 4.51644 12.8401L4.89632 10.2912C4.98194 9.71734 5.47627 9.2895 6.06234 9.28148L6.69124 9.27171C7.11716 9.26594 7.52243 9.02649 7.77425 8.64281ZM14.4959 7.91429C13.8284 7.91436 13.2869 8.45568 13.2869 9.12327C13.2871 9.79078 13.8284 10.3322 14.4959 10.3323C15.1635 10.3323 15.7048 9.79082 15.7049 9.12327C15.7049 8.45564 15.1636 7.91429 14.4959 7.91429Z" fill="#E94500"/>
</svg>

After

Width:  |  Height:  |  Size: 1.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.9 KiB

+2 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@browser-skill/extension",
"version": "0.1.2",
"version": "0.1.5",
"private": true,
"type": "module",
"scripts": {
@@ -14,6 +14,7 @@
"dependencies": {
"@browser-skill/i18n": "workspace:*",
"@browser-skill/ui": "workspace:*",
"@browser-skill/vom": "workspace:*",
"@remixicon/react": "^4.9.0",
"react": "^19.0.0",
"react-dom": "^19.0.0"
@@ -119,27 +119,60 @@ describe("ChromiumCdp", () => {
expect(api.sendCommand).toHaveBeenCalledWith({ tabId: 9 }, "Page.enable", {});
});
it("enables console capture domains best-effort during attach", async () => {
const { api } = fakeApi();
(api.sendCommand as ReturnType<typeof vi.fn>).mockImplementation(
async (_target, method: string) => {
if (method === "Log.enable") throw new Error("restricted");
return {};
},
);
const cdp = new ChromiumCdp(api);
await expect(cdp.ensureAttached(12)).resolves.toBeUndefined();
expect(cdp.isAttached(12)).toBe(true);
expect(api.sendCommand).toHaveBeenCalledWith({ tabId: 12 }, "Runtime.enable", {});
expect(api.sendCommand).toHaveBeenCalledWith({ tabId: 12 }, "Log.enable", {});
await cdp.ensureConsoleCapture(12);
const enableCalls = (api.sendCommand as ReturnType<typeof vi.fn>).mock.calls.filter(
([, method]) => method === "Runtime.enable" || method === "Log.enable",
);
expect(enableCalls).toHaveLength(2);
});
it("retries and surfaces Network.enable failures for explicit capture", async () => {
const { api } = fakeApi();
(api.sendCommand as ReturnType<typeof vi.fn>).mockImplementation(
async (_target, method: string) => {
if (method === "Network.enable") throw new Error("network restricted");
return {};
},
);
const cdp = new ChromiumCdp(api);
await expect(cdp.ensureAttached(13)).resolves.toBeUndefined();
await expect(cdp.ensureNetworkCapture(13)).rejects.toThrow("network restricted");
const enableCalls = (api.sendCommand as ReturnType<typeof vi.fn>).mock.calls.filter(
([, method]) => method === "Network.enable",
);
expect(enableCalls).toHaveLength(2);
});
it("records javascriptDialogOpening and auto-accepts", async () => {
const { api, onEvent } = fakeApi();
const cdp = new ChromiumCdp(api);
await cdp.ensureAttached(3);
const cursor = cdp.dialogCursor(3);
onEvent.fire(
{ tabId: 3 },
"Page.javascriptDialogOpening",
{
type: "alert",
message: "hello",
url: "https://example.com/",
hasBrowserHandler: false,
},
);
onEvent.fire({ tabId: 3 }, "Page.javascriptDialogOpening", {
type: "alert",
message: "hello",
url: "https://example.com/",
hasBrowserHandler: false,
});
await Promise.resolve();
expect(api.sendCommand).toHaveBeenCalledWith(
{ tabId: 3 },
"Page.handleJavaScriptDialog",
{ accept: true },
);
expect(api.sendCommand).toHaveBeenCalledWith({ tabId: 3 }, "Page.handleJavaScriptDialog", {
accept: true,
});
const dialogs = cdp.dialogsSince(3, cursor);
expect(dialogs).toHaveLength(1);
expect(dialogs[0]).toMatchObject({
@@ -170,11 +203,11 @@ describe("ChromiumCdp", () => {
await cdp.ensureAttached(5);
const pending = cdp.send(5, "Runtime.evaluate", { expression: "1+1" });
await Promise.resolve();
onEvent.fire(
{ tabId: 5 },
"Page.javascriptDialogOpening",
{ type: "alert", message: "blocked", url: "https://example.com/" },
);
onEvent.fire({ tabId: 5 }, "Page.javascriptDialogOpening", {
type: "alert",
message: "blocked",
url: "https://example.com/",
});
releaseEvaluate();
await expect(pending).resolves.toEqual({ result: { value: 2 } });
});
@@ -183,17 +216,266 @@ describe("ChromiumCdp", () => {
const { api, onEvent } = fakeApi();
const cdp = new ChromiumCdp(api);
await cdp.ensureAttached(11);
onEvent.fire(
{ tabId: 11 },
"Page.javascriptDialogOpening",
{ type: "alert", message: "x", url: "https://example.com/" },
);
onEvent.fire({ tabId: 11 }, "Page.javascriptDialogOpening", {
type: "alert",
message: "x",
url: "https://example.com/",
});
await Promise.resolve();
expect(cdp.dialogsSince(11, 0)).toHaveLength(1);
await cdp.detach(11);
expect(cdp.dialogsSince(11, 0)).toHaveLength(0);
});
it("records Runtime console calls with bounded text and optional stack frames", async () => {
const { api, onEvent } = fakeApi();
const cdp = new ChromiumCdp(api);
await cdp.ensureAttached(21);
onEvent.fire({ tabId: 21 }, "Runtime.consoleAPICalled", {
type: "warn",
args: [{ value: "abcdefghi" }, { description: "Object { ok: true }" }],
timestamp: 1234.5,
stackTrace: {
callFrames: [
{
functionName: "render",
url: "https://example.test/app.js",
lineNumber: 4,
columnNumber: 8,
},
],
},
});
const withoutStack = cdp.consoleEntriesSince(21, 0, 50, 5, false);
expect(withoutStack).toMatchObject({
next_since: 1,
truncated: true,
entries: [
{
sequence: 1,
kind: "console",
level: "warn",
text: "abcde",
url: "https://example.test/app.js",
line: 5,
column: 9,
truncated: true,
},
],
});
expect(withoutStack.entries[0].stack_trace).toBeUndefined();
const withStack = cdp.consoleEntriesSince(21, 0, 50, 1000, true);
expect(withStack.entries[0].stack_trace).toEqual([
{
function_name: "render",
url: "https://example.test/app.js",
line: 5,
column: 9,
},
]);
});
it("records exceptions and engine log entries", async () => {
const { api, onEvent } = fakeApi();
const cdp = new ChromiumCdp(api);
await cdp.ensureAttached(22);
onEvent.fire({ tabId: 22 }, "Runtime.exceptionThrown", {
timestamp: 2000,
exceptionDetails: {
text: "Uncaught",
url: "https://example.test/app.js",
lineNumber: 40,
columnNumber: 6,
exception: { description: "TypeError: boom\n at render (app.js:41:7)" },
},
});
onEvent.fire({ tabId: 22 }, "Log.entryAdded", {
entry: {
level: "error",
text: "Failed to load resource: 404",
url: "https://example.test/missing.png",
lineNumber: 0,
timestamp: 2100,
},
});
const result = cdp.consoleEntriesSince(22, 0, 50, 1000, false);
expect(result.next_since).toBe(2);
expect(result.entries).toMatchObject([
{
sequence: 1,
kind: "exception",
level: "error",
text: "TypeError: boom",
url: "https://example.test/app.js",
line: 41,
column: 7,
},
{
sequence: 2,
kind: "log",
level: "error",
text: "Failed to load resource: 404",
url: "https://example.test/missing.png",
},
]);
});
it("filters console entries by cursor and caps result size", async () => {
const { api, onEvent } = fakeApi();
const cdp = new ChromiumCdp(api);
await cdp.ensureAttached(23);
for (let i = 1; i <= 3; i += 1) {
onEvent.fire({ tabId: 23 }, "Runtime.consoleAPICalled", {
type: "log",
args: [{ value: `message ${i}` }],
});
}
const result = cdp.consoleEntriesSince(23, 1, 1, 1000, false);
expect(result).toMatchObject({
next_since: 2,
truncated: true,
entries: [{ sequence: 2, text: "message 2" }],
});
expect(cdp.consoleEntriesSince(23, undefined, 1, 1000, false)).toMatchObject({
next_since: 3,
truncated: true,
entries: [{ sequence: 3, text: "message 3" }],
});
});
it("reports dropped buffered entries only when they affect the cursor", async () => {
const { api, onEvent } = fakeApi();
const cdp = new ChromiumCdp(api);
await cdp.ensureAttached(25);
for (let i = 1; i <= 201; i += 1) {
onEvent.fire({ tabId: 25 }, "Runtime.consoleAPICalled", {
type: "log",
args: [{ value: `message ${i}` }],
});
}
expect(cdp.consoleEntriesSince(25, 0, 200, 1000, false).truncated).toBe(true);
expect(cdp.consoleEntriesSince(25, 200, 200, 1000, false)).toMatchObject({
next_since: 201,
truncated: false,
entries: [{ sequence: 201, text: "message 201" }],
});
});
it("clears console state on detach", async () => {
const { api, onEvent } = fakeApi();
const cdp = new ChromiumCdp(api);
await cdp.ensureAttached(24);
onEvent.fire({ tabId: 24 }, "Runtime.consoleAPICalled", {
type: "log",
args: [{ value: "x" }],
});
expect(cdp.consoleEntriesSince(24, 0, 50, 1000, false).entries).toHaveLength(1);
await cdp.detach(24);
expect(cdp.consoleEntriesSince(24, 0, 50, 1000, false).entries).toHaveLength(0);
});
it("isolates network request metadata by tab and consumes it", async () => {
const { api, onEvent } = fakeApi();
const cdp = new ChromiumCdp(api);
await cdp.ensureAttached(31);
await cdp.ensureAttached(32);
onEvent.fire({ tabId: 31 }, "Network.requestWillBeSent", {
requestId: "shared-id",
request: { url: "https://one.test/api", method: "GET" },
type: "Fetch",
});
onEvent.fire({ tabId: 32 }, "Network.requestWillBeSent", {
requestId: "shared-id",
request: { url: "https://two.test/script.js", method: "POST" },
type: "Script",
});
onEvent.fire({ tabId: 31 }, "Network.responseReceived", {
requestId: "shared-id",
response: { status: 204 },
type: "Fetch",
});
onEvent.fire({ tabId: 32 }, "Network.loadingFailed", {
requestId: "shared-id",
errorText: "net::ERR_FAILED",
type: "Script",
});
onEvent.fire({ tabId: 31 }, "Network.loadingFailed", {
requestId: "shared-id",
errorText: "net::ERR_ABORTED",
});
expect(cdp.networkEntriesSince(31, 0, 50, 1000).entries).toMatchObject([
{
kind: "response",
method: "GET",
url: "https://one.test/api",
status: 204,
},
{
kind: "failure",
url: undefined,
error_text: "net::ERR_ABORTED",
},
]);
expect(cdp.networkEntriesSince(32, 0, 50, 1000).entries).toMatchObject([
{
kind: "failure",
method: "POST",
url: "https://two.test/script.js",
error_text: "net::ERR_FAILED",
},
]);
});
it("bounds request metadata and clears it on detach", async () => {
const { api, onEvent } = fakeApi();
const cdp = new ChromiumCdp(api);
await cdp.ensureAttached(33);
const longUrl = `https://example.test/${"x".repeat(5000)}`;
onEvent.fire({ tabId: 33 }, "Network.requestWillBeSent", {
requestId: "long",
request: { url: longUrl, method: "GET" },
});
onEvent.fire({ tabId: 33 }, "Network.loadingFailed", {
requestId: "long",
errorText: "net::ERR_FAILED",
});
const bounded = cdp.networkEntriesSince(33, 0, 50, 10_000).entries[0];
expect(bounded.url).toHaveLength(4096);
expect(bounded.truncated).toBe(true);
onEvent.fire({ tabId: 33 }, "Network.requestWillBeSent", {
requestId: "long-method",
request: { url: "https://example.test/ok", method: "M".repeat(5000) },
});
onEvent.fire({ tabId: 33 }, "Network.responseReceived", {
requestId: "long-method",
response: { url: "https://example.test/ok", status: 200 },
});
const boundedResponse = cdp.networkEntriesSince(33, 1, 50, 10_000).entries[0];
expect(boundedResponse.method).toHaveLength(4096);
expect(boundedResponse.truncated).toBe(true);
onEvent.fire({ tabId: 33 }, "Network.requestWillBeSent", {
requestId: "stale",
request: { url: "https://example.test/stale", method: "GET" },
});
await cdp.detach(33);
await cdp.ensureAttached(33);
onEvent.fire({ tabId: 33 }, "Network.loadingFailed", {
requestId: "stale",
errorText: "net::ERR_ABORTED",
});
expect(cdp.networkEntriesSince(33, 0, 50, 1000).entries[0].url).toBeUndefined();
});
it("detachSession only detaches tabs no other session owns", async () => {
const { api } = fakeApi();
const cdp = new ChromiumCdp(api);
@@ -20,7 +20,17 @@
// record the payload for tool results, and auto-accept so automation
// can continue.
import type { JavaScriptDialogInfo, JavaScriptDialogType } from "@/transport/types";
import type {
ConsoleEntry,
ConsoleEntryKind,
ConsoleResult,
ConsoleStackFrame,
JavaScriptDialogInfo,
JavaScriptDialogType,
NetworkEntry,
NetworkEntryKind,
NetworkResult,
} from "@/transport/types";
/**
* Minimal slice of `chrome.debugger` the rest of the extension
@@ -75,8 +85,31 @@ export const CDP_PROTOCOL_VERSION = "1.3";
/** Per-tab monotonic sequence returned by [`ChromiumCdp.dialogCursor`]. */
export type DialogCursor = number;
/** CDP `Emulation.setDeviceMetricsOverride` payload. */
export interface DeviceMetricsOverride {
width: number;
height: number;
/** `0` asks Chrome to use the display's native factor. */
deviceScaleFactor: number;
mobile: boolean;
}
/** CDP `Emulation.setUserAgentOverride` payload. */
export interface UserAgentOverride {
userAgent: string;
acceptLanguage?: string;
/** CDP `Emulation.UserAgentMetadata` (already camelCase). */
userAgentMetadata?: Record<string, unknown>;
}
const MAX_DIALOG_BUFFER = 32;
const MAX_DIALOG_FIELD_LENGTH = 4096;
const MAX_CONSOLE_BUFFER = 200;
const MAX_CONSOLE_FIELD_LENGTH = 4096;
const MAX_CONSOLE_STACK_FRAMES = 20;
const MAX_NETWORK_BUFFER = 200;
const MAX_NETWORK_FIELD_LENGTH = 4096;
const MAX_NETWORK_REQUEST_META = 1024;
interface ParsedDialogOpening {
type: JavaScriptDialogType;
@@ -86,6 +119,18 @@ interface ParsedDialogOpening {
hasBrowserHandler?: boolean;
}
interface ParsedConsoleEntry extends Omit<ConsoleEntry, "sequence"> {}
interface ParsedNetworkEntry extends Omit<NetworkEntry, "sequence"> {}
/** Bounded request metadata remembered from `requestWillBeSent`. */
interface NetworkRequestMeta {
url: string;
method?: string;
resourceType?: string;
truncated: boolean;
}
/**
* Wrapper around `chrome.debugger` that owns the "attach once per
* tabId" cache and exposes typed `send<T>()`.
@@ -97,13 +142,24 @@ export class ChromiumCdp {
private readonly tabOwners = new Map<number, Set<string>>();
private readonly dialogBuffers = new Map<number, JavaScriptDialogInfo[]>();
private readonly dialogSequences = new Map<number, number>();
private readonly consoleBuffers = new Map<number, ConsoleEntry[]>();
private readonly consoleSequences = new Map<number, number>();
private readonly consoleDomainsAttemptedTabs = new Set<number>();
private readonly networkBuffers = new Map<number, NetworkEntry[]>();
private readonly networkSequences = new Map<number, number>();
private readonly networkDomainsEnabledTabs = new Set<number>();
private readonly networkRequestMeta = new Map<number, Map<string, NetworkRequestMeta>>();
private detachSubscription: { dispose(): void } | null = null;
private dialogSubscription: { dispose(): void } | null = null;
private consoleSubscription: { dispose(): void } | null = null;
private networkSubscription: { dispose(): void } | null = null;
constructor(api: CdpDebuggerApi = chromeDebuggerApi) {
this.api = api;
this.bindAutoDetach();
this.bindDialogHandler();
this.bindConsoleHandler();
this.bindNetworkHandler();
}
/** Attach to `tabId` if we haven't already in this driver. */
@@ -117,6 +173,8 @@ export class ChromiumCdp {
const attach = (async () => {
await this.api.attach({ tabId }, CDP_PROTOCOL_VERSION);
await this.enablePageDomain(tabId);
await this.enableConsoleDomains(tabId);
await this.enableNetworkDomainBestEffort(tabId);
this.attachedTabs.add(tabId);
})()
.catch((err) => {
@@ -153,18 +211,111 @@ export class ChromiumCdp {
return this.dialogSequences.get(tabId) ?? 0;
}
/** `Emulation.setDeviceMetricsOverride` — pin the tab's viewport metrics. */
async setDeviceMetricsOverride(tabId: number, metrics: DeviceMetricsOverride): Promise<void> {
await this.send(tabId, "Emulation.setDeviceMetricsOverride", { ...metrics });
}
/** `Emulation.clearDeviceMetricsOverride` — restore native viewport metrics. */
async clearDeviceMetricsOverride(tabId: number): Promise<void> {
await this.send(tabId, "Emulation.clearDeviceMetricsOverride", {});
}
/** `Emulation.setUserAgentOverride` — an empty `userAgent` restores the real UA. */
async setUserAgentOverride(tabId: number, override: UserAgentOverride): Promise<void> {
await this.send(tabId, "Emulation.setUserAgentOverride", { ...override });
}
/** `Emulation.setTouchEmulationEnabled`. */
async setTouchEmulationEnabled(
tabId: number,
enabled: boolean,
maxTouchPoints?: number,
): Promise<void> {
await this.send(tabId, "Emulation.setTouchEmulationEnabled", {
enabled,
...(maxTouchPoints !== undefined ? { maxTouchPoints } : {}),
});
}
/** Dialogs observed on `tabId` with sequence strictly greater than `cursor`. */
dialogsSince(tabId: number, cursor: DialogCursor): JavaScriptDialogInfo[] {
const buf = this.dialogBuffers.get(tabId) ?? [];
return buf.filter((entry) => entry.sequence > cursor);
}
/** Ensure CDP domains for console capture are enabled for this tab. */
async ensureConsoleCapture(tabId: number): Promise<void> {
if (!this.attachedTabs.has(tabId)) {
await this.ensureAttached(tabId);
return;
}
await this.enableConsoleDomains(tabId);
}
/** Console entries observed on `tabId`, bounded for agent context safety. */
consoleEntriesSince(
tabId: number,
since: number | undefined,
limit: number,
maxTextChars: number,
includeStack: boolean,
): ConsoleResult {
const buf = this.consoleBuffers.get(tabId) ?? [];
const { entries, nextSince, truncated } = readBufferedEntries(
buf,
this.consoleSequences.get(tabId) ?? 0,
since,
limit,
(entry) => projectConsoleEntry(entry, maxTextChars, includeStack),
);
return {
tab_id: tabId,
entries,
next_since: nextSince,
truncated,
};
}
/** Ensure CDP domains for network capture are enabled for this tab. */
async ensureNetworkCapture(tabId: number): Promise<void> {
if (!this.attachedTabs.has(tabId)) {
await this.ensureAttached(tabId);
}
await this.enableNetworkDomain(tabId);
}
/** Network entries observed on `tabId`, bounded for agent context safety. */
networkEntriesSince(
tabId: number,
since: number | undefined,
limit: number,
maxTextChars: number,
): NetworkResult {
const buf = this.networkBuffers.get(tabId) ?? [];
const { entries, nextSince, truncated } = readBufferedEntries(
buf,
this.networkSequences.get(tabId) ?? 0,
since,
limit,
(entry) => projectNetworkEntry(entry, maxTextChars),
);
return {
tab_id: tabId,
entries,
next_since: nextSince,
truncated,
};
}
/** Detach if attached; never throws. */
async detach(tabId: number): Promise<void> {
this.attachInFlight.delete(tabId);
if (!this.attachedTabs.has(tabId)) return;
this.attachedTabs.delete(tabId);
this.clearDialogState(tabId);
this.clearConsoleState(tabId);
this.clearNetworkState(tabId);
try {
await this.api.detach({ tabId });
} catch (err) {
@@ -204,6 +355,13 @@ export class ChromiumCdp {
this.attachedTabs.clear();
this.dialogBuffers.clear();
this.dialogSequences.clear();
this.consoleBuffers.clear();
this.consoleSequences.clear();
this.consoleDomainsAttemptedTabs.clear();
this.networkBuffers.clear();
this.networkSequences.clear();
this.networkDomainsEnabledTabs.clear();
this.networkRequestMeta.clear();
await Promise.all(
tabs.map(async (tabId) => {
try {
@@ -219,6 +377,32 @@ export class ChromiumCdp {
await this.api.sendCommand({ tabId }, "Page.enable", {});
}
private async enableConsoleDomains(tabId: number): Promise<void> {
if (this.consoleDomainsAttemptedTabs.has(tabId)) return;
this.consoleDomainsAttemptedTabs.add(tabId);
for (const method of ["Runtime.enable", "Log.enable"]) {
try {
await this.api.sendCommand({ tabId }, method, {});
} catch (err) {
console.debug("[bsk cdp] console domain enable failed", { tabId, method, err });
}
}
}
private async enableNetworkDomain(tabId: number): Promise<void> {
if (this.networkDomainsEnabledTabs.has(tabId)) return;
await this.api.sendCommand({ tabId }, "Network.enable", {});
this.networkDomainsEnabledTabs.add(tabId);
}
private async enableNetworkDomainBestEffort(tabId: number): Promise<void> {
try {
await this.enableNetworkDomain(tabId);
} catch (err) {
console.debug("[bsk cdp] network domain enable failed", { tabId, err });
}
}
private bindDialogHandler(): void {
if (this.dialogSubscription) return;
const listener = (source: chrome.debugger.Debuggee, method: string, params: unknown) => {
@@ -272,6 +456,129 @@ export class ChromiumCdp {
this.dialogSequences.delete(tabId);
}
private bindConsoleHandler(): void {
if (this.consoleSubscription) return;
const listener = (source: chrome.debugger.Debuggee, method: string, params: unknown) => {
const tabId = source.tabId;
if (typeof tabId !== "number") return;
switch (method) {
case "Runtime.consoleAPICalled":
this.appendConsole(tabId, parseConsoleApiCalled(params));
break;
case "Runtime.exceptionThrown":
this.appendConsole(tabId, parseExceptionThrown(params));
break;
case "Log.entryAdded":
this.appendConsole(tabId, parseLogEntry(params));
break;
}
};
this.api.onEvent.addListener(listener);
this.consoleSubscription = {
dispose: () => this.api.onEvent.removeListener(listener),
};
}
private appendConsole(tabId: number, entry: ParsedConsoleEntry | null): void {
if (!entry) return;
const sequence = (this.consoleSequences.get(tabId) ?? 0) + 1;
this.consoleSequences.set(tabId, sequence);
appendBoundedEntry(this.consoleBuffers, tabId, { ...entry, sequence }, MAX_CONSOLE_BUFFER);
}
private clearConsoleState(tabId: number): void {
this.consoleBuffers.delete(tabId);
this.consoleSequences.delete(tabId);
this.consoleDomainsAttemptedTabs.delete(tabId);
}
private bindNetworkHandler(): void {
if (this.networkSubscription) return;
const listener = (source: chrome.debugger.Debuggee, method: string, params: unknown) => {
const tabId = source.tabId;
if (typeof tabId !== "number") return;
switch (method) {
case "Network.requestWillBeSent":
this.rememberNetworkRequest(tabId, params);
break;
case "Network.responseReceived":
this.appendNetwork(
tabId,
parseNetworkResponse(this.takeNetworkRequest(tabId, params), params),
);
break;
case "Network.loadingFailed":
this.appendNetwork(
tabId,
parseNetworkFailure(this.takeNetworkRequest(tabId, params), params),
);
break;
}
};
this.api.onEvent.addListener(listener);
this.networkSubscription = {
dispose: () => this.api.onEvent.removeListener(listener),
};
}
private appendNetwork(tabId: number, entry: ParsedNetworkEntry | null): void {
if (!entry) return;
const sequence = (this.networkSequences.get(tabId) ?? 0) + 1;
this.networkSequences.set(tabId, sequence);
appendBoundedEntry(this.networkBuffers, tabId, { ...entry, sequence }, MAX_NETWORK_BUFFER);
}
/** Remember `requestWillBeSent` metadata so responses/failures can be attributed. */
private rememberNetworkRequest(tabId: number, params: unknown): void {
const raw = (params ?? {}) as Record<string, unknown>;
const id = typeof raw.requestId === "string" ? raw.requestId : undefined;
const request = (raw.request ?? {}) as Record<string, unknown>;
if (!id || typeof request.url !== "string") return;
const url = truncateText(request.url, MAX_NETWORK_FIELD_LENGTH);
const method = truncateOptionalText(
typeof request.method === "string" ? request.method : undefined,
MAX_NETWORK_FIELD_LENGTH,
);
const resourceType = truncateOptionalText(
typeof raw.type === "string" ? raw.type : undefined,
MAX_NETWORK_FIELD_LENGTH,
);
const tabMeta = this.networkRequestMeta.get(tabId) ?? new Map<string, NetworkRequestMeta>();
tabMeta.delete(id);
tabMeta.set(id, {
url: url.text,
method: method?.text,
resourceType: resourceType?.text,
truncated:
url.truncated || (method?.truncated ?? false) || (resourceType?.truncated ?? false),
});
while (tabMeta.size > MAX_NETWORK_REQUEST_META) {
const oldest = tabMeta.keys().next().value;
if (oldest === undefined) break;
tabMeta.delete(oldest);
}
this.networkRequestMeta.set(tabId, tabMeta);
}
/** Consume request metadata once its response or failure has been observed. */
private takeNetworkRequest(tabId: number, params: unknown): NetworkRequestMeta | undefined {
const raw = (params ?? {}) as Record<string, unknown>;
const id = typeof raw.requestId === "string" ? raw.requestId : undefined;
if (!id) return undefined;
const tabMeta = this.networkRequestMeta.get(tabId);
const info = tabMeta?.get(id);
tabMeta?.delete(id);
if (tabMeta?.size === 0) this.networkRequestMeta.delete(tabId);
return info;
}
private clearNetworkState(tabId: number): void {
this.networkBuffers.delete(tabId);
this.networkSequences.delete(tabId);
this.networkDomainsEnabledTabs.delete(tabId);
this.networkRequestMeta.delete(tabId);
}
private bindAutoDetach(): void {
if (this.detachSubscription) return;
const listener = (source: chrome.debugger.Debuggee, _reason: string) => {
@@ -280,6 +587,8 @@ export class ChromiumCdp {
this.attachInFlight.delete(source.tabId);
this.tabOwners.delete(source.tabId);
this.clearDialogState(source.tabId);
this.clearConsoleState(source.tabId);
this.clearNetworkState(source.tabId);
}
};
this.api.onDetach.addListener(listener);
@@ -294,6 +603,10 @@ export class ChromiumCdp {
this.detachSubscription = null;
this.dialogSubscription?.dispose();
this.dialogSubscription = null;
this.consoleSubscription?.dispose();
this.consoleSubscription = null;
this.networkSubscription?.dispose();
this.networkSubscription = null;
}
/** Detach tabs only when no other live session has claimed them. */
@@ -310,6 +623,45 @@ export class ChromiumCdp {
}
}
function appendBoundedEntry<T>(
buffers: Map<number, T[]>,
tabId: number,
entry: T,
maxEntries: number,
): void {
const buffer = buffers.get(tabId) ?? [];
buffer.push(entry);
while (buffer.length > maxEntries) buffer.shift();
buffers.set(tabId, buffer);
}
function readBufferedEntries<
TEntry extends { sequence: number },
TProjected extends { sequence: number; truncated: boolean },
>(
buffer: TEntry[],
currentSequence: number,
since: number | undefined,
limit: number,
project: (entry: TEntry) => TProjected,
): { entries: TProjected[]; nextSince: number; truncated: boolean } {
const hasCursor = typeof since === "number";
const candidates = hasCursor ? buffer.filter((entry) => entry.sequence > since) : buffer;
const limited = hasCursor ? candidates.slice(0, limit) : candidates.slice(-limit);
const entries = limited.map(project);
const oldestSequence = buffer[0]?.sequence ?? currentSequence + 1;
const droppedEntries =
currentSequence > buffer.length && (!hasCursor || (since ?? 0) < oldestSequence - 1);
return {
entries,
nextSince: entries.at(-1)?.sequence ?? currentSequence,
truncated:
droppedEntries ||
candidates.length > limited.length ||
entries.some((entry) => entry.truncated),
};
}
function parseDialogOpeningParams(params: unknown): ParsedDialogOpening {
const raw = (params ?? {}) as Record<string, unknown>;
const type = normalizeDialogType(raw.type);
@@ -339,6 +691,257 @@ function truncateDialogField(value: string): string {
return `${value.slice(0, MAX_DIALOG_FIELD_LENGTH)}... [truncated]`;
}
function parseConsoleApiCalled(params: unknown): ParsedConsoleEntry | null {
const raw = (params ?? {}) as Record<string, unknown>;
const args = Array.isArray(raw.args) ? raw.args : [];
const text = args.map(remoteObjectToText).filter(Boolean).join(" ");
const stackTrace = parseStackTrace(raw.stackTrace);
const top = stackTrace[0];
return makeConsoleEntry({
kind: "console",
level: typeof raw.type === "string" ? raw.type : "log",
text,
url: top?.url,
line: top?.line,
column: top?.column,
timestamp: typeof raw.timestamp === "number" ? raw.timestamp : undefined,
stack_trace: stackTrace,
});
}
function parseExceptionThrown(params: unknown): ParsedConsoleEntry | null {
const raw = (params ?? {}) as Record<string, unknown>;
const details = (raw.exceptionDetails ?? {}) as Record<string, unknown>;
const exception = (details.exception ?? {}) as Record<string, unknown>;
const text =
firstLine(typeof exception.description === "string" ? exception.description : undefined) ||
firstLine(typeof details.text === "string" ? details.text : undefined) ||
"Uncaught (unknown error)";
return makeConsoleEntry({
kind: "exception",
level: "error",
text,
url: typeof details.url === "string" ? details.url : undefined,
line: typeof details.lineNumber === "number" ? details.lineNumber + 1 : undefined,
column: typeof details.columnNumber === "number" ? details.columnNumber + 1 : undefined,
timestamp: typeof raw.timestamp === "number" ? raw.timestamp : undefined,
stack_trace: parseStackTrace(details.stackTrace),
});
}
function parseLogEntry(params: unknown): ParsedConsoleEntry | null {
const raw = (params ?? {}) as Record<string, unknown>;
const entry = (raw.entry ?? {}) as Record<string, unknown>;
return makeConsoleEntry({
kind: "log",
level: typeof entry.level === "string" ? entry.level : "info",
text: typeof entry.text === "string" ? entry.text : "",
url: typeof entry.url === "string" ? entry.url : undefined,
line: typeof entry.lineNumber === "number" ? entry.lineNumber : undefined,
timestamp: typeof entry.timestamp === "number" ? entry.timestamp : undefined,
stack_trace: [],
});
}
function makeConsoleEntry(input: {
kind: ConsoleEntryKind;
level: string;
text: string;
url?: string;
line?: number;
column?: number;
timestamp?: number;
stack_trace: ConsoleStackFrame[];
}): ParsedConsoleEntry {
const projected = truncateText(input.text, MAX_CONSOLE_FIELD_LENGTH);
return {
kind: input.kind,
level: input.level,
text: projected.text,
url: truncateOptionalConsoleField(input.url),
line: input.line,
column: input.column,
timestamp: input.timestamp,
stack_trace: input.stack_trace.slice(0, MAX_CONSOLE_STACK_FRAMES).map((frame) => ({
function_name: truncateOptionalConsoleField(frame.function_name),
url: truncateOptionalConsoleField(frame.url),
line: frame.line,
column: frame.column,
})),
truncated: projected.truncated || input.stack_trace.length > MAX_CONSOLE_STACK_FRAMES,
};
}
function projectConsoleEntry(
entry: ConsoleEntry,
maxTextChars: number,
includeStack: boolean,
): ConsoleEntry {
const projected = truncateText(entry.text, maxTextChars);
return {
sequence: entry.sequence,
kind: entry.kind,
level: entry.level,
text: projected.text,
url: entry.url,
line: entry.line,
column: entry.column,
timestamp: entry.timestamp,
...(includeStack && entry.stack_trace && entry.stack_trace.length > 0
? { stack_trace: entry.stack_trace }
: {}),
truncated: entry.truncated || projected.truncated,
};
}
function remoteObjectToText(value: unknown): string {
const raw = (value ?? {}) as Record<string, unknown>;
if ("value" in raw && raw.value !== undefined) return String(raw.value);
if (typeof raw.description === "string") return raw.description;
if (typeof raw.unserializableValue === "string") return raw.unserializableValue;
if (typeof raw.type === "string") return `[${raw.type}]`;
return "";
}
function parseStackTrace(value: unknown): ConsoleStackFrame[] {
const raw = (value ?? {}) as Record<string, unknown>;
const frames = Array.isArray(raw.callFrames) ? raw.callFrames : [];
return frames
.filter(
(frame): frame is Record<string, unknown> => frame !== null && typeof frame === "object",
)
.map((frame) => ({
function_name:
typeof frame.functionName === "string" && frame.functionName.length > 0
? frame.functionName
: undefined,
url: typeof frame.url === "string" && frame.url.length > 0 ? frame.url : undefined,
line: typeof frame.lineNumber === "number" ? frame.lineNumber + 1 : undefined,
column: typeof frame.columnNumber === "number" ? frame.columnNumber + 1 : undefined,
}));
}
function firstLine(value: string | undefined): string | undefined {
if (!value) return undefined;
return value.split(/\r?\n/, 1)[0] || undefined;
}
function truncateText(value: string, maxChars: number): { text: string; truncated: boolean } {
if (value.length <= maxChars) return { text: value, truncated: false };
return { text: value.slice(0, maxChars), truncated: true };
}
function truncateOptionalText(
value: string | undefined,
maxChars: number,
): { text: string; truncated: boolean } | undefined {
return value === undefined ? undefined : truncateText(value, maxChars);
}
function truncateOptionalConsoleField(value: string | undefined): string | undefined {
return truncateOptionalText(value, MAX_CONSOLE_FIELD_LENGTH)?.text;
}
function parseNetworkResponse(
info: NetworkRequestMeta | undefined,
params: unknown,
): ParsedNetworkEntry | null {
const raw = (params ?? {}) as Record<string, unknown>;
const response = (raw.response ?? {}) as Record<string, unknown>;
const hasResponseUrl = typeof response.url === "string";
const url = hasResponseUrl ? (response.url as string) : info?.url;
if (!url) return null;
return makeNetworkEntry({
kind: "response",
method: info?.method,
url,
status: typeof response.status === "number" ? response.status : undefined,
status_text: typeof response.statusText === "string" ? response.statusText : undefined,
mime_type: typeof response.mimeType === "string" ? response.mimeType : undefined,
resource_type: typeof raw.type === "string" ? raw.type : info?.resourceType,
timestamp: typeof raw.timestamp === "number" ? raw.timestamp : undefined,
truncated: info?.truncated === true,
});
}
function parseNetworkFailure(
info: NetworkRequestMeta | undefined,
params: unknown,
): ParsedNetworkEntry | null {
const raw = (params ?? {}) as Record<string, unknown>;
return makeNetworkEntry({
kind: "failure",
method: info?.method,
url: info?.url,
error_text: typeof raw.errorText === "string" ? raw.errorText : undefined,
resource_type: typeof raw.type === "string" ? raw.type : info?.resourceType,
timestamp: typeof raw.timestamp === "number" ? raw.timestamp : undefined,
truncated: info?.truncated === true,
});
}
function makeNetworkEntry(input: {
kind: NetworkEntryKind;
method?: string;
url?: string;
status?: number;
status_text?: string;
mime_type?: string;
resource_type?: string;
error_text?: string;
timestamp?: number;
truncated?: boolean;
}): ParsedNetworkEntry {
const projectedMethod = truncateOptionalText(input.method, MAX_NETWORK_FIELD_LENGTH);
const projectedUrl = truncateOptionalText(input.url, MAX_NETWORK_FIELD_LENGTH);
const projectedStatusText = truncateOptionalText(input.status_text, MAX_NETWORK_FIELD_LENGTH);
const projectedMimeType = truncateOptionalText(input.mime_type, MAX_NETWORK_FIELD_LENGTH);
const projectedResourceType = truncateOptionalText(input.resource_type, MAX_NETWORK_FIELD_LENGTH);
const projectedError =
input.error_text !== undefined
? truncateText(input.error_text, MAX_NETWORK_FIELD_LENGTH)
: undefined;
return {
kind: input.kind,
method: projectedMethod?.text,
url: projectedUrl?.text,
status: input.status,
status_text: projectedStatusText?.text,
mime_type: projectedMimeType?.text,
resource_type: projectedResourceType?.text,
error_text: projectedError?.text,
timestamp: input.timestamp,
truncated:
input.truncated === true ||
(projectedMethod?.truncated ?? false) ||
(projectedUrl?.truncated ?? false) ||
(projectedStatusText?.truncated ?? false) ||
(projectedMimeType?.truncated ?? false) ||
(projectedResourceType?.truncated ?? false) ||
(projectedError?.truncated ?? false),
};
}
function projectNetworkEntry(entry: NetworkEntry, maxTextChars: number): NetworkEntry {
const projectedUrl = truncateOptionalText(entry.url, maxTextChars);
const projectedError =
entry.error_text !== undefined ? truncateText(entry.error_text, maxTextChars) : undefined;
return {
sequence: entry.sequence,
kind: entry.kind,
method: entry.method,
url: projectedUrl?.text,
status: entry.status,
status_text: entry.status_text,
mime_type: entry.mime_type,
resource_type: entry.resource_type,
error_text: projectedError?.text,
timestamp: entry.timestamp,
truncated:
entry.truncated || (projectedUrl?.truncated ?? false) || (projectedError?.truncated ?? false),
};
}
function normalizeError(err: unknown): Error {
if (err instanceof Error) return err;
if (typeof err === "string") return new Error(err);
@@ -1,6 +1,6 @@
import { useTranslation } from "@browser-skill/i18n/react";
import { RiCloseLine } from "@remixicon/react";
import { useEffect, useRef, useState, type TransitionEvent } from "react";
import { type TransitionEvent, useEffect, useRef, useState } from "react";
import logoUrl from "../../assets/logo.png";
export interface BorrowRequestData {
@@ -62,33 +62,25 @@ function BorrowToastStack({ children }: { children: React.ReactNode }) {
);
}
/** Matches BorrowProgressBar / CountdownRing transition duration (duration-1000). */
const PROGRESS_TRANSITION_MS = 1000;
function BorrowRequestItem({ request, isModal }: { request: BorrowRequestData; isModal: boolean }) {
const { t } = useTranslation("extension");
const totalSeconds = Math.ceil(request.timeoutMs / 1000);
const [secondsLeft, setSecondsLeft] = useState(totalSeconds);
const [exiting, setExiting] = useState(false);
const [awaitingAutoAllow, setAwaitingAutoAllow] = useState(false);
const allowedRef = useRef(false);
const [awaitingAutoDeny, setAwaitingAutoDeny] = useState(false);
const settledRef = useRef(false);
const onAllowRef = useRef(request.onAllow);
const onDenyRef = useRef(request.onDeny);
onAllowRef.current = request.onAllow;
onDenyRef.current = request.onDeny;
useEffect(() => {
if (secondsLeft <= 0) {
if (!allowedRef.current) {
setAwaitingAutoAllow(true);
}
return;
}
const id = setTimeout(() => setSecondsLeft((s) => s - 1), 1000);
return () => clearTimeout(id);
}, [secondsLeft]);
function triggerAllow() {
if (allowedRef.current) return;
allowedRef.current = true;
setAwaitingAutoAllow(false);
if (settledRef.current) return;
settledRef.current = true;
setAwaitingAutoDeny(false);
setExiting(true);
setTimeout(() => onAllowRef.current(), EXIT_ANIMATION_MS);
}
@@ -98,13 +90,34 @@ function BorrowRequestItem({ request, isModal }: { request: BorrowRequestData; i
}
function handleDeny() {
if (allowedRef.current) return;
allowedRef.current = true;
setAwaitingAutoAllow(false);
if (settledRef.current) return;
settledRef.current = true;
setAwaitingAutoDeny(false);
setExiting(true);
setTimeout(() => onDenyRef.current(), EXIT_ANIMATION_MS);
}
useEffect(() => {
if (secondsLeft <= 0) {
if (!settledRef.current) {
setAwaitingAutoDeny(true);
}
return;
}
const id = setTimeout(() => setSecondsLeft((s) => s - 1), 1000);
return () => clearTimeout(id);
}, [secondsLeft]);
// Prefer transitionend for visual sync, but fall back to a timer so
// background tabs / suppressed CSS transitions still auto-deny.
useEffect(() => {
if (!awaitingAutoDeny || settledRef.current) return;
const id = setTimeout(() => {
handleDeny();
}, PROGRESS_TRANSITION_MS);
return () => clearTimeout(id);
}, [awaitingAutoDeny]);
const progress = secondsLeft / totalSeconds;
const truncatedTitle =
request.tabTitle.length > 30 ? `${request.tabTitle.slice(0, 30)}…` : request.tabTitle;
@@ -128,13 +141,13 @@ function BorrowRequestItem({ request, isModal }: { request: BorrowRequestData; i
progress={progress}
seconds={secondsLeft}
onProgressTransitionEnd={(propertyName) => {
if (!awaitingAutoAllow || secondsLeft !== 0) return;
if (!awaitingAutoDeny || secondsLeft !== 0) return;
if (propertyName !== "stroke-dashoffset") return;
triggerAllow();
handleDeny();
}}
/>
<span className="text-[13px] text-gray-500">
{t("borrowConfirmation.autoAllow", { count: secondsLeft })}
{t("borrowConfirmation.autoDeny", { count: secondsLeft })}
</span>
</div>
<ActionButtons onDeny={handleDeny} onAllow={handleAllow} />
@@ -156,9 +169,9 @@ function BorrowRequestItem({ request, isModal }: { request: BorrowRequestData; i
<BorrowProgressBar
progress={progress}
onProgressTransitionEnd={(event) => {
if (!awaitingAutoAllow || secondsLeft !== 0) return;
if (!awaitingAutoDeny || secondsLeft !== 0) return;
if (event.propertyName !== "width") return;
triggerAllow();
handleDeny();
}}
/>
<ActionButtons onDeny={handleDeny} onAllow={handleAllow} gapClass="gap-2" />
+37 -3
View File
@@ -1,6 +1,6 @@
import { useTranslation } from "@browser-skill/i18n/react";
import { RiStopCircleLine } from "@remixicon/react";
import { useEffect, useState } from "react";
import { useEffect, useRef, useState } from "react";
import logoUrl from "../../assets/logo.png";
export interface ControlOverlayProps {
@@ -18,6 +18,7 @@ export function ControlOverlay({
}: ControlOverlayProps) {
const { t } = useTranslation("extension");
const [show, setShow] = useState(false);
const blockerRef = useRef<HTMLDivElement>(null);
useEffect(() => {
if (visible) {
@@ -27,6 +28,36 @@ export function ControlOverlay({
setShow(false);
}, [visible]);
useEffect(() => {
const blocker = blockerRef.current;
if (!blocker) return;
const stopScroll = (event: WheelEvent | TouchEvent) => {
if (automationBypass) return;
event.preventDefault();
event.stopPropagation();
};
blocker.addEventListener("wheel", stopScroll, { passive: false });
blocker.addEventListener("touchmove", stopScroll, { passive: false });
return () => {
blocker.removeEventListener("wheel", stopScroll);
blocker.removeEventListener("touchmove", stopScroll);
};
}, [automationBypass]);
useEffect(() => {
if (!visible || automationBypass) return;
const stopScroll = (event: WheelEvent | TouchEvent) => {
event.preventDefault();
event.stopPropagation();
};
window.addEventListener("wheel", stopScroll, { capture: true, passive: false });
window.addEventListener("touchmove", stopScroll, { capture: true, passive: false });
return () => {
window.removeEventListener("wheel", stopScroll, { capture: true });
window.removeEventListener("touchmove", stopScroll, { capture: true });
};
}, [visible, automationBypass]);
if (!visible) return null;
const pointerEvents = automationBypass ? "none" : "auto";
@@ -58,6 +89,8 @@ export function ControlOverlay({
/>
<div
ref={blockerRef}
data-slot="control-overlay-blocker"
onPointerDown={(event) => {
if (automationBypass) return;
event.preventDefault();
@@ -80,13 +113,14 @@ export function ControlOverlay({
/>
<div
data-slot="control-overlay-pill"
style={{
position: "fixed",
bottom: 32,
left: "50%",
transform: "translateX(-50%)",
zIndex: 2147483647,
pointerEvents,
pointerEvents: "auto",
display: "flex",
alignItems: "center",
gap: 12,
@@ -122,7 +156,7 @@ export function ControlOverlay({
disabled={interrupting}
onClick={onInterrupt}
style={{
pointerEvents,
pointerEvents: "auto",
display: "flex",
alignItems: "center",
gap: 6,
+201 -136
View File
@@ -1,12 +1,12 @@
import { useTranslation } from "@browser-skill/i18n/react";
import { RiArrowDownSLine } from "@remixicon/react";
import {
type PointerEvent as ReactPointerEvent,
useCallback,
useEffect,
useLayoutEffect,
useRef,
useState,
type PointerEvent as ReactPointerEvent,
} from "react";
import logoUrl from "../../assets/logo.png";
@@ -15,6 +15,8 @@ export interface HelpRequestData {
prompt: string;
/** Custom overlay title; falls back to i18n when omitted. */
title?: string;
/** Full task UI on the subject tab; compact status UI on related tabs. */
displayMode?: "full" | "compact";
/** CSS selectors to scroll to + flash-highlight. */
selectors: string[];
onContinue: (note: string) => void;
@@ -46,6 +48,7 @@ const VIEWPORT_MARGIN = 16;
const PANEL_WIDTH = 420;
const FALLBACK_PANEL_H = 180;
const FALLBACK_PANEL_H_COLLAPSED = 44;
const EMPTY_SELECTORS: string[] = [];
function clamp(value: number, min: number, max: number): number {
return Math.max(min, Math.min(max, value));
@@ -174,6 +177,8 @@ function measure(selectors: string[]): Box[] {
export function HelpRequestOverlay({ request }: Props) {
const { t } = useTranslation("extension");
const isCompact = request?.displayMode === "compact";
const effectiveSelectors = isCompact ? EMPTY_SELECTORS : (request?.selectors ?? EMPTY_SELECTORS);
const [note, setNote] = useState("");
const [collapsed, setCollapsed] = useState(false);
const [boxes, setBoxes] = useState<Box[]>([]);
@@ -207,8 +212,9 @@ export function HelpRequestOverlay({ request }: Props) {
// Scroll the first matched target into view once per distinct request id.
useLayoutEffect(() => {
if (!request) return;
if (isCompact) return;
if (lastScrolledIdRef.current === request.id) return;
for (const sel of request.selectors) {
for (const sel of effectiveSelectors) {
const el = querySelectorSafe(sel);
if (el) {
lastScrolledIdRef.current = request.id;
@@ -216,16 +222,16 @@ export function HelpRequestOverlay({ request }: Props) {
break;
}
}
}, [request]);
}, [request, isCompact, effectiveSelectors]);
// Keep highlight boxes aligned with the page as it scrolls / resizes.
useEffect(() => {
if (!request) {
setBoxes([]);
setBoxes((prev) => (prev.length === 0 ? prev : []));
return;
}
const update = () => {
const nextBoxes = measure(request.selectors);
const nextBoxes = measure(effectiveSelectors);
setBoxes((prev) => (boxesEqual(prev, nextBoxes) ? prev : nextBoxes));
};
update();
@@ -237,27 +243,31 @@ export function HelpRequestOverlay({ request }: Props) {
window.removeEventListener("resize", update);
window.clearInterval(interval);
};
}, [request]);
}, [request, effectiveSelectors]);
const onHeaderPointerDown = useCallback((e: ReactPointerEvent<HTMLDivElement>) => {
if (e.button !== 0) return;
if (e.target instanceof Element && e.target.closest("button, input, textarea, a, select")) return;
const banner = bannerRef.current;
if (!banner) return;
const rect = banner.getBoundingClientRect();
dragStartRef.current = {
pointerX: e.clientX,
pointerY: e.clientY,
originLeft: rect.left,
originTop: rect.top,
panelW: banner.offsetWidth || PANEL_WIDTH,
panelH: banner.offsetHeight || (collapsed ? FALLBACK_PANEL_H_COLLAPSED : FALLBACK_PANEL_H),
};
userMovedRef.current = true;
setDragPos({ top: rect.top, left: rect.left });
setDragging(true);
e.preventDefault();
}, [collapsed]);
const onHeaderPointerDown = useCallback(
(e: ReactPointerEvent<HTMLDivElement>) => {
if (e.button !== 0) return;
if (e.target instanceof Element && e.target.closest("button, input, textarea, a, select"))
return;
const banner = bannerRef.current;
if (!banner) return;
const rect = banner.getBoundingClientRect();
dragStartRef.current = {
pointerX: e.clientX,
pointerY: e.clientY,
originLeft: rect.left,
originTop: rect.top,
panelW: banner.offsetWidth || PANEL_WIDTH,
panelH: banner.offsetHeight || (collapsed ? FALLBACK_PANEL_H_COLLAPSED : FALLBACK_PANEL_H),
};
userMovedRef.current = true;
setDragPos({ top: rect.top, left: rect.left });
setDragging(true);
e.preventDefault();
},
[collapsed],
);
useEffect(() => {
if (!dragging) return;
@@ -290,8 +300,12 @@ export function HelpRequestOverlay({ request }: Props) {
setPanelPos(null);
return;
}
if (isCompact) {
setPanelPos(null);
return;
}
if (userMovedRef.current) return;
const measured = boxes.length > 0 ? boxes : measure(request.selectors);
const measured = boxes.length > 0 ? boxes : measure(effectiveSelectors);
const anchor = unionBox(measured);
const banner = bannerRef.current;
if (!anchor || !banner) {
@@ -315,7 +329,7 @@ export function HelpRequestOverlay({ request }: Props) {
}
const nextPos = placePanel(anchor, panelW, panelH, placementRef.current.placement);
setPanelPos((prev) => (panelPosEqual(prev, nextPos) ? prev : nextPos));
}, [request, boxes, collapsed]);
}, [request, boxes, collapsed, isCompact, effectiveSelectors]);
if (!request) return null;
@@ -338,6 +352,8 @@ export function HelpRequestOverlay({ request }: Props) {
z-index: 2147483647;
display: flex;
flex-direction: column;
justify-content: flex-start;
min-height: 0;
gap: 10px;
width: var(--bsk-help-banner-width);
max-width: calc(100vw - ${VIEWPORT_MARGIN * 2}px);
@@ -355,16 +371,25 @@ export function HelpRequestOverlay({ request }: Props) {
.bsk-help-banner[data-collapsed="true"] {
gap: 0;
width: var(--bsk-help-banner-width);
padding: 10px 12px;
padding: 12px;
}
.bsk-help-banner[data-display-mode="compact"] {
width: min(360px, calc(100vw - ${VIEWPORT_MARGIN * 2}px));
padding: 12px 14px;
gap: 0;
border-radius: 12px;
}
.bsk-help-drag-strip {
flex-shrink: 0;
position: absolute;
top: 6px;
left: 0;
right: 0;
display: flex;
align-items: center;
justify-content: center;
height: 12px;
margin: -8px 0 -10px;
cursor: grab;
user-select: none;
touch-action: none;
@@ -391,15 +416,20 @@ export function HelpRequestOverlay({ request }: Props) {
}
.bsk-help-banner[data-collapsed="true"] .bsk-help-drag-strip {
margin: -6px 0 -8px;
top: 4px;
}
.bsk-help-header {
flex-shrink: 0;
display: flex;
align-items: center;
gap: 10px;
}
.bsk-help-banner[data-display-mode="compact"] .bsk-help-header {
gap: 8px;
}
.bsk-help-title {
flex: 1 1 auto;
min-width: 0;
@@ -410,31 +440,34 @@ export function HelpRequestOverlay({ request }: Props) {
color: #111;
}
.bsk-help-compact-title {
flex: 1 1 auto;
min-width: 0;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
font-size: 13px;
font-weight: 500;
color: #374151;
}
.bsk-help-banner[data-collapsed="true"] .bsk-help-title {
white-space: nowrap;
}
.bsk-help-header-actions {
flex-shrink: 0;
display: flex;
display: none;
align-items: center;
gap: 10px;
margin-left: auto;
overflow: hidden;
max-width: 0;
opacity: 0;
transform: translateX(8px);
pointer-events: none;
transition:
max-width var(--bsk-help-duration) var(--bsk-help-ease),
opacity 220ms var(--bsk-help-ease),
transform var(--bsk-help-duration) var(--bsk-help-ease);
}
.bsk-help-banner[data-collapsed="true"] .bsk-help-header-actions {
display: flex;
max-width: 280px;
opacity: 1;
transform: translateX(0);
pointer-events: auto;
}
@@ -467,15 +500,13 @@ export function HelpRequestOverlay({ request }: Props) {
}
.bsk-help-body {
display: grid;
grid-template-rows: 1fr;
display: block;
flex-shrink: 0;
min-width: 0;
transition: grid-template-rows var(--bsk-help-duration) var(--bsk-help-ease);
}
.bsk-help-banner[data-collapsed="true"] .bsk-help-body {
position: absolute;
grid-template-rows: 0fr;
width: 0;
height: 0;
overflow: hidden;
@@ -524,6 +555,7 @@ export function HelpRequestOverlay({ request }: Props) {
.bsk-help-footer-actions {
display: flex;
justify-content: flex-end;
flex-wrap: wrap;
gap: 10px;
}
@@ -535,7 +567,8 @@ export function HelpRequestOverlay({ request }: Props) {
font-size: 13px;
font-weight: 500;
color: #4b5563;
flex-shrink: 0;
min-width: 0;
white-space: normal;
}
.bsk-help-btn-continue {
@@ -546,16 +579,30 @@ export function HelpRequestOverlay({ request }: Props) {
font-size: 13px;
font-weight: 600;
color: #fff;
flex-shrink: 0;
min-width: 0;
white-space: normal;
}
.bsk-help-header-actions .bsk-help-btn-cancel,
.bsk-help-header-actions .bsk-help-btn-continue {
padding: 6px 12px;
white-space: nowrap;
}
.bsk-help-header-actions .bsk-help-btn-continue {
padding: 6px 14px;
max-width: 180px;
}
.bsk-help-banner[data-display-mode="compact"] .bsk-help-header-actions {
display: flex;
max-width: none;
}
.bsk-help-banner[data-display-mode="compact"] .bsk-help-header-actions .bsk-help-btn-cancel,
.bsk-help-banner[data-display-mode="compact"] .bsk-help-header-actions .bsk-help-btn-continue {
padding: 6px 10px;
font-size: 12px;
}
.bsk-help-footer-actions .bsk-help-btn-cancel {
@@ -580,29 +627,31 @@ export function HelpRequestOverlay({ request }: Props) {
}
`}</style>
{boxes.map((b, i) => (
<div
// biome-ignore lint/suspicious/noArrayIndexKey: boxes are positional
key={i}
data-slot="help-highlight"
style={{
position: "fixed",
top: b.top,
left: b.left,
width: b.width,
height: b.height,
borderRadius: 6,
zIndex: 2147483646,
pointerEvents: "none",
animation: "bsk-help-flash 1.2s ease-in-out infinite",
}}
/>
))}
{!isCompact &&
boxes.map((b, i) => (
<div
// biome-ignore lint/suspicious/noArrayIndexKey: boxes are positional
key={i}
data-slot="help-highlight"
style={{
position: "fixed",
top: b.top,
left: b.left,
width: b.width,
height: b.height,
borderRadius: 6,
zIndex: 2147483646,
pointerEvents: "none",
animation: "bsk-help-flash 1.2s ease-in-out infinite",
}}
/>
))}
<div
ref={bannerRef}
data-slot="help-request-banner"
data-collapsed={collapsed ? "true" : "false"}
data-display-mode={isCompact ? "compact" : "full"}
data-anchored={panelPos ? "true" : "false"}
data-dragging={dragging ? "true" : "false"}
data-placement={panelPos?.placement}
@@ -625,89 +674,105 @@ export function HelpRequestOverlay({ request }: Props) {
: undefined
}
>
<div
data-slot="help-drag-handle"
className="bsk-help-drag-strip"
role="img"
aria-label={t("helpRequest.dragHandle")}
onPointerDown={onHeaderPointerDown}
>
<span className="bsk-help-drag-pill" aria-hidden />
</div>
{!isCompact && (
<div
data-slot="help-drag-handle"
className="bsk-help-drag-strip"
role="img"
aria-label={t("helpRequest.dragHandle")}
onPointerDown={onHeaderPointerDown}
>
<span className="bsk-help-drag-pill" aria-hidden />
</div>
)}
<div className="bsk-help-header">
<img src={logoUrl} alt="browser-skill" style={{ width: 22, height: 22, borderRadius: 4 }} />
<span className="bsk-help-title">{request.title ?? t("helpRequest.title")}</span>
<div className="bsk-help-header-actions" aria-hidden={!collapsed}>
<img
src={logoUrl}
alt="browser-skill"
style={{ width: 22, height: 22, borderRadius: 4 }}
/>
{isCompact ? (
<span className="bsk-help-compact-title">{t("helpRequest.compactStatus")}</span>
) : (
<span className="bsk-help-title">{request.title ?? t("helpRequest.title")}</span>
)}
{!isCompact && (
<div className="bsk-help-header-actions" aria-hidden={!collapsed}>
<button
type="button"
data-slot={collapsed ? "help-cancel-button" : undefined}
className="bsk-help-btn-cancel"
tabIndex={collapsed ? 0 : -1}
onClick={() => request.onCancel()}
>
{t("helpRequest.cancel")}
</button>
<button
type="button"
data-slot={collapsed ? "help-continue-button" : undefined}
className="bsk-help-btn-continue"
tabIndex={collapsed ? 0 : -1}
onClick={() => request.onContinue(note)}
>
{t("helpRequest.continue")}
</button>
</div>
)}
{!isCompact && (
<button
type="button"
data-slot={collapsed ? "help-cancel-button" : undefined}
className="bsk-help-btn-cancel"
tabIndex={collapsed ? 0 : -1}
onClick={() => request.onCancel()}
data-slot="help-collapse-toggle"
className="bsk-help-collapse-toggle"
aria-label={collapsed ? t("helpRequest.expand") : t("helpRequest.collapse")}
aria-expanded={!collapsed}
onClick={() => setCollapsed((c) => !c)}
>
{t("helpRequest.cancel")}
<span className="bsk-help-collapse-icon">
<RiArrowDownSLine size={20} aria-hidden />
</span>
</button>
<button
type="button"
data-slot={collapsed ? "help-continue-button" : undefined}
className="bsk-help-btn-continue"
tabIndex={collapsed ? 0 : -1}
onClick={() => request.onContinue(note)}
>
{t("helpRequest.continue")}
</button>
</div>
<button
type="button"
data-slot="help-collapse-toggle"
className="bsk-help-collapse-toggle"
aria-label={collapsed ? t("helpRequest.expand") : t("helpRequest.collapse")}
aria-expanded={!collapsed}
onClick={() => setCollapsed((c) => !c)}
>
<span className="bsk-help-collapse-icon">
<RiArrowDownSLine size={20} aria-hidden />
</span>
</button>
)}
</div>
<div className="bsk-help-body" aria-hidden={collapsed}>
<div className="bsk-help-body-inner">
<div className="bsk-help-body-content">
<p className="bsk-help-prompt">{request.prompt}</p>
<input
type="text"
className="bsk-help-note-input"
aria-label={t("helpRequest.noteLabel")}
placeholder={t("helpRequest.notePlaceholder")}
value={note}
tabIndex={collapsed ? -1 : 0}
onChange={(e) => setNote(e.target.value)}
/>
<div className="bsk-help-footer-actions">
<button
type="button"
data-slot={collapsed ? undefined : "help-cancel-button"}
className="bsk-help-btn-cancel"
{!isCompact && (
<div className="bsk-help-body" aria-hidden={collapsed}>
<div className="bsk-help-body-inner">
<div className="bsk-help-body-content">
<p className="bsk-help-prompt">{request.prompt}</p>
<input
type="text"
className="bsk-help-note-input"
aria-label={t("helpRequest.noteLabel")}
placeholder={t("helpRequest.notePlaceholder")}
value={note}
tabIndex={collapsed ? -1 : 0}
onClick={() => request.onCancel()}
>
{t("helpRequest.cancel")}
</button>
<button
type="button"
data-slot={collapsed ? undefined : "help-continue-button"}
className="bsk-help-btn-continue"
tabIndex={collapsed ? -1 : 0}
onClick={() => request.onContinue(note)}
>
{t("helpRequest.continue")}
</button>
onChange={(e) => setNote(e.target.value)}
/>
<div className="bsk-help-footer-actions">
<button
type="button"
data-slot={collapsed ? undefined : "help-cancel-button"}
className="bsk-help-btn-cancel"
tabIndex={collapsed ? -1 : 0}
onClick={() => request.onCancel()}
>
{t("helpRequest.cancel")}
</button>
<button
type="button"
data-slot={collapsed ? undefined : "help-continue-button"}
className="bsk-help-btn-continue"
tabIndex={collapsed ? -1 : 0}
onClick={() => request.onContinue(note)}
>
{t("helpRequest.continue")}
</button>
</div>
</div>
</div>
</div>
</div>
)}
</div>
</>
);
@@ -0,0 +1,287 @@
import { useTranslation } from "@browser-skill/i18n/react";
import {
type PointerEvent as ReactPointerEvent,
useCallback,
useEffect,
useRef,
useState,
} from "react";
import recordIconUrl from "../../assets/record-icon.png";
export interface RecordRequestData {
id: string;
/**
* Epoch ms when the recording began. Supplied by the background so the timer
* keeps counting the whole session across navigations; falls back to mount
* time when the background could not report it.
*/
startedAtMs?: number;
onFinish: () => void;
}
type Props = {
request: RecordRequestData | null;
};
const VIEWPORT_MARGIN = 16;
const DRAG_THRESHOLD_PX = 8;
const INTRO_DURATION_MS = 10_000;
const TIMER_REFRESH_MS = 250;
function formatElapsed(seconds: number): string {
const minutes = Math.floor(seconds / 60);
const remainingSeconds = seconds % 60;
return `${String(minutes).padStart(2, "0")}:${String(remainingSeconds).padStart(2, "0")}`;
}
function clamp(value: number, min: number, max: number): number {
return Math.max(min, Math.min(max, value));
}
function clampDragPos(
top: number,
left: number,
panelW: number,
panelH: number,
): { top: number; left: number } {
const maxLeft = window.innerWidth - panelW - VIEWPORT_MARGIN;
const maxTop = window.innerHeight - panelH - VIEWPORT_MARGIN;
return {
top: clamp(top, VIEWPORT_MARGIN, Math.max(VIEWPORT_MARGIN, maxTop)),
left: clamp(left, VIEWPORT_MARGIN, Math.max(VIEWPORT_MARGIN, maxLeft)),
};
}
export function RecordOverlay({ request }: Props) {
const { t } = useTranslation("extension");
const [show, setShow] = useState(false);
const [elapsedSeconds, setElapsedSeconds] = useState(0);
const [dragPos, setDragPos] = useState<{ top: number; left: number } | null>(null);
const [dragging, setDragging] = useState(false);
const pillRef = useRef<HTMLDivElement>(null);
const dragStartRef = useRef<{
pointerX: number;
pointerY: number;
originLeft: number;
originTop: number;
panelW: number;
panelH: number;
armed: boolean;
} | null>(null);
useEffect(() => {
if (request) {
const raf = requestAnimationFrame(() => setShow(true));
return () => cancelAnimationFrame(raf);
}
setShow(false);
}, [request]);
const requestStartedAtMs = request?.startedAtMs;
useEffect(() => {
setDragPos(null);
setDragging(false);
dragStartRef.current = null;
if (!request) {
setElapsedSeconds(0);
return;
}
const startedAt = requestStartedAtMs ?? Date.now();
const updateElapsed = () => {
setElapsedSeconds(Math.max(0, Math.floor((Date.now() - startedAt) / 1_000)));
};
updateElapsed();
const timer = window.setInterval(updateElapsed, TIMER_REFRESH_MS);
return () => window.clearInterval(timer);
}, [request?.id, requestStartedAtMs]);
const onPillPointerDown = useCallback((e: ReactPointerEvent<HTMLDivElement>) => {
if (e.button !== 0) return;
if (e.target instanceof Element && e.target.closest("button, a, input, textarea, select")) {
return;
}
const pill = pillRef.current;
if (!pill) return;
const rect = pill.getBoundingClientRect();
dragStartRef.current = {
pointerX: e.clientX,
pointerY: e.clientY,
originLeft: rect.left,
originTop: rect.top,
panelW: pill.offsetWidth || rect.width,
panelH: pill.offsetHeight || rect.height,
armed: false,
};
}, []);
useEffect(() => {
const onMove = (e: PointerEvent) => {
const start = dragStartRef.current;
if (!start) return;
const dx = e.clientX - start.pointerX;
const dy = e.clientY - start.pointerY;
if (!start.armed) {
if (Math.hypot(dx, dy) < DRAG_THRESHOLD_PX) return;
start.armed = true;
setDragging(true);
setDragPos({ top: start.originTop, left: start.originLeft });
}
setDragPos(
clampDragPos(start.originTop + dy, start.originLeft + dx, start.panelW, start.panelH),
);
};
const onUp = () => {
const start = dragStartRef.current;
if (!start) return;
if (start.armed) {
setDragging(false);
}
dragStartRef.current = null;
};
window.addEventListener("pointermove", onMove);
window.addEventListener("pointerup", onUp);
window.addEventListener("pointercancel", onUp);
return () => {
window.removeEventListener("pointermove", onMove);
window.removeEventListener("pointerup", onUp);
window.removeEventListener("pointercancel", onUp);
};
}, []);
if (!request) return null;
const introVisible = elapsedSeconds * 1_000 < INTRO_DURATION_MS;
const positioned = dragPos !== null;
const defaultTransform = show
? "translateX(-50%) translateY(0)"
: "translateX(-50%) translateY(8px)";
return (
<>
<style>{`
@keyframes bsk-rec-pulse {
0%, 100% { opacity: 1; transform: scale(1); }
50% { opacity: 0.35; transform: scale(0.82); }
}
`}</style>
<div
ref={pillRef}
data-slot="record-overlay-pill"
data-phase={introVisible ? "intro" : "timer"}
data-dragging={dragging ? "true" : "false"}
onPointerDown={onPillPointerDown}
style={{
position: "fixed",
...(positioned
? { top: dragPos.top, left: dragPos.left, bottom: "auto" }
: { bottom: 32, left: "50%" }),
zIndex: 2147483647,
pointerEvents: "auto",
display: "flex",
alignItems: "center",
gap: 5,
backgroundColor: "#fff",
borderRadius: 9999,
height: 50,
boxSizing: "border-box",
padding: "6px 8px 6px 16px",
boxShadow: dragging
? "0 12px 40px rgba(15,23,42,0.22), 0 2px 8px rgba(0,0,0,0.12)"
: "0 8px 32px rgba(15,23,42,0.16), 0 2px 8px rgba(0,0,0,0.1)",
opacity: show ? 1 : 0,
transform: positioned ? "none" : defaultTransform,
transition: dragging
? "box-shadow 150ms ease-out"
: "opacity 300ms ease-out, transform 300ms ease-out, box-shadow 150ms ease-out",
fontFamily:
'-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif',
maxWidth: "min(420px, calc(100vw - 32px))",
cursor: dragging ? "grabbing" : "grab",
userSelect: "none",
touchAction: "none",
}}
>
<div
data-slot="record-overlay-status"
style={{
display: "flex",
alignItems: "center",
gap: 8,
minWidth: 0,
flex: 1,
borderRadius: 9999,
outline: "none",
minHeight: 32,
paddingRight: 4,
}}
>
<span
data-slot="record-overlay-indicator"
style={{
width: 9,
height: 9,
borderRadius: "50%",
backgroundColor: "#ef4444",
flexShrink: 0,
animation: "bsk-rec-pulse 1.4s ease-in-out infinite",
}}
/>
<span
data-slot="record-overlay-label"
style={{
display: "block",
flex: "0 1 auto",
fontSize: 16,
fontWeight: 500,
color: "#333",
whiteSpace: "nowrap",
userSelect: "none",
overflow: "hidden",
textOverflow: "ellipsis",
width: introVisible ? "auto" : 48,
maxWidth: introVisible ? 280 : 48,
fontVariantNumeric: introVisible ? undefined : "tabular-nums",
textAlign: introVisible ? "left" : "center",
margin: 0,
padding: 0,
minWidth: 0,
transition: "max-width 180ms ease-out",
}}
>
{introVisible ? t("recordOverlay.recording") : formatElapsed(elapsedSeconds)}
</span>
</div>
<button
type="button"
data-slot="record-overlay-finish"
aria-label={t("recordOverlay.finish")}
onClick={request.onFinish}
style={{
pointerEvents: "auto",
display: "block",
width: 32,
height: 32,
border: "none",
borderRadius: 9999,
padding: 0,
backgroundColor: "transparent",
cursor: "pointer",
flexShrink: 0,
}}
>
<img
src={recordIconUrl}
alt=""
width={24}
height={24}
draggable={false}
style={{ display: "block", margin: "auto" }}
/>
</button>
</div>
</>
);
}
@@ -12,7 +12,7 @@ describe("BorrowConfirmationOverlay", () => {
vi.useRealTimers();
});
it("auto-allows after countdown reaches zero and progress transition ends", async () => {
it("auto-denies after countdown reaches zero and progress transition ends", async () => {
const onAllow = vi.fn();
const onDeny = vi.fn();
@@ -37,10 +37,7 @@ describe("BorrowConfirmationOverlay", () => {
await vi.advanceTimersByTimeAsync(1000);
});
}
expect(screen.getByText("0 秒后自动允许")).toBeTruthy();
await act(async () => {
await vi.advanceTimersByTimeAsync(1000);
});
expect(screen.getByText("0 秒后自动拒绝")).toBeTruthy();
const progressCircle = container.querySelectorAll("svg circle")[1];
expect(progressCircle).toBeTruthy();
await act(() => {
@@ -54,8 +51,44 @@ describe("BorrowConfirmationOverlay", () => {
await act(async () => {
await vi.advanceTimersByTimeAsync(150);
});
expect(onAllow).toHaveBeenCalledTimes(1);
expect(onDeny).not.toHaveBeenCalled();
expect(onDeny).toHaveBeenCalledTimes(1);
expect(onAllow).not.toHaveBeenCalled();
});
it("auto-denies via timer fallback when progress transitionend never fires", async () => {
const onAllow = vi.fn();
const onDeny = vi.fn();
render(
<BorrowConfirmationOverlay
requests={[
{
id: "req-fallback",
isActiveTab: true,
tabTitle: "Example",
timeoutMs: 5000,
onAllow,
onDeny,
},
]}
/>,
);
for (let i = 0; i < 5; i++) {
await act(async () => {
await vi.advanceTimersByTimeAsync(1000);
});
}
expect(screen.getByText("0 秒后自动拒绝")).toBeTruthy();
// No transitionend — the PROGRESS_TRANSITION_MS fallback must still deny.
await act(async () => {
await vi.advanceTimersByTimeAsync(1000);
});
await act(async () => {
await vi.advanceTimersByTimeAsync(150);
});
expect(onDeny).toHaveBeenCalledTimes(1);
expect(onAllow).not.toHaveBeenCalled();
});
it("only invokes onDeny once when deny is clicked repeatedly", async () => {
@@ -1,5 +1,5 @@
import { cleanup, render } from "@testing-library/react";
import { afterEach, describe, expect, it } from "vitest";
import { afterEach, describe, expect, it, vi } from "vitest";
import { ControlOverlay } from "../ControlOverlay";
describe("ControlOverlay", () => {
@@ -7,7 +7,7 @@ describe("ControlOverlay", () => {
cleanup();
});
it("sets pointer-events none on blocker and pill when automationBypass is true", () => {
it("keeps page blocker none under automationBypass but Interrupt stays clickable", () => {
const { container } = render(
<ControlOverlay
visible={true}
@@ -17,13 +17,17 @@ describe("ControlOverlay", () => {
/>,
);
const blocker = container.querySelector("[data-slot='control-overlay']")?.nextElementSibling;
const blocker = container.querySelector("[data-slot='control-overlay-blocker']");
expect(blocker).toBeTruthy();
expect((blocker as HTMLElement).style.pointerEvents).toBe("none");
const pill = container.querySelector("[data-slot='control-overlay-pill']");
expect(pill).toBeTruthy();
expect((pill as HTMLElement).style.pointerEvents).toBe("auto");
const stopBtn = container.querySelector("[data-slot='control-overlay-stop-all']");
expect(stopBtn).toBeTruthy();
expect((stopBtn as HTMLElement).style.pointerEvents).toBe("none");
expect((stopBtn as HTMLElement).style.pointerEvents).toBe("auto");
});
it("uses pointer-events auto on blocker when automationBypass is false", () => {
@@ -36,8 +40,25 @@ describe("ControlOverlay", () => {
/>,
);
const blocker = container.querySelector("[data-slot='control-overlay']")?.nextElementSibling;
const blocker = container.querySelector("[data-slot='control-overlay-blocker']");
expect(blocker).toBeTruthy();
expect((blocker as HTMLElement).style.pointerEvents).toBe("auto");
});
it("calls onInterrupt from the stop button", () => {
const onInterrupt = vi.fn();
const { container } = render(
<ControlOverlay
visible={true}
interrupting={false}
automationBypass={false}
onInterrupt={onInterrupt}
/>,
);
const stopBtn = container.querySelector("[data-slot='control-overlay-stop-all']");
(stopBtn as HTMLButtonElement).click();
expect(onInterrupt).toHaveBeenCalledTimes(1);
});
});
@@ -3,9 +3,9 @@ import { I18nextProvider } from "@browser-skill/i18n/react";
import { cleanup, fireEvent, render, screen } from "@testing-library/react";
import { createElement } from "react";
import { afterEach, describe, expect, it, vi } from "vitest";
import { HelpRequestOverlay, type HelpRequestData } from "../HelpRequestOverlay";
import { type HelpRequestData, HelpRequestOverlay } from "../HelpRequestOverlay";
function renderOverlay(req: HelpRequestData) {
function renderOverlay(req: HelpRequestData | null) {
return render(
createElement(I18nextProvider, { i18n }, createElement(HelpRequestOverlay, { request: req })),
);
@@ -38,6 +38,45 @@ describe("HelpRequestOverlay", () => {
expect(screen.getByText("Please complete the captcha")).toBeTruthy();
});
it("keeps the inactive render stable", () => {
const { container, rerender } = renderOverlay(null);
rerender(
createElement(
I18nextProvider,
{ i18n },
createElement(HelpRequestOverlay, { request: null }),
),
);
expect(container.querySelector("[data-slot='help-request-banner']")).toBeNull();
});
it("renders a compact status without full request controls", () => {
const el = document.createElement("div");
el.id = "login";
el.getBoundingClientRect = () =>
({ top: 10, left: 10, width: 100, height: 40, right: 110, bottom: 50 }) as DOMRect;
document.body.append(el);
const { container } = renderOverlay(
baseRequest({
displayMode: "compact",
selectors: ["#login"],
}),
);
expect(screen.getByText(i18n.t("helpRequest.compactStatus", { ns: "extension" }))).toBeTruthy();
expect(screen.queryByText("Please complete the captcha")).toBeNull();
expect(screen.queryByRole("textbox")).toBeNull();
expect(screen.queryByLabelText(i18n.t("helpRequest.collapse", { ns: "extension" }))).toBeNull();
expect(container.querySelector("[data-slot='help-continue-button']")).toBeNull();
expect(container.querySelector("[data-slot='help-cancel-button']")).toBeNull();
expect(document.querySelectorAll("[data-slot='help-highlight']").length).toBe(0);
el.remove();
});
it("renders a custom title when provided", () => {
renderOverlay(baseRequest({ title: "Verify your identity" }));
expect(screen.getByText("Verify your identity")).toBeTruthy();
@@ -79,7 +118,7 @@ describe("HelpRequestOverlay", () => {
renderOverlay(baseRequest({ selectors: ["#login", "#missing"] }));
// One highlight box rendered for the matched selector.
expect(document.querySelectorAll("[data-slot='help-highlight']").length).toBe(1);
expect((el.scrollIntoView as ReturnType<typeof vi.fn>)).toHaveBeenCalled();
expect(el.scrollIntoView as ReturnType<typeof vi.fn>).toHaveBeenCalled();
el.remove();
});
@@ -115,8 +154,8 @@ describe("HelpRequestOverlay", () => {
),
);
expect((elA.scrollIntoView as ReturnType<typeof vi.fn>)).toHaveBeenCalled();
expect((elB.scrollIntoView as ReturnType<typeof vi.fn>)).toHaveBeenCalled();
expect(elA.scrollIntoView as ReturnType<typeof vi.fn>).toHaveBeenCalled();
expect(elB.scrollIntoView as ReturnType<typeof vi.fn>).toHaveBeenCalled();
elA.remove();
elB.remove();
});
@@ -204,6 +243,15 @@ describe("HelpRequestOverlay", () => {
expect(styles).toContain("width: 0");
});
it("uses natural body height in the expanded layout", () => {
const { container } = renderOverlay(baseRequest());
const styles = overlayStyles(container);
expect(styles).toMatch(/\.bsk-help-body\s*\{[^}]*display:\s*block;/s);
expect(styles).not.toMatch(/\.bsk-help-body\s*\{[^}]*grid-template-rows/s);
expect(styles).toMatch(/\.bsk-help-banner\s*\{[^}]*justify-content:\s*flex-start;/s);
});
it("keeps the same banner width when collapsed", () => {
const { container } = renderOverlay(baseRequest());
const styles = overlayStyles(container);
@@ -286,9 +334,7 @@ describe("HelpRequestOverlay", () => {
it("does not start drag when pointer down on a button in the header", () => {
const { container } = renderOverlay(baseRequest());
const banner = container.querySelector("[data-slot='help-request-banner']") as HTMLElement;
const collapseBtn = screen.getByLabelText(
i18n.t("helpRequest.collapse", { ns: "extension" }),
);
const collapseBtn = screen.getByLabelText(i18n.t("helpRequest.collapse", { ns: "extension" }));
fireEvent.pointerDown(collapseBtn, { button: 0, clientX: 10, clientY: 10, pointerId: 1 });
@@ -0,0 +1,181 @@
import { i18n } from "@browser-skill/i18n";
import { I18nextProvider } from "@browser-skill/i18n/react";
import { act, cleanup, fireEvent, render, screen } from "@testing-library/react";
import { createElement } from "react";
import { afterEach, describe, expect, it, vi } from "vitest";
import { RecordOverlay } from "../RecordOverlay";
function renderOverlay(request: { id: string; startedAtMs?: number; onFinish: () => void } | null) {
return render(
createElement(I18nextProvider, { i18n }, createElement(RecordOverlay, { request })),
);
}
function advanceRecording(ms: number) {
act(() => {
vi.advanceTimersByTime(ms);
});
}
describe("RecordOverlay", () => {
afterEach(() => {
cleanup();
vi.useRealTimers();
});
it("renders the bottom pill with the icon-only finish button", () => {
const onFinish = vi.fn();
const { container } = renderOverlay({ id: "rec-1", onFinish });
const pill = container.querySelector("[data-slot='record-overlay-pill']");
expect(pill).toBeTruthy();
expect((pill as HTMLElement).style.borderRadius).toBe("9999px");
expect((pill as HTMLElement).style.backgroundColor).toBe("#fff");
expect((pill as HTMLElement).style.gap).toBe("5px");
expect((pill as HTMLElement).style.padding).toBe("6px 8px 6px 16px");
expect((pill as HTMLElement).style.height).toBe("50px");
expect((pill as HTMLElement).style.boxSizing).toBe("border-box");
expect(container.querySelector("[data-slot='record-overlay-collapse']")).toBeNull();
const status = container.querySelector("[data-slot='record-overlay-status']") as HTMLElement;
expect(status.style.minHeight).toBe("32px");
const finish = screen.getByRole("button", {
name: i18n.t("recordOverlay.finish", { ns: "extension" }),
});
expect(finish).toBeTruthy();
expect(finish.textContent).toBe("");
const icon = finish.querySelector("img");
expect(icon).toBeTruthy();
expect(icon?.getAttribute("src")).toContain("record-icon.png");
expect((finish as HTMLElement).style.width).toBe("32px");
expect((finish as HTMLElement).style.height).toBe("32px");
expect(icon?.getAttribute("width")).toBe("24");
expect(icon?.getAttribute("height")).toBe("24");
expect((finish as HTMLElement).style.borderRadius).toBe("9999px");
fireEvent.click(finish);
expect(onFinish).toHaveBeenCalledTimes(1);
});
it("shows a pulsing recording indicator and no full-screen glow layer", () => {
const { container } = renderOverlay({ id: "rec-1", onFinish: vi.fn() });
expect(container.querySelector("[data-slot='record-overlay']")).toBeNull();
const indicator = container.querySelector("[data-slot='record-overlay-indicator']");
expect(indicator).toBeTruthy();
expect((indicator as HTMLElement).style.animation).toContain("bsk-rec-pulse");
});
it("shows the recording message for ten seconds, then displays elapsed time", () => {
vi.useFakeTimers();
const { container } = renderOverlay({ id: "rec-1", onFinish: vi.fn() });
const pill = container.querySelector("[data-slot='record-overlay-pill']") as HTMLElement;
const label = container.querySelector("[data-slot='record-overlay-label']") as HTMLElement;
expect(pill.dataset.phase).toBe("intro");
expect(label.textContent).toBe(i18n.t("recordOverlay.recording", { ns: "extension" }));
advanceRecording(9999);
expect(pill.dataset.phase).toBe("intro");
advanceRecording(1);
expect(pill.dataset.phase).toBe("timer");
expect(label.textContent).toBe("00:10");
});
it("formats elapsed time as zero-padded minutes and seconds", () => {
vi.useFakeTimers();
const { container } = renderOverlay({ id: "rec-1", onFinish: vi.fn() });
const label = container.querySelector("[data-slot='record-overlay-label']") as HTMLElement;
advanceRecording(125_000);
expect(label.textContent).toBe("02:05");
});
it("keeps a fixed timer label width with tabular numerals", () => {
vi.useFakeTimers();
const { container } = renderOverlay({ id: "rec-1", onFinish: vi.fn() });
const label = container.querySelector("[data-slot='record-overlay-label']") as HTMLElement;
advanceRecording(10_000);
expect(label.style.width).toBe("48px");
expect(label.style.maxWidth).toBe("48px");
expect(label.style.fontVariantNumeric).toBe("tabular-nums");
expect(label.style.textAlign).toBe("center");
advanceRecording(1_000);
expect(label.textContent).toBe("00:11");
expect(label.style.width).toBe("48px");
});
it("resets the intro and elapsed time for a new request", () => {
vi.useFakeTimers();
const first = { id: "rec-1", onFinish: vi.fn() };
const second = { id: "rec-2", onFinish: vi.fn() };
const { container, rerender } = renderOverlay(first);
advanceRecording(65_000);
rerender(
createElement(I18nextProvider, { i18n }, createElement(RecordOverlay, { request: second })),
);
const pill = container.querySelector("[data-slot='record-overlay-pill']") as HTMLElement;
const label = container.querySelector("[data-slot='record-overlay-label']") as HTMLElement;
expect(pill.dataset.phase).toBe("intro");
expect(label.textContent).toBe(i18n.t("recordOverlay.recording", { ns: "extension" }));
advanceRecording(10_000);
expect(pill.dataset.phase).toBe("timer");
expect(label.textContent).toBe("00:10");
});
it("continues the session timer when the overlay remounts on a new page", () => {
vi.useFakeTimers();
const startedAtMs = Date.now() - 65_000;
const { container } = renderOverlay({ id: "rec-1", startedAtMs, onFinish: vi.fn() });
const pill = container.querySelector("[data-slot='record-overlay-pill']") as HTMLElement;
const label = container.querySelector("[data-slot='record-overlay-label']") as HTMLElement;
expect(pill.dataset.phase).toBe("timer");
expect(label.textContent).toBe("01:05");
advanceRecording(5_000);
expect(label.textContent).toBe("01:10");
});
it("does not start drag when pointer down on finish", () => {
const { container } = renderOverlay({ id: "rec-1", onFinish: vi.fn() });
const pill = container.querySelector("[data-slot='record-overlay-pill']") as HTMLElement;
const finish = container.querySelector("[data-slot='record-overlay-finish']") as HTMLElement;
fireEvent.pointerDown(finish, { button: 0, clientX: 10, clientY: 10, pointerId: 1 });
expect(pill.getAttribute("data-dragging")).toBe("false");
});
it("moves the pill when dragging the non-button area without toggling collapse", () => {
const { container } = renderOverlay({ id: "rec-1", onFinish: vi.fn() });
const pill = container.querySelector("[data-slot='record-overlay-pill']") as HTMLElement;
pill.getBoundingClientRect = () =>
({
top: 600,
left: 200,
width: 320,
height: 48,
right: 520,
bottom: 648,
}) as DOMRect;
Object.defineProperty(pill, "offsetWidth", { value: 320, configurable: true });
Object.defineProperty(pill, "offsetHeight", { value: 48, configurable: true });
fireEvent.pointerDown(pill, { button: 0, clientX: 300, clientY: 620, pointerId: 1 });
fireEvent(window, new PointerEvent("pointermove", { clientX: 360, clientY: 580 }));
fireEvent(window, new PointerEvent("pointerup", { pointerId: 1 }));
expect(pill.getAttribute("data-phase")).toBe("intro");
expect(pill.style.left).toBe("260px");
expect(pill.style.top).toBe("560px");
expect(pill.style.bottom).toBe("auto");
expect(pill.style.transform).toBe("none");
});
});
@@ -0,0 +1,125 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { CAPTURE_SUPPRESS, type CaptureSuppressAck } from "@/lib/capture-suppress-bridge";
import { CAPTURE_HIDDEN_ATTR, createCaptureSuppressController } from "../capture-suppress";
/**
* rAF is stubbed with a manual queue so tests can drive the compositor
* frame-by-frame and assert the ack only lands after the second frame.
*/
describe("capture-suppress controller", () => {
let rafQueue: FrameRequestCallback[];
let host: HTMLElement;
beforeEach(() => {
rafQueue = [];
vi.stubGlobal("requestAnimationFrame", (cb: FrameRequestCallback) => {
rafQueue.push(cb);
return rafQueue.length;
});
vi.stubGlobal("cancelAnimationFrame", vi.fn());
document.body.innerHTML = "";
host = document.createElement("div");
document.body.append(host);
});
/** Run every queued rAF callback, draining microtasks between rounds. */
async function flushFrames(rounds = 4): Promise<void> {
for (let round = 0; round < rounds && rafQueue.length > 0; round += 1) {
const callbacks = rafQueue.splice(0);
for (const cb of callbacks) cb(0);
await Promise.resolve();
await Promise.resolve();
}
}
it("begin hides the host immediately but only acks after two frames", async () => {
const controller = createCaptureSuppressController(() => host);
const sendResponse = vi.fn();
const needsAsync = controller.handleMessage(
{ type: CAPTURE_SUPPRESS, phase: "begin" },
sendResponse,
);
expect(needsAsync).toBe(true);
expect(controller.pendingCount).toBe(1);
expect(host.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(true);
expect(sendResponse).not.toHaveBeenCalled();
// First frame commits the style change; the ack must still be pending.
await flushFrames(1);
expect(sendResponse).not.toHaveBeenCalled();
// Second frame: the compositor has produced an overlay-free frame.
await flushFrames();
const expected: CaptureSuppressAck = { type: CAPTURE_SUPPRESS, ok: true };
expect(sendResponse).toHaveBeenCalledWith(expected);
});
it("end removes the attribute once the count returns to zero", async () => {
const controller = createCaptureSuppressController(() => host);
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "begin" }, vi.fn());
await flushFrames();
expect(host.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(true);
const sendResponse = vi.fn();
const needsAsync = controller.handleMessage(
{ type: CAPTURE_SUPPRESS, phase: "end" },
sendResponse,
);
expect(needsAsync).toBe(false);
expect(controller.pendingCount).toBe(0);
expect(host.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(false);
expect(sendResponse).toHaveBeenCalledWith({ type: CAPTURE_SUPPRESS, ok: true });
});
it("reference-counts concurrent begins so one end does not unhide early", async () => {
const controller = createCaptureSuppressController(() => host);
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "begin" }, vi.fn());
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "begin" }, vi.fn());
await flushFrames();
expect(controller.pendingCount).toBe(2);
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "end" }, vi.fn());
expect(controller.pendingCount).toBe(1);
expect(host.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(true);
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "end" }, vi.fn());
expect(controller.pendingCount).toBe(0);
expect(host.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(false);
});
it("treats a stray end as a no-op", () => {
const controller = createCaptureSuppressController(() => host);
const sendResponse = vi.fn();
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "end" }, sendResponse);
expect(controller.pendingCount).toBe(0);
expect(host.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(false);
expect(sendResponse).toHaveBeenCalledWith({ type: CAPTURE_SUPPRESS, ok: true });
});
it("re-applies the attribute when the host is remounted mid-capture", () => {
let currentHost: HTMLElement | null = host;
const controller = createCaptureSuppressController(() => currentHost);
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "begin" }, vi.fn());
// Host lost and rebuilt before `end` arrives.
currentHost = null;
const rebuilt = document.createElement("div");
controller.onHostMounted(rebuilt);
expect(rebuilt.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(true);
});
it("leaves a remounted host visible once suppression has ended", () => {
const controller = createCaptureSuppressController(() => host);
const rebuilt = document.createElement("div");
controller.onHostMounted(rebuilt);
expect(rebuilt.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(false);
});
});
@@ -1,5 +1,5 @@
import { describe, expect, it, vi } from "vitest";
import { OverlayController } from "../overlay-controller";
import { OverlayController, shouldShowAgentControlOverlay } from "../overlay-controller";
describe("OverlayController", () => {
it("resets agent overlays without clearing user-tab borrow requests", () => {
@@ -31,6 +31,7 @@ describe("OverlayController", () => {
expect(state.controlVisible).toBe(false);
expect(state.activeSessionId).toBeNull();
expect(state.activeHelp).toBeNull();
expect(state.activeRecord).toBeNull();
expect(state.automationBypassCount).toBe(0);
});
@@ -69,4 +70,64 @@ describe("OverlayController", () => {
expect(previous?.id).toBe("help-1");
expect(controller.snapshot().activeHelp?.id).toBe("help-2");
});
it("tracks active record request state", () => {
const controller = new OverlayController();
controller.activateAgentSession("sess-1");
controller.setAgentRecordRequest({
id: "rec-1",
onFinish: vi.fn(),
});
expect(controller.snapshot().activeRecord?.id).toBe("rec-1");
});
it("hides agent control overlay while recording is active", () => {
const controller = new OverlayController();
controller.activateAgentSession("sess-1");
expect(shouldShowAgentControlOverlay(controller.snapshot())).toBe(true);
controller.setAgentRecordRequest({
id: "rec-1",
onFinish: vi.fn(),
});
expect(shouldShowAgentControlOverlay(controller.snapshot())).toBe(false);
expect(controller.snapshot().controlVisible).toBe(true);
});
it("keeps control hidden after record ends until session overlays reset", () => {
const controller = new OverlayController();
controller.activateAgentSession("sess-1");
controller.setAgentRecordRequest({
id: "rec-1",
onFinish: vi.fn(),
});
controller.setAutomationBypass(true);
controller.setAutomationBypass(true);
controller.clearAgentRecordRequest("rec-1");
expect(controller.snapshot().activeRecord).toBeNull();
expect(controller.snapshot().suppressControlAfterRecord).toBe(true);
expect(controller.snapshot().automationBypassCount).toBe(0);
expect(shouldShowAgentControlOverlay(controller.snapshot())).toBe(false);
controller.resetAgentOverlays("sess-1");
expect(controller.snapshot().suppressControlAfterRecord).toBe(false);
expect(controller.snapshot().controlVisible).toBe(false);
});
it("hides control while paused and shows it again only after authoritative control mode", () => {
const controller = new OverlayController();
controller.applyAgentControlMode("sess-1", "paused");
expect(controller.snapshot().activeSessionId).toBe("sess-1");
expect(controller.snapshot().controlVisible).toBe(false);
expect(shouldShowAgentControlOverlay(controller.snapshot())).toBe(false);
controller.applyAgentControlMode("sess-1", "control");
expect(controller.snapshot().activeSessionId).toBe("sess-1");
expect(controller.snapshot().controlVisible).toBe(true);
expect(shouldShowAgentControlOverlay(controller.snapshot())).toBe(true);
});
});
@@ -0,0 +1,159 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { RECORD_STOP, type RecordStepPayload } from "@/lib/record-bridge";
import { handleRecordContentMessage, startRecordCapture } from "../record-capture";
vi.stubGlobal("chrome", {
runtime: {
sendMessage: vi.fn(() => Promise.resolve()),
},
});
describe("handleRecordContentMessage stop/cancel", () => {
it("ignores STOP when no recording is active", () => {
const dispose = vi.fn();
const onStop = vi.fn();
const setActiveRequestId = vi.fn();
const setCapture = vi.fn();
const sendResponse = vi.fn();
const needsAsync = handleRecordContentMessage(
{ type: RECORD_STOP, requestId: "rec-stale" },
{
activeRequestId: null,
capture: { dispose },
setActiveRequestId,
setCapture,
onStart: vi.fn(),
onStop,
},
sendResponse,
);
expect(needsAsync).toBe(false);
expect(dispose).not.toHaveBeenCalled();
expect(onStop).not.toHaveBeenCalled();
expect(setActiveRequestId).not.toHaveBeenCalled();
expect(setCapture).not.toHaveBeenCalled();
expect(sendResponse).not.toHaveBeenCalled();
});
it("ignores STOP for a mismatched requestId", () => {
const dispose = vi.fn();
const onStop = vi.fn();
const needsAsync = handleRecordContentMessage(
{ type: RECORD_STOP, requestId: "rec-other" },
{
activeRequestId: "rec-1",
capture: { dispose },
setActiveRequestId: vi.fn(),
setCapture: vi.fn(),
onStart: vi.fn(),
onStop,
},
vi.fn(),
);
expect(needsAsync).toBe(false);
expect(dispose).not.toHaveBeenCalled();
expect(onStop).not.toHaveBeenCalled();
});
});
describe("record-capture semantic", () => {
let steps: RecordStepPayload[];
beforeEach(() => {
steps = [];
document.body.innerHTML = `
<label for="q">查询</label>
<input id="q" name="q" />
<button type="button" aria-label="搜索">搜索</button>
<div role="listbox" id="sug">
<div role="option">建议项</div>
</div>
`;
});
it("commits final fill value and records semantic click", () => {
const capture = startRecordCapture("rec-1", (step) => steps.push(step));
const input = document.querySelector("input")!;
const button = document.querySelector("button")!;
input.dispatchEvent(new FocusEvent("focusin", { bubbles: true }));
input.value = "hello";
input.dispatchEvent(new Event("input", { bubbles: true }));
input.dispatchEvent(new FocusEvent("focusout", { bubbles: true }));
button.dispatchEvent(new MouseEvent("click", { bubbles: true, button: 0, detail: 1 }));
capture.dispose();
expect(steps).toEqual(
expect.arrayContaining([
expect.objectContaining({
op: "fill",
value: "hello",
target: expect.objectContaining({ name: "查询", tag: "input" }),
}),
expect.objectContaining({
op: "click",
target: expect.objectContaining({ name: "搜索", role: "button" }),
}),
]),
);
expect(JSON.stringify(steps)).not.toMatch(/@e\d+/);
expect(steps.some((s) => "selector" in s)).toBe(false);
});
it("ignores autocomplete suggestion clicks while a fill session is open", () => {
const capture = startRecordCapture("rec-2", (step) => steps.push(step));
const input = document.querySelector("input")!;
const option = document.querySelector('[role="option"]')!;
input.dispatchEvent(new FocusEvent("focusin", { bubbles: true }));
input.value = "hel";
input.dispatchEvent(new Event("input", { bubbles: true }));
option.dispatchEvent(new MouseEvent("click", { bubbles: true, button: 0, detail: 1 }));
input.value = "hello";
input.dispatchEvent(new Event("input", { bubbles: true }));
input.dispatchEvent(new FocusEvent("focusout", { bubbles: true }));
capture.dispose();
expect(steps.filter((s) => s.op === "click")).toHaveLength(0);
expect(steps.some((s) => s.op === "fill" && s.value === "hello")).toBe(true);
});
it("records Enter press but not bare typing keys", () => {
const capture = startRecordCapture("rec-press", (step) => steps.push(step));
const input = document.querySelector("input")!;
input.dispatchEvent(new FocusEvent("focusin", { bubbles: true }));
input.dispatchEvent(new KeyboardEvent("keydown", { key: "a", bubbles: true }));
input.dispatchEvent(new KeyboardEvent("keydown", { key: "Enter", bubbles: true }));
capture.dispose();
expect(steps.filter((s) => s.op === "press")).toEqual([
expect.objectContaining({ op: "press", key: "Enter" }),
]);
});
it("does not record clicks on anonymous layout divs", () => {
document.body.innerHTML = `
<div id="chrome">page chrome</div>
<button type="button" aria-label="下一步">下一步</button>
`;
const capture = startRecordCapture("rec-3", (step) => steps.push(step));
document
.querySelector("#chrome")!
.dispatchEvent(new MouseEvent("click", { bubbles: true, button: 0, detail: 1 }));
document
.querySelector("button")!
.dispatchEvent(new MouseEvent("click", { bubbles: true, button: 0, detail: 1 }));
capture.dispose();
expect(steps).toHaveLength(1);
expect(steps[0]).toMatchObject({
op: "click",
target: { name: "下一步", role: "button" },
});
});
});
@@ -0,0 +1,99 @@
/**
* Content-script side of the capture-suppress bridge: hides the overlay
* shadow host while the background captures a screenshot.
*
* `begin` sets `data-bsk-capture-hidden` on the host (overlay.css maps it
* to `display: none !important`, which removes the whole shadow subtree
* from compositing) and only acks after two animation frames — the first
* frame commits the style change, the second guarantees the compositor
* has produced a frame without the overlay, so the capture API cannot
* pick up a stale frame that still contains it. `end` is
* reference-counted so overlapping captures do not unhide the host early.
*/
import {
CAPTURE_SUPPRESS,
type CaptureSuppressAck,
type CaptureSuppressMessage,
} from "@/lib/capture-suppress-bridge";
export const CAPTURE_HIDDEN_ATTR = "data-bsk-capture-hidden";
/**
* Safety net for hidden/background tabs where rAF callbacks never fire:
* the attribute is already set synchronously, so any frame the compositor
* produces from now on excludes the overlay. Waiting longer would just
* deadlock the capture.
*/
const FRAME_FALLBACK_MS = 500;
function nextFrame(): Promise<void> {
return new Promise((resolve) => {
const raf = requestAnimationFrame(() => {
clearTimeout(timer);
resolve();
});
const timer = setTimeout(() => {
cancelAnimationFrame(raf);
resolve();
}, FRAME_FALLBACK_MS);
});
}
export interface CaptureSuppressController {
/**
* Handle one bridge message. Returns true when the ack is sent
* asynchronously and the caller must keep the message channel open.
*/
handleMessage(
message: CaptureSuppressMessage,
sendResponse: (ack: CaptureSuppressAck) => void,
): boolean;
/** Re-apply the hidden attribute when the overlay host is (re)mounted. */
onHostMounted(host: HTMLElement): void;
/** Number of in-flight `begin` phases (exposed for tests). */
readonly pendingCount: number;
}
export function createCaptureSuppressController(
getHost: () => HTMLElement | null,
): CaptureSuppressController {
let pending = 0;
const ack: CaptureSuppressAck = { type: CAPTURE_SUPPRESS, ok: true };
function setHidden(host: HTMLElement | null, hidden: boolean): void {
if (!host) return;
if (hidden) host.setAttribute(CAPTURE_HIDDEN_ATTR, "");
else host.removeAttribute(CAPTURE_HIDDEN_ATTR);
}
return {
get pendingCount() {
return pending;
},
handleMessage(message, sendResponse) {
if (message.phase === "begin") {
pending += 1;
setHidden(getHost(), true);
void (async () => {
await nextFrame();
await nextFrame();
sendResponse(ack);
})();
return true;
}
// `end` without a matching `begin` (e.g. a retried message) must not
// drive the counter negative or clear a concurrent capture's flag.
pending = Math.max(0, pending - 1);
if (pending === 0) setHidden(getHost(), false);
sendResponse(ack);
return false;
},
onHostMounted(host) {
// Host lost + rebuilt mid-capture: the fresh host must stay hidden
// until the last `end` arrives.
setHidden(host, pending > 0);
},
};
}
@@ -1,13 +1,23 @@
import type { OverlayMode } from "@/lib/overlay-bridge";
import type { BorrowRequestData } from "./BorrowConfirmationOverlay";
import type { HelpRequestData } from "./HelpRequestOverlay";
import type { RecordRequestData } from "./RecordOverlay";
export interface OverlayState {
borrowRequests: BorrowRequestData[];
activeHelp: HelpRequestData | null;
activeRecord: RecordRequestData | null;
controlVisible: boolean;
interrupting: boolean;
activeSessionId: string | null;
controlMode: OverlayMode;
automationBypassCount: number;
/**
* After record Finish/Stop clears `activeRecord`, the session is still
* alive until CLI `session.stop`. Suppress the control mask for that gap
* so 「Agent 正在控制」does not flash between RecordOverlay and teardown.
*/
suppressControlAfterRecord: boolean;
}
type MutableOverlayState = Omit<OverlayState, "controlVisible">;
@@ -20,9 +30,12 @@ export class OverlayController {
private state: MutableOverlayState = {
borrowRequests: [],
activeHelp: null,
activeRecord: null,
interrupting: false,
activeSessionId: null,
controlMode: "hidden",
automationBypassCount: 0,
suppressControlAfterRecord: false,
};
snapshot(): OverlayState {
@@ -34,7 +47,7 @@ export class OverlayController {
}
isControlVisible(): boolean {
return this.state.activeSessionId !== null;
return this.state.activeSessionId !== null && this.state.controlMode === "control";
}
addBorrowRequest(request: BorrowRequestData): void {
@@ -51,6 +64,25 @@ export class OverlayController {
activateAgentSession(sessionId: string): void {
this.state.activeSessionId = sessionId;
this.state.interrupting = false;
this.state.controlMode = "control";
}
applyAgentControlMode(sessionId: string | null, mode: OverlayMode): void {
if (mode === "control" && sessionId) {
this.state.activeSessionId = sessionId;
this.state.interrupting = false;
this.state.controlMode = "control";
return;
}
if ((mode === "interrupting" || mode === "paused") && sessionId) {
this.state.activeSessionId = sessionId;
this.state.interrupting = mode === "interrupting";
this.state.controlMode = mode;
return;
}
this.state.activeSessionId = null;
this.state.interrupting = false;
this.state.controlMode = "hidden";
}
setInterrupting(interrupting: boolean): void {
@@ -70,6 +102,25 @@ export class OverlayController {
this.state.activeHelp = null;
}
setAgentRecordRequest(request: RecordRequestData): RecordRequestData | null {
const previous = this.state.activeRecord;
this.state.activeRecord = request;
this.state.suppressControlAfterRecord = false;
return previous;
}
clearAgentRecordRequest(requestId?: string): void {
if (!this.state.activeRecord) return;
if (requestId && this.state.activeRecord.id !== requestId) return;
this.state.activeRecord = null;
// Hide control until session teardown — see OverlayState.suppressControlAfterRecord.
this.state.suppressControlAfterRecord = true;
// Record start/rearm may have stacked automation-bypass refs; drop them so a
// stray ControlOverlay cannot sit with pointer-events:none (page usable,
// Interrupt unclickable).
this.state.automationBypassCount = 0;
}
setAutomationBypass(enabled: boolean): void {
if (enabled) {
this.state.automationBypassCount += 1;
@@ -86,10 +137,23 @@ export class OverlayController {
this.state = {
...this.state,
activeHelp: null,
activeRecord: null,
interrupting: false,
activeSessionId: null,
controlMode: "hidden",
automationBypassCount: 0,
suppressControlAfterRecord: false,
};
return previousHelp;
}
}
/** Control mask ("Agent 正在控制") must hide while help/record overlays own the chrome. */
export function shouldShowAgentControlOverlay(state: OverlayState): boolean {
return (
state.controlVisible &&
!state.suppressControlAfterRecord &&
state.activeHelp === null &&
state.activeRecord === null
);
}
+21
View File
@@ -1,2 +1,23 @@
@import "tailwindcss";
@source "../content";
:host([data-bsk-overlay-surface]) {
position: fixed !important;
inset: 0 !important;
display: block !important;
width: 100vw !important;
height: 100vh !important;
z-index: 2147483647 !important;
pointer-events: none !important;
}
:host([data-bsk-overlay-surface][data-bsk-overlay-blocking]) {
pointer-events: auto !important;
}
/* Screenshot suppression: the host lives in the main DOM, so hiding it
removes the whole shadow subtree from compositing and captured frames
only contain page content. */
:host([data-bsk-capture-hidden]) {
display: none !important;
}
@@ -0,0 +1,537 @@
import { describeEventTarget, describeTarget, type TargetDescriptor } from "@/lib/describe-target";
import {
isRecordCancelMessage,
isRecordStartMessage,
isRecordStopMessage,
RECORD_CANCEL,
RECORD_START,
RECORD_STEP,
RECORD_STOP,
type RecordCancelMessage,
type RecordStartAck,
type RecordStartMessage,
type RecordStepPayload,
type RecordStopAck,
type RecordStopMessage,
} from "@/lib/record-bridge";
import { shouldRecordPress } from "@/lib/trace-reducer";
const pendingStepSends = new Map<string, Set<Promise<boolean>>>();
const failedStepDeliveries = new Set<string>();
const knownRecordRequests = new Set<string>();
function sendRecordStep(requestId: string, step: RecordStepPayload): void {
const payload = { type: RECORD_STEP, requestId, step };
const pending = Promise.resolve(chrome.runtime.sendMessage(payload)).then(
() => true,
() => {
failedStepDeliveries.add(requestId);
return false;
},
);
const sends = pendingStepSends.get(requestId) ?? new Set<Promise<boolean>>();
sends.add(pending);
pendingStepSends.set(requestId, sends);
void pending.then(() => {
sends.delete(pending);
if (sends.size === 0) pendingStepSends.delete(requestId);
});
}
export interface RecordCaptureController {
dispose(): void;
}
interface FillSession {
element: FillableElement;
target: TargetDescriptor;
baselineValue: string;
lastValue: string;
}
type FillableElement = HTMLInputElement | HTMLTextAreaElement | HTMLElement;
function eventTarget(event: Event): EventTarget | null {
return event.composedPath()[0] ?? event.target;
}
function isOverlayTarget(target: EventTarget | null): boolean {
if (!(target instanceof Node)) return false;
const root = document.documentElement;
let node: Node | null = target;
while (node && node !== root) {
if (node instanceof Element && node.hasAttribute("data-bsk-overlay")) {
return true;
}
const rootNode: Node | Document | ShadowRoot = node.getRootNode();
if (rootNode instanceof ShadowRoot) {
const host: Element = rootNode.host;
if (host.hasAttribute("data-bsk-overlay")) {
return true;
}
node = host;
} else {
node = node.parentNode;
}
}
return false;
}
function isTextFillable(el: Element): el is FillableElement {
if (el instanceof HTMLElement && (el.isContentEditable || el.contentEditable === "true")) {
return true;
}
if (el instanceof HTMLTextAreaElement) return true;
if (!(el instanceof HTMLInputElement)) return false;
const type = el.type.toLowerCase();
return type !== "checkbox" && type !== "radio" && type !== "file" && type !== "button";
}
function fillableFromTarget(target: EventTarget | null): FillableElement | null {
if (!(target instanceof Element)) return null;
if (isTextFillable(target)) return target;
const el = target.closest('input,textarea,[contenteditable]:not([contenteditable="false"])');
if (el && isTextFillable(el)) return el;
return null;
}
/** Clicks on search chrome that only focus the nearby input should not become steps. */
function nearbyFillableFromSearchChrome(target: Element): FillableElement | null {
const container = target.closest(
'[id*="chat-input"], [id*="search"], [class*="search"], form, [role="search"]',
);
if (!container) return null;
const fillable = container.querySelector(
'textarea, input[type="search"], input[name="q"], #chat-textarea',
);
if (
fillable instanceof HTMLElement &&
isTextFillable(fillable) &&
fillable !== target &&
!fillable.contains(target)
) {
return fillable;
}
return null;
}
function fillableValue(el: FillableElement): string {
if (el instanceof HTMLInputElement || el instanceof HTMLTextAreaElement) {
return el.value;
}
return el.textContent ?? "";
}
/** Clicks that only pick an autocomplete/suggestion value — not a semantic submit action. */
function isInputCompletionClick(target: EventTarget | null, session: FillSession | null): boolean {
if (!session || !(target instanceof Element)) return false;
if (session.element.contains(target)) return false;
const option = target.closest('[role="option"]');
if (option?.closest('[role="listbox"]')) return true;
const suggestionRoot = target.closest(
[
'[id*="Sug"]',
'[id*="sug"]',
'[class*="suggest"]',
'[class*="autocomplete"]',
'[class*="typeahead"]',
"[data-autocomplete]",
].join(", "),
);
if (suggestionRoot && !suggestionRoot.contains(session.element)) return true;
return false;
}
function scheduleInputCompletionCommit(
sessionElement: FillableElement,
syncFillSessionValue: (el: FillableElement) => void,
commitFillSession: () => void,
hasSessionFor: (el: FillableElement) => boolean,
): void {
const syncAndCommit = () => {
if (!hasSessionFor(sessionElement)) return;
syncFillSessionValue(sessionElement);
commitFillSession();
};
syncFillSessionValue(sessionElement);
queueMicrotask(syncAndCommit);
setTimeout(syncAndCommit, 0);
}
export function startRecordCapture(
_requestId: string,
sendStep: (step: RecordStepPayload) => void,
): RecordCaptureController {
const emitStep = (step: RecordStepPayload) => {
sendStep({ page_url: location.href, ...step });
};
let fillSession: FillSession | null = null;
let composing = false;
let lastUrl = location.href;
let keyboardActivation: { target: EventTarget | null; recordedAt: number } | undefined;
let generatedControlClick: Element | null = null;
let navigationActionPending = false;
let navigationActionVersion = 0;
const committedValues = new WeakMap<FillableElement, string>();
const markNavigationAction = () => {
navigationActionPending = true;
const version = ++navigationActionVersion;
setTimeout(() => {
if (navigationActionVersion === version) {
navigationActionPending = false;
}
}, 0);
};
const emitFill = (session: FillSession) => {
if (session.lastValue === session.baselineValue) return;
const isPassword =
session.element instanceof HTMLInputElement && session.element.type === "password";
const value = isPassword ? "***" : session.lastValue;
emitStep({
op: "fill",
target: session.target,
value,
...(isPassword ? { redacted: true } : {}),
});
};
const commitFillSession = () => {
if (!fillSession || composing) return;
const session = fillSession;
fillSession = null;
emitFill(session);
committedValues.set(session.element, session.lastValue);
};
const syncFillSessionValue = (el: FillableElement) => {
if (!fillSession || fillSession.element !== el) return;
fillSession.lastValue = fillableValue(el);
};
const ensureFillSession = (el: FillableElement) => {
if (fillSession?.element === el) return;
if (fillSession) commitFillSession();
const currentValue = fillableValue(el);
const baselineValue = committedValues.get(el) ?? currentValue;
fillSession = {
element: el,
target: describeTarget(el),
baselineValue,
lastValue: currentValue,
};
};
const emitNavigateIfChanged = (causedByAction?: boolean) => {
if (location.href === lastUrl) return;
commitFillSession();
lastUrl = location.href;
emitStep({
op: "navigate",
url: location.href,
...(causedByAction !== undefined ? { navigation_caused_by_action: causedByAction } : {}),
});
};
const emitClick = (event: MouseEvent) => {
// Only record clicks an LLM can re-identify (named interactive controls).
const target = describeEventTarget(eventTarget(event));
if (!target) return;
markNavigationAction();
emitStep({
op: "click",
target,
expects_navigation: true,
});
};
const onClick = (event: MouseEvent) => {
if (event.button !== 0) return;
const target = eventTarget(event);
if (isOverlayTarget(target)) return;
if (event.detail === 0 && generatedControlClick !== null && target === generatedControlClick) {
generatedControlClick = null;
return;
}
if (
event.detail === 0 &&
keyboardActivation?.target === target &&
Date.now() - keyboardActivation.recordedAt < 500
) {
keyboardActivation = undefined;
return;
}
const label = target instanceof Element ? target.closest("label") : null;
const nestedInteractive =
target instanceof Element ? target.closest("a,button,input,select,textarea") : null;
if (label instanceof HTMLLabelElement && !nestedInteractive) {
commitFillSession();
generatedControlClick = label.control;
emitClick(event);
return;
}
const fillable = fillableFromTarget(target);
if (fillable) {
ensureFillSession(fillable);
return;
}
if (target instanceof Element) {
const nearbyFillable = nearbyFillableFromSearchChrome(target);
if (nearbyFillable) {
ensureFillSession(nearbyFillable);
return;
}
}
if (isInputCompletionClick(target, fillSession)) {
markNavigationAction();
const sessionElement = fillSession!.element;
scheduleInputCompletionCommit(
sessionElement,
syncFillSessionValue,
commitFillSession,
(el) => fillSession?.element === el,
);
return;
}
commitFillSession();
if (target instanceof Element && target.closest("select")) return;
emitClick(event);
};
const onFocusIn = (event: FocusEvent) => {
const target = fillableFromTarget(eventTarget(event));
if (target) ensureFillSession(target);
};
const onFocusOut = (event: FocusEvent) => {
const target = fillableFromTarget(eventTarget(event));
if (!target) return;
if (!fillSession || fillSession.element !== target) return;
syncFillSessionValue(target);
commitFillSession();
};
const onInput = (event: Event) => {
const target = fillableFromTarget(eventTarget(event));
if (!target) return;
if (composing) return;
ensureFillSession(target);
syncFillSessionValue(target);
};
const onCompositionStart = () => {
composing = true;
};
const onCompositionEnd = (event: CompositionEvent) => {
composing = false;
const target = fillableFromTarget(eventTarget(event));
if (target) {
ensureFillSession(target);
syncFillSessionValue(target);
}
};
const onChange = (event: Event) => {
commitFillSession();
const target = eventTarget(event);
if (target instanceof HTMLSelectElement) {
const values = Array.from(target.selectedOptions).map((opt) => opt.value);
const labels = Array.from(target.selectedOptions).map((opt) =>
(opt.label || opt.textContent || opt.value).trim(),
);
const desc = describeTarget(target);
markNavigationAction();
emitStep({
op: "select",
target: desc,
values,
labels,
expects_navigation: true,
});
}
};
const onKeyDown = (event: KeyboardEvent) => {
if (isOverlayTarget(eventTarget(event))) return;
const target = eventTarget(event);
const fillable = fillableFromTarget(target);
if (fillable) {
ensureFillSession(fillable);
syncFillSessionValue(fillable);
}
const modifiers = [
...(event.altKey ? (["alt"] as const) : []),
...(event.ctrlKey ? (["ctrl"] as const) : []),
...(event.metaKey ? (["meta"] as const) : []),
...(event.shiftKey ? (["shift"] as const) : []),
];
if (!shouldRecordPress(event.key, modifiers)) {
return;
}
markNavigationAction();
if (event.key === "Enter" || event.key === " ") {
const submitTarget =
event.key === "Enter" && fillable
? fillable
.closest("form")
?.querySelector(
'button:not([type]),button[type="submit"],input[type="submit"],input[type="image"]',
)
: null;
keyboardActivation = {
target: submitTarget ?? target,
recordedAt: Date.now(),
};
}
if (fillable) {
commitFillSession();
}
const desc = describeEventTarget(target);
if (!desc && !event.key) return;
emitStep({
op: "press",
key: event.key,
...(desc ? { target: desc } : {}),
...(modifiers.length ? { modifiers } : {}),
expects_navigation: event.key === "Enter",
});
};
document.addEventListener("click", onClick, true);
document.addEventListener("focusin", onFocusIn, true);
document.addEventListener("focusout", onFocusOut, true);
document.addEventListener("input", onInput, true);
document.addEventListener("compositionstart", onCompositionStart, true);
document.addEventListener("compositionend", onCompositionEnd, true);
document.addEventListener("change", onChange, true);
document.addEventListener("keydown", onKeyDown, true);
const urlObserver = new MutationObserver(() => emitNavigateIfChanged());
urlObserver.observe(document, { subtree: true, childList: true });
const onUrlEvent = () => emitNavigateIfChanged();
window.addEventListener("hashchange", onUrlEvent);
window.addEventListener("popstate", onUrlEvent);
window.addEventListener("pagehide", commitFillSession);
const originalPushState = history.pushState;
const originalReplaceState = history.replaceState;
history.pushState = function (...args: Parameters<History["pushState"]>) {
originalPushState.apply(this, args);
emitNavigateIfChanged(navigationActionPending ? true : undefined);
};
history.replaceState = function (...args: Parameters<History["replaceState"]>) {
originalReplaceState.apply(this, args);
emitNavigateIfChanged(navigationActionPending ? true : undefined);
};
return {
dispose() {
commitFillSession();
document.removeEventListener("click", onClick, true);
document.removeEventListener("focusin", onFocusIn, true);
document.removeEventListener("focusout", onFocusOut, true);
document.removeEventListener("input", onInput, true);
document.removeEventListener("compositionstart", onCompositionStart, true);
document.removeEventListener("compositionend", onCompositionEnd, true);
document.removeEventListener("change", onChange, true);
document.removeEventListener("keydown", onKeyDown, true);
urlObserver.disconnect();
window.removeEventListener("hashchange", onUrlEvent);
window.removeEventListener("popstate", onUrlEvent);
window.removeEventListener("pagehide", commitFillSession);
history.pushState = originalPushState;
history.replaceState = originalReplaceState;
},
};
}
export type RecordContentMessage = RecordStartMessage | RecordStopMessage | RecordCancelMessage;
export function isRecordContentMessage(msg: unknown): msg is RecordContentMessage {
return isRecordStartMessage(msg) || isRecordStopMessage(msg) || isRecordCancelMessage(msg);
}
export function handleRecordContentMessage(
message: RecordContentMessage,
state: {
activeRequestId: string | null;
capture: RecordCaptureController | null;
setActiveRequestId(id: string | null): void;
setCapture(capture: RecordCaptureController | null): void;
onStart(requestId: string, startedAtMs?: number): void;
onStop(): void;
},
sendResponse?: (response: RecordStartAck | RecordStopAck) => void,
): boolean {
if (isRecordStartMessage(message)) {
if (!knownRecordRequests.has(message.requestId)) {
knownRecordRequests.add(message.requestId);
failedStepDeliveries.delete(message.requestId);
}
state.capture?.dispose();
state.setCapture(
startRecordCapture(message.requestId, (step) => {
sendRecordStep(message.requestId, step);
}),
);
state.setActiveRequestId(message.requestId);
state.onStart(message.requestId, message.startedAtMs);
sendResponse?.({ ok: true });
return sendResponse !== undefined;
}
if (isRecordStopMessage(message) || isRecordCancelMessage(message)) {
// Require an active recording that matches this requestId — otherwise a
// stray STOP/CANCEL (e.g. after teardown) would still run finishStop and
// clear overlay state even though nothing was capturing.
if (!state.activeRequestId || state.activeRequestId !== message.requestId) {
return false;
}
state.capture?.dispose();
state.setCapture(null);
const finishStop = () => {
state.onStop();
state.setActiveRequestId(null);
};
if (isRecordStopMessage(message) && sendResponse) {
const pending = [...(pendingStepSends.get(message.requestId) ?? [])];
void Promise.all(pending).then((delivered) => {
finishStop();
const succeeded = delivered.every(Boolean) && !failedStepDeliveries.has(message.requestId);
if (succeeded) {
failedStepDeliveries.delete(message.requestId);
knownRecordRequests.delete(message.requestId);
}
sendResponse(
succeeded
? { ok: true }
: {
ok: false,
error: "failed to deliver one or more recorded steps",
},
);
});
return true;
}
if (isRecordCancelMessage(message)) {
failedStepDeliveries.delete(message.requestId);
knownRecordRequests.delete(message.requestId);
}
finishStop();
return false;
}
return false;
}
export { RECORD_CANCEL, RECORD_START, RECORD_STEP, RECORD_STOP };
@@ -1,4 +1,4 @@
import { describe, it, expect, vi } from "vitest";
import { describe, expect, it, vi } from "vitest";
// `background.ts` is a WXT entrypoint that calls the global `defineBackground`
// at module load. WXT injects that helper at build time via auto-imports;
+176 -17
View File
@@ -1,6 +1,7 @@
import { i18n } from "@browser-skill/i18n";
import { ChromiumCdp } from "@/browser-driver/chromium-cdp";
import { ConnectionController } from "@/lib/connection-controller";
import { startHeartbeat } from "@/lib/heartbeat";
import {
getConnectionEnabled,
setConnectionEnabled as persistConnectionEnabled,
@@ -8,11 +9,16 @@ import {
} from "@/lib/instance-id";
import { startKeepalive } from "@/lib/keepalive";
import {
OVERLAY_AGENT_STATE,
OVERLAY_AUTOMATION_BYPASS,
OVERLAY_MSG_INTERRUPT,
OVERLAY_MSG_READY,
OVERLAY_MSG_WHO_AM_I,
type OverlayAgentStateMessage,
type OverlayInterruptRequest,
type OverlayInterruptResponse,
type OverlayMessage,
type OverlayMode,
} from "@/lib/overlay-bridge";
import { POPUP_PORT_NAME, type PopupInbound, type PopupOutbound } from "@/lib/popup-bridge";
import { attachSessionsLiveFlag } from "@/lib/sessions-live-flag";
@@ -27,19 +33,106 @@ import {
requestBorrowConfirmation,
} from "@/tools/borrow-confirmation";
import { ToolDispatcher } from "@/tools/dispatcher";
import {
attachRecordFinishListener,
attachRecordQueryListener,
attachRecordStepListener,
} from "@/tools/record";
import { detectBrowserMeta } from "@/transport/handshake";
import type { Transport } from "@/transport/transport";
import { WSTransport } from "@/transport/ws-transport";
export const DEFAULT_DAEMON_PORT = 52800;
export default defineBackground(() => {
const port = DEFAULT_DAEMON_PORT;
const controller = new ConnectionController();
const transport = new WSTransport({ url: `ws://127.0.0.1:${port}` });
const transport = new WSTransport({ url: __BSK_DAEMON_WS_URL__ });
const sessions = new SessionManager();
const cdp = new ChromiumCdp();
const sessionsLive = attachSessionsLiveFlag({ manager: sessions });
let overlayGeneration = 0;
const controlModes = new Map<string, OverlayMode>();
function setControlMode(sessionId: string, mode: OverlayMode): void {
if (controlModes.get(sessionId) === mode) return;
controlModes.set(sessionId, mode);
overlayGeneration += 1;
const ctx = sessions.get(sessionId);
if (ctx) void pushOverlayStateForWindow(ctx.agentWindowId);
}
function overlayStateForWindow(windowId?: number): OverlayAgentStateMessage {
const ctx = typeof windowId === "number" ? sessions.findByWindowId(windowId) : null;
if (!ctx) {
return {
type: OVERLAY_AGENT_STATE,
sessionId: null,
mode: "hidden",
generation: overlayGeneration,
};
}
return {
type: OVERLAY_AGENT_STATE,
sessionId: ctx.sessionId,
mode: controlModes.get(ctx.sessionId) ?? "control",
generation: overlayGeneration,
};
}
async function pushOverlayStateToTab(
tabId: number,
state: OverlayAgentStateMessage,
): Promise<void> {
try {
await chrome.tabs.sendMessage(tabId, state);
} catch {
// Restricted pages and not-yet-loaded content scripts cannot receive messages.
}
}
async function pushOverlayStateForWindow(windowId: number): Promise<void> {
const state = overlayStateForWindow(windowId);
const tabs = await chrome.tabs.query({ windowId });
await Promise.all(
tabs.map((tab) =>
typeof tab.id === "number" ? pushOverlayStateToTab(tab.id, state) : Promise.resolve(),
),
);
}
function pushAllAgentOverlayStates(): void {
const windowIds = new Set(sessions.list().map((ctx) => ctx.agentWindowId));
for (const windowId of windowIds) {
void pushOverlayStateForWindow(windowId);
}
}
function onOverlaySessionStateChanged(): void {
void sessionsLive.syncFromManager();
const liveSessionIds = new Set(sessions.list().map((ctx) => ctx.sessionId));
for (const sessionId of controlModes.keys()) {
if (!liveSessionIds.has(sessionId)) controlModes.delete(sessionId);
}
overlayGeneration += 1;
pushAllAgentOverlayStates();
}
function onBrowserControlResumed(sessionId: string): void {
const ctx = sessions.get(sessionId);
if (!ctx) return;
setControlMode(sessionId, "control");
}
function pushOverlayStateForAgentWindow(windowId: number): void {
if (!sessions.findByWindowId(windowId)) return;
void pushOverlayStateForWindow(windowId);
}
chrome.tabs.onActivated.addListener((activeInfo) => {
pushOverlayStateForAgentWindow(activeInfo.windowId);
});
chrome.tabs.onUpdated.addListener((_tabId, changeInfo, tab) => {
if (changeInfo.status !== "complete") return;
if (typeof tab.windowId !== "number") return;
pushOverlayStateForAgentWindow(tab.windowId);
});
// Re-sync the storage.session flag on SW startup so a previous SW's
// stale `true` does not keep waking us on every page load until the
// first mutation (review M4/M5 round 3 m-R3-1).
@@ -48,17 +141,15 @@ export default defineBackground(() => {
transport,
sessions,
cdp,
onSessionsChanged: () => {
void sessionsLive.syncFromManager();
},
onSessionsChanged: onOverlaySessionStateChanged,
onBrowserControlResumed,
approveBorrow: (ctx) =>
requestBorrowConfirmation(ctx.tabId, {
...(ctx.signal !== undefined ? { signal: ctx.signal } : {}),
deps: {
// Skip every Agent Window when choosing where to render the
// overlay — Agent Windows boot on about:blank, which has no
// content script, so sendMessage would fail-open and silently
// allow the borrow without any UI shown.
// content script, so they cannot surface an authorization decision.
isAgentWindowId: (windowId) => sessions.findByWindowId(windowId) !== null,
// Resolve i18n strings per-borrow so language switches take effect
// without re-creating the dispatcher.
@@ -71,6 +162,27 @@ export default defineBackground(() => {
}),
});
dispatcher.start();
const recordDeps = {
tabsApi: chrome.tabs,
sendToTab: (tabId: number, msg: Parameters<typeof chrome.tabs.sendMessage>[1]) =>
chrome.tabs.sendMessage(tabId, msg),
bypassOverlay: async (tabId: number, enabled: boolean) => {
try {
await chrome.tabs.sendMessage(tabId, {
type: OVERLAY_AUTOMATION_BYPASS,
enabled,
});
} catch {
// Content script may be unavailable on restricted pages.
}
},
};
// Message listeners stay up (cheap; fire only for record message types).
// Tab / webNavigation observation attaches lazily while a recording is
// active — see ensureBrowserObservationListeners in tools/record.ts.
attachRecordStepListener();
attachRecordFinishListener(recordDeps);
attachRecordQueryListener(recordDeps);
if (typeof chrome.notifications?.onClicked?.addListener === "function") {
attachBorrowNotificationClickHandler({
onClicked: chrome.notifications.onClicked,
@@ -85,8 +197,8 @@ export default defineBackground(() => {
// The Allow / Deny buttons on the OS notification are the *explicit*
// authorization fallback when every candidate user window's content
// script was missing (extension just reloaded, page in BFCache, etc.).
// Without this listener those button clicks would land nowhere and we'd
// be back to fail-open-after-sendMessage-failure.
// Without this listener those button clicks would land nowhere and the
// request would only resolve via the fail-closed background timeout.
if (typeof chrome.notifications?.onButtonClicked?.addListener === "function") {
attachBorrowNotificationButtonHandler({
onButtonClicked: chrome.notifications.onButtonClicked,
@@ -100,9 +212,7 @@ export default defineBackground(() => {
manager: sessions,
transport,
cdp,
onSessionsChanged: () => {
void sessionsLive.syncFromManager();
},
onSessionsChanged: onOverlaySessionStateChanged,
});
// MV3 service worker keepalive + reconnect supervisor (review M4/M5
@@ -116,6 +226,45 @@ export default defineBackground(() => {
shouldConnect: () => controller.isConnectionEnabled,
});
// Application-level heartbeat (Chrome 116+): while the post-handshake
// link is live, beat every 20s so WebSocket activity keeps the service
// worker — and thus the daemon connection — alive during use, rather
// than depending on the boundary-hugging 30s keepalive alarm. The
// daemon also uses these beats to reap a silently-dead browser.
startHeartbeat({
send: (frame) => transport.send(frame),
onActiveChange: (cb) =>
controller.subscribe((snap) =>
cb(snap.state === "connected" || snap.state === "version_skew"),
),
});
// Wake-driven reconnect (best effort). An MV3 service worker is killed
// across OS sleep regardless of any keepalive, and the setTimeout-based
// transport backoff dies with it. These Chrome lifecycle events revive
// the worker and let us reconnect immediately instead of waiting for
// the next 30s alarm tick. They only help when the daemon is actually
// running; a cold daemon is (re)spawned by the next `bsk` command.
const reconnectIfNeeded = () => {
if (!controller.isConnectionEnabled) return;
if (transport.state === "connected") return;
void transport.connect().catch((err) => {
console.debug("[browser-skill] wake reconnect attempt failed", err);
});
};
if (typeof chrome.runtime?.onStartup?.addListener === "function") {
chrome.runtime.onStartup.addListener(reconnectIfNeeded);
}
if (typeof chrome.idle?.onStateChanged?.addListener === "function") {
chrome.idle.onStateChanged.addListener((state) => {
// "active" fires when the user returns from idle/locked — the most
// reliable "machine just woke" signal we get.
if (state === "active") reconnectIfNeeded();
});
// Treat >60s of no input as idle so the active transition is timely.
chrome.idle.setDetectionInterval?.(60);
}
void (async () => {
const connectionEnabled = await getConnectionEnabled();
await controller.attach(transport, detectBrowserMeta(), connectionEnabled);
@@ -134,9 +283,19 @@ export default defineBackground(() => {
return false;
}
if (msg.kind === OVERLAY_MSG_READY) {
sendResponse(overlayStateForWindow(sender.tab?.windowId));
return false;
}
if (msg.kind === OVERLAY_MSG_INTERRUPT) {
const req = msg as OverlayInterruptRequest;
const ctx = sessions.get(req.sessionId);
if (ctx) setControlMode(req.sessionId, "interrupting");
void handleOverlayInterrupt(transport, req.sessionId).then((reply) => {
if (reply.ok && sessions.get(req.sessionId)) {
setControlMode(req.sessionId, "paused");
}
sendResponse(reply);
});
return true; // keep channel open
@@ -167,9 +326,9 @@ export default defineBackground(() => {
// silently doing nothing (review M4/M5 C2).
console.warn("[browser-skill] set_port is not wired yet; ignoring", msg.value);
} else if (msg.kind === "set_connection_enabled") {
void controller.setConnectionEnabled(msg.value).then(() =>
persistConnectionEnabled(msg.value),
);
void controller
.setConnectionEnabled(msg.value)
.then(() => persistConnectionEnabled(msg.value));
}
}
});
+330 -100
View File
@@ -1,35 +1,59 @@
import { i18n } from "@browser-skill/i18n";
import { I18nextProvider } from "@browser-skill/i18n/react";
import React from "react";
import { flushSync } from "react-dom";
import ReactDOM from "react-dom/client";
import { BorrowConfirmationOverlay } from "@/content/BorrowConfirmationOverlay";
import { ControlOverlay } from "@/content/ControlOverlay";
import { createCaptureSuppressController } from "@/content/capture-suppress";
import { HelpRequestOverlay } from "@/content/HelpRequestOverlay";
import overlayCss from "@/content/overlay.css?inline";
import { OverlayController } from "@/content/overlay-controller";
import { OverlayController, shouldShowAgentControlOverlay } from "@/content/overlay-controller";
import { RecordOverlay } from "@/content/RecordOverlay";
import {
HELP_RESPONSE,
handleRecordContentMessage,
isRecordContentMessage,
type RecordCaptureController,
} from "@/content/record-capture";
import {
type CaptureSuppressAck,
type CaptureSuppressMessage,
isCaptureSuppressMessage,
} from "@/lib/capture-suppress-bridge";
import {
HELP_ACK,
HELP_FINISH,
HELP_QUERY,
type HelpAckMessage,
type HelpCancelMessage,
type HelpFinishMessage,
type HelpQueryResponse,
type HelpRequestMessage,
type HelpResponseMessage,
isHelpCancelMessage,
isHelpRequestMessage,
} from "@/lib/help-bridge";
import {
OVERLAY_AUTOMATION_BYPASS,
OVERLAY_MSG_WHO_AM_I,
type OverlayAgentOverlayResetMessage,
type OverlayAutomationBypassMessage,
type OverlayWhoAmIResponse,
isOverlayAgentOverlayResetMessage,
isOverlayAgentStateMessage,
OVERLAY_AUTOMATION_BYPASS,
OVERLAY_MSG_READY,
type OverlayAgentOverlayResetMessage,
type OverlayAgentStateMessage,
type OverlayAutomationBypassMessage,
} from "@/lib/overlay-bridge";
import { sendInterrupt } from "@/lib/overlay-interrupt-client";
import { SESSIONS_LIVE_FLAG_KEY } from "@/lib/sessions-live-flag";
import {
RECORD_FINISH,
RECORD_QUERY,
type RecordQueryResponse,
type RecordStartAck,
type RecordStopAck,
} from "@/lib/record-bridge";
import type {
BorrowCancelMessage,
BorrowRequestMessage,
BorrowResponseMessage,
} from "@/tools/borrow-confirmation";
import { i18n } from "@browser-skill/i18n";
import { I18nextProvider } from "@browser-skill/i18n/react";
import React from "react";
import ReactDOM from "react-dom/client";
// Run at document_end so the overlay does not block first paint. Only attach
// in the top-level frame so iframes do not double-render overlays.
@@ -43,13 +67,17 @@ export default defineContentScript({
if (window.top !== window) return;
const overlays = new OverlayController();
let activeHelpRespond: ((outcome: "continued" | "cancelled", note?: string) => void) | null =
null;
let recordCapture: RecordCaptureController | null = null;
let activeRecordRequestId: string | null = null;
let reactRoot: ReactDOM.Root | null = null;
let overlayHost: HTMLElement | null = null;
let overlayContainer: HTMLElement | null = null;
let activeAgentState: OverlayAgentStateMessage | null = null;
let hostLossReported = false;
let remountInProgress = false;
const captureSuppress = createCaptureSuppressController(() => overlayHost);
const ui = await createShadowRootUi(ctx, {
name: "browser-skill-overlay",
position: "inline",
@@ -59,52 +87,141 @@ export default defineContentScript({
shadowHost.setAttribute("aria-hidden", "true");
shadowHost.setAttribute("data-bsk-overlay", "");
overlayHost = shadowHost;
overlayContainer = container;
hostLossReported = false;
// A host rebuilt mid-capture must stay hidden until `end` arrives.
captureSuppress.onHostMounted(shadowHost);
const app = document.createElement("div");
app.className = "bsk-overlay-root";
container.append(app);
reactRoot = ReactDOM.createRoot(app);
renderOverlay();
renderAll();
void requestOverlayState();
return reactRoot;
},
onRemove(root) {
overlayHost = null;
overlayContainer = null;
root?.unmount();
reactRoot = null;
},
});
function renderOverlay() {
function setOverlaySurfaceState(active: boolean, blocking: boolean): void {
const host = overlayHost;
const container = overlayContainer;
if (!host || !container) return;
if (active) {
host.setAttribute("data-bsk-overlay-surface", "");
if (blocking) {
host.setAttribute("data-bsk-overlay-blocking", "");
} else {
host.removeAttribute("data-bsk-overlay-blocking");
}
Object.assign(container.style, {
position: "fixed",
inset: "0",
pointerEvents: "none",
});
return;
}
host.removeAttribute("data-bsk-overlay-surface");
host.removeAttribute("data-bsk-overlay-blocking");
container.style.removeProperty("position");
container.style.removeProperty("inset");
container.style.removeProperty("pointer-events");
}
function setOverlayHostHiddenFromAccessibility(hidden: boolean): void {
const host = overlayHost;
if (!host) return;
if (!hidden) {
host.removeAttribute("aria-hidden");
return;
}
const focusedInShadow = host.shadowRoot?.activeElement;
if (focusedInShadow instanceof HTMLElement) {
focusedInShadow.blur();
}
if (document.activeElement === host) {
host.blur();
}
host.setAttribute("aria-hidden", "true");
}
function renderReactOverlays(): void {
const overlayState = overlays.snapshot();
reactRoot?.render(
React.createElement(
I18nextProvider,
{ i18n },
React.createElement(
React.Fragment,
null,
React.createElement(BorrowConfirmationOverlay, {
requests: overlayState.borrowRequests,
}),
React.createElement(ControlOverlay, {
visible: overlayState.controlVisible && overlayState.activeHelp === null,
interrupting: overlayState.interrupting,
automationBypass: overlayState.automationBypassCount > 0,
onInterrupt: handleInterrupt,
}),
React.createElement(HelpRequestOverlay, { request: overlayState.activeHelp }),
),
),
const controlOverlayVisible = shouldShowAgentControlOverlay(overlayState);
const interactiveOverlayVisible =
overlayState.borrowRequests.length > 0 ||
overlayState.activeHelp !== null ||
overlayState.activeRecord !== null;
setOverlayHostHiddenFromAccessibility(!interactiveOverlayVisible);
setOverlaySurfaceState(
controlOverlayVisible,
controlOverlayVisible && overlayState.automationBypassCount === 0,
);
const root = reactRoot;
if (!root) return;
flushSync(() => {
root.render(
React.createElement(
I18nextProvider,
{ i18n },
React.createElement(
React.Fragment,
null,
React.createElement(BorrowConfirmationOverlay, {
requests: overlayState.borrowRequests,
}),
React.createElement(HelpRequestOverlay, { request: overlayState.activeHelp }),
React.createElement(RecordOverlay, { request: overlayState.activeRecord }),
React.createElement(ControlOverlay, {
visible: controlOverlayVisible,
interrupting: overlayState.interrupting,
automationBypass: overlayState.automationBypassCount > 0,
onInterrupt: handleInterrupt,
}),
),
),
);
});
}
function renderAll(): void {
renderReactOverlays();
}
async function waitForRenderedOverlayUpdate(): Promise<void> {
await new Promise<void>((resolve) => window.requestAnimationFrame(() => resolve()));
await new Promise<void>((resolve) => window.requestAnimationFrame(() => resolve()));
}
function clearCurrentAgentSession(): void {
const sessionId = overlays.snapshot().activeSessionId;
if (!sessionId) return;
resetAgentOverlayState(sessionId);
}
function applyOverlayState(state: OverlayAgentStateMessage): void {
activeAgentState = state;
overlays.applyAgentControlMode(state.sessionId, state.mode);
renderAll();
}
function resetAgentOverlayState(sessionId: string) {
const previousHelp = overlays.resetAgentOverlays(sessionId);
if (previousHelp) {
activeHelpRespond?.("cancelled");
activeHelpRespond = null;
void sendHelpFinish(previousHelp.id, "cancelled");
}
renderOverlay();
recordCapture?.dispose();
recordCapture = null;
activeRecordRequestId = null;
renderAll();
}
function handleInterrupt() {
@@ -116,15 +233,8 @@ export default defineContentScript({
return;
}
overlays.setInterrupting(true);
renderOverlay();
void sendInterrupt(
(msg) => chrome.runtime.sendMessage(msg),
sessionId,
).then((reply) => {
// Always retract the mask after the round trip resolves
// (success, failure, or timeout). Cancellation is fire-and-
// forget on the daemon side; the user must not be stuck
// behind a transient issue. The Agent Window stays open.
renderAll();
void sendInterrupt((msg) => chrome.runtime.sendMessage(msg), sessionId).then((reply) => {
resetAgentOverlayState(sessionId);
if (!reply.ok) {
console.warn("[bsk overlay] interrupt did not get a clean ack from daemon");
@@ -138,11 +248,54 @@ export default defineContentScript({
| BorrowCancelMessage
| HelpRequestMessage
| HelpCancelMessage
| CaptureSuppressMessage
| OverlayAgentOverlayResetMessage
| OverlayAgentStateMessage
| OverlayAutomationBypassMessage,
_sender: chrome.runtime.MessageSender,
sendResponse: (response: BorrowResponseMessage | HelpResponseMessage) => void,
sendResponse: (response: BorrowResponseMessage | HelpAckMessage | CaptureSuppressAck) => void,
) => {
if (isCaptureSuppressMessage(message)) {
return captureSuppress.handleMessage(message, sendResponse);
}
if (isRecordContentMessage(message)) {
const needsAsync = handleRecordContentMessage(
message,
{
activeRequestId: activeRecordRequestId,
capture: recordCapture,
setActiveRequestId: (id) => {
activeRecordRequestId = id;
},
setCapture: (capture) => {
recordCapture = capture;
},
onStart: (requestId, startedAtMs) => {
overlays.setAgentRecordRequest({
id: requestId,
...(typeof startedAtMs === "number" ? { startedAtMs } : {}),
onFinish: () => {
void chrome.runtime.sendMessage({
type: RECORD_FINISH,
requestId,
});
},
});
renderAll();
},
onStop: () => {
overlays.clearAgentRecordRequest(activeRecordRequestId ?? undefined);
renderAll();
},
},
sendResponse as unknown as
| ((response: RecordStartAck | RecordStopAck) => void)
| undefined,
);
return needsAsync;
}
if (
message &&
typeof message === "object" &&
@@ -151,7 +304,12 @@ export default defineContentScript({
) {
const bypassMsg = message as OverlayAutomationBypassMessage;
overlays.setAutomationBypass(bypassMsg.enabled);
renderOverlay();
renderAll();
return false;
}
if (isOverlayAgentStateMessage(message)) {
applyOverlayState(message);
return false;
}
@@ -162,48 +320,37 @@ export default defineContentScript({
if (message.type === "borrow-cancel") {
overlays.removeBorrowRequest(message.requestId);
renderOverlay();
renderAll();
return false;
}
if (isHelpCancelMessage(message)) {
const state = overlays.snapshot();
if (state.activeHelp && state.activeHelp.id === message.requestId) {
activeHelpRespond?.("cancelled");
overlays.clearAgentHelpRequest(message.requestId);
renderAll();
}
return false;
}
if (isHelpRequestMessage(message)) {
const helpMsg = message as HelpRequestMessage;
let responded = false;
const respond = (outcome: "continued" | "cancelled", note?: string) => {
if (responded) return;
responded = true;
const reply: HelpResponseMessage = {
type: HELP_RESPONSE,
outcome,
...(note ? { note } : {}),
};
sendResponse(reply);
activeHelpRespond = null;
overlays.clearAgentHelpRequest(helpMsg.requestId);
renderOverlay();
};
const previousHelp = overlays.setAgentHelpRequest({
id: helpMsg.requestId,
prompt: helpMsg.prompt,
...(helpMsg.title ? { title: helpMsg.title } : {}),
...(helpMsg.displayMode ? { displayMode: helpMsg.displayMode } : {}),
selectors: helpMsg.selectors,
onContinue: (note: string) => respond("continued", note.trim() ? note : undefined),
onCancel: () => respond("cancelled"),
onContinue: (note: string) =>
void sendHelpFinish(helpMsg.requestId, "continued", note.trim() ? note : undefined),
onCancel: () => void sendHelpFinish(helpMsg.requestId, "cancelled"),
});
if (previousHelp) {
activeHelpRespond?.("cancelled");
if (previousHelp && previousHelp.id !== helpMsg.requestId) {
void sendHelpFinish(previousHelp.id, "cancelled");
}
activeHelpRespond = respond;
renderOverlay();
return true; // async sendResponse
renderAll();
sendResponse({ type: HELP_ACK, ok: true });
return false;
}
if (message.type === "borrow-request") {
@@ -213,7 +360,7 @@ export default defineContentScript({
responded = true;
sendResponse({ type: "borrow-response", allowed });
overlays.removeBorrowRequest(message.requestId);
renderOverlay();
renderAll();
};
overlays.addBorrowRequest({
@@ -224,30 +371,119 @@ export default defineContentScript({
onAllow: () => respond(true),
onDeny: () => respond(false),
});
renderOverlay();
renderAll();
return true;
}
return false;
};
async function mountOverlayIfAgent(): Promise<void> {
if (!(await anySessionLive())) return;
try {
const reply = (await chrome.runtime.sendMessage({
kind: OVERLAY_MSG_WHO_AM_I,
})) as OverlayWhoAmIResponse | undefined;
if (!reply?.sessionId) return;
overlays.activateAgentSession(reply.sessionId);
renderOverlay();
} catch (err) {
console.debug("[bsk overlay] who_am_i failed", err);
async function sendHelpFinish(
requestId: string,
outcome: "continued" | "cancelled",
note?: string,
): Promise<void> {
const msg: HelpFinishMessage = {
type: HELP_FINISH,
requestId,
outcome,
...(note ? { note } : {}),
};
overlays.clearAgentHelpRequest(requestId);
renderAll();
await waitForRenderedOverlayUpdate();
await chrome.runtime.sendMessage(msg).catch((err) => {
console.debug("[bsk overlay] help finish failed", err);
});
}
function mountHelpRequest(helpMsg: Omit<HelpRequestMessage, "type">): void {
overlays.setAgentHelpRequest({
id: helpMsg.requestId,
prompt: helpMsg.prompt,
...(helpMsg.title ? { title: helpMsg.title } : {}),
...(helpMsg.displayMode ? { displayMode: helpMsg.displayMode } : {}),
selectors: helpMsg.selectors,
onContinue: (note: string) =>
void sendHelpFinish(helpMsg.requestId, "continued", note.trim() ? note : undefined),
onCancel: () => void sendHelpFinish(helpMsg.requestId, "cancelled"),
});
}
async function queryActiveHelpWithRetry(): Promise<void> {
for (let attempt = 0; attempt < 6; attempt += 1) {
try {
const helpQuery = (await chrome.runtime.sendMessage({
type: HELP_QUERY,
})) as HelpQueryResponse | undefined;
if (helpQuery?.active && helpQuery.request) {
mountHelpRequest(helpQuery.request);
renderAll();
return;
}
} catch (err) {
console.debug("[bsk overlay] help query failed", err);
}
await new Promise((resolve) => window.setTimeout(resolve, 150));
}
}
async function queryActiveRecord(): Promise<void> {
try {
const recordQuery = (await chrome.runtime.sendMessage({
type: RECORD_QUERY,
})) as RecordQueryResponse | undefined;
if (
recordQuery?.active &&
typeof recordQuery.requestId === "string" &&
overlays.snapshot().activeRecord === null
) {
const requestId = recordQuery.requestId;
const startedAtMs = recordQuery.startedAtMs;
overlays.setAgentRecordRequest({
id: requestId,
...(typeof startedAtMs === "number" ? { startedAtMs } : {}),
onFinish: () => {
void chrome.runtime.sendMessage({
type: RECORD_FINISH,
requestId,
});
},
});
renderAll();
}
} catch (err) {
console.debug("[bsk overlay] record query failed", err);
}
}
async function refreshAuxiliaryOverlayState(): Promise<void> {
await Promise.all([queryActiveHelpWithRetry(), queryActiveRecord()]);
}
async function requestOverlayState(): Promise<void> {
try {
const state = (await chrome.runtime.sendMessage({
kind: OVERLAY_MSG_READY,
})) as OverlayAgentStateMessage | undefined;
if (state && isOverlayAgentStateMessage(state)) {
applyOverlayState(state);
}
void refreshAuxiliaryOverlayState();
} catch (err) {
console.debug("[bsk overlay] overlay.ready failed", err);
}
}
const onPageShow = (event: PageTransitionEvent) => {
if (event.persisted) void requestOverlayState();
};
ui.mount();
chrome.runtime.onMessage.addListener(onMessage);
void mountOverlayIfAgent();
void requestOverlayState();
window.addEventListener("pageshow", onPageShow);
const hostObserver = new MutationObserver(() => {
const connected = overlayHost?.isConnected ?? false;
@@ -257,6 +493,8 @@ export default defineContentScript({
remountInProgress = true;
try {
ui.mount();
if (activeAgentState) applyOverlayState(activeAgentState);
void requestOverlayState();
} finally {
remountInProgress = false;
}
@@ -271,19 +509,11 @@ export default defineContentScript({
ctx.onInvalidated(() => {
hostObserver.disconnect();
chrome.runtime.onMessage.removeListener(onMessage);
window.removeEventListener("pageshow", onPageShow);
// Restore history hooks / remove capture listeners before the CS unloads.
recordCapture?.dispose();
recordCapture = null;
activeRecordRequestId = null;
});
},
});
async function anySessionLive(): Promise<boolean> {
if (!chrome.storage?.session?.get) return true;
try {
const result = (await chrome.storage.session.get({
[SESSIONS_LIVE_FLAG_KEY]: false,
})) as Record<string, unknown> | undefined;
return Boolean(result?.[SESSIONS_LIVE_FLAG_KEY]);
} catch (err) {
console.debug("[bsk overlay] sessions-live flag read failed", err);
return true;
}
}
@@ -24,6 +24,11 @@ const baseSnapshot: SnapshotInfo = {
connectionEnabled: true,
};
function openRecordView() {
fireEvent.click(screen.getByRole("button", { name: "快捷功能" }));
fireEvent.click(screen.getByRole("button", { name: /操作录制/ }));
}
describe("App", () => {
const setLabel = vi.fn();
const setConnectionEnabled = vi.fn();
@@ -63,13 +68,35 @@ describe("App", () => {
expect(screen.queryByText("请先打开 BrowserSkill。")).toBeNull();
});
it("uses flex gap for label field spacing", () => {
const { container } = render(<App />);
it("does not render record UI on the main view", () => {
render(<App />);
const labelField = container.querySelector("[data-slot='popup-label-field']");
expect(labelField?.className).toContain("flex");
expect(labelField?.className).toContain("flex-col");
expect(labelField?.className).toContain("gap-3");
expect(screen.queryByRole("button", { name: "复制录制指令" })).toBeNull();
expect(screen.queryByRole("button", { name: "操作录制" })).toBeNull();
});
it("opens the feature list from the launcher and navigates to record", () => {
render(<App />);
fireEvent.click(screen.getByRole("button", { name: "快捷功能" }));
expect(screen.getByText("操作录制")).toBeTruthy();
expect(screen.getByText("录制你的操作,供 Agent 参考")).toBeTruthy();
expect(screen.queryByText("未连接")).toBeNull();
fireEvent.click(screen.getByRole("button", { name: /操作录制/ }));
expect(screen.getByRole("button", { name: "复制录制指令" })).toBeTruthy();
});
it("returns from the feature list to the main view", () => {
render(<App />);
fireEvent.click(screen.getByRole("button", { name: "快捷功能" }));
fireEvent.click(screen.getByRole("button", { name: "返回" }));
expect(screen.getByText("未连接")).toBeTruthy();
expect(screen.queryByText("录制你的操作,供 Agent 参考")).toBeNull();
});
it("shows single-line compact metadata and copies the instance id", async () => {
@@ -132,8 +159,88 @@ describe("App", () => {
render(<App />);
expect(screen.getByText("连接已关闭")).toBeTruthy();
expect(screen.getByRole("switch", { name: "BrowserSkill 连接" }).getAttribute("aria-checked")).toBe(
"false",
);
expect(
screen.getByRole("switch", { name: "BrowserSkill 连接" }).getAttribute("aria-checked"),
).toBe("false");
});
it("copies the record prompt with instance id and --browser when connected", async () => {
mockUseConnectionState.mockReturnValue({
snapshot: {
...baseSnapshot,
state: "connected",
instanceId: "03c3e47f",
handshake: {
server: "bh",
version: mockDaemonVersion,
protocol_version: "1.0",
},
},
statusState: "connected",
setLabel,
setConnectionEnabled,
});
render(<App />);
openRecordView();
fireEvent.change(screen.getByPlaceholderText("例如:发布一篇文章"), {
target: { value: "发布 wiki 文档" },
});
const copyButton = screen.getByRole("button", { name: "复制录制指令" });
expect(copyButton.getAttribute("disabled")).toBeNull();
fireEvent.click(copyButton);
expect(navigator.clipboard.writeText).toHaveBeenCalledTimes(1);
const copied = vi.mocked(navigator.clipboard.writeText).mock.calls[0]?.[0] ?? "";
expect(copied).toContain("03c3e47f");
expect(copied).toContain("--browser 03c3e47f");
expect(copied).toContain('--purpose "发布 wiki 文档"');
expect(copied).not.toMatch(/bsk record start[^\n]*--url/);
// Button label stays static; a transient toast confirms the copy.
expect(copyButton.textContent).toContain("复制录制指令");
await waitFor(() => expect(screen.getByRole("status").textContent).toContain("已复制"));
});
it("includes --url in record prompt when a start URL is provided", async () => {
mockUseConnectionState.mockReturnValue({
snapshot: {
...baseSnapshot,
state: "connected",
instanceId: "03c3e47f",
handshake: {
server: "bh",
version: mockDaemonVersion,
protocol_version: "1.0",
},
},
statusState: "connected",
setLabel,
setConnectionEnabled,
});
render(<App />);
openRecordView();
fireEvent.change(screen.getByPlaceholderText("https://…"), {
target: { value: "https://example.com/" },
});
fireEvent.click(screen.getByRole("button", { name: "复制录制指令" }));
const copied = vi.mocked(navigator.clipboard.writeText).mock.calls[0]?.[0] ?? "";
expect(copied).toContain("--url https://example.com/");
expect(copied).not.toContain("--purpose");
});
it("disables record copy when disconnected", () => {
render(<App />);
openRecordView();
const copyButton = screen.getByRole("button", { name: "复制录制指令" });
expect(copyButton.getAttribute("disabled")).not.toBeNull();
expect(screen.getByText("连接后可用")).toBeTruthy();
});
});
+312 -134
View File
@@ -1,9 +1,11 @@
import { useTranslation } from "@browser-skill/i18n/react";
import { Badge, Button, Input, Label, cn } from "@browser-skill/ui";
import { RiCheckLine, RiFileCopyLine } from "@remixicon/react";
import { Badge, Button, cn, Input, Label } from "@browser-skill/ui";
import { RiArrowLeftLine, RiArrowRightSLine, RiCheckLine, RiFileCopyLine } from "@remixicon/react";
import { type ChangeEvent, useEffect, useState } from "react";
import { PROTOCOL_VERSION } from "@/transport/handshake";
import functionIconUrl from "../../../assets/function.svg";
import { ConnectionStatusIndicator } from "./connection-status-indicator";
import { POPUP_FEATURES, type PopupView } from "./features";
import { type PopupStatusState, useConnectionState } from "./use-connection-state";
const STATE_LABEL_KEYS = {
@@ -29,9 +31,15 @@ function getLogoSrc() {
export function App() {
const { t } = useTranslation("extension");
const { snapshot, statusState, setLabel, setConnectionEnabled } = useConnectionState();
const [labelDraft, setLabelDraft] = useState(snapshot.label);
const { snapshot, statusState, setConnectionEnabled } = useConnectionState();
const [view, setView] = useState<PopupView>("main");
const [copiedInstanceId, setCopiedInstanceId] = useState(false);
const [purposeDraft, setPurposeDraft] = useState("");
const [startUrlDraft, setStartUrlDraft] = useState("");
// Bumped on every successful copy so the "copied" toast re-shows (and its
// auto-hide timer restarts) even when the copied content is unchanged.
const [copiedTick, setCopiedTick] = useState(0);
const showCopiedToast = copiedTick > 0;
useEffect(() => {
const query = window.matchMedia("(prefers-color-scheme: dark)");
@@ -44,17 +52,22 @@ export function App() {
return () => query.removeEventListener("change", onChange);
}, []);
useEffect(() => {
setLabelDraft(snapshot.label);
}, [snapshot.label]);
useEffect(() => {
setCopiedInstanceId(false);
setCopiedTick(0);
}, [snapshot.instanceId]);
const onLabelBlur = () => {
if (labelDraft !== snapshot.label) setLabel(labelDraft);
};
// Hide the copied toast when the command changes (topic / start URL edits).
useEffect(() => {
setCopiedTick(0);
}, [purposeDraft, startUrlDraft]);
// Auto-hide the copied toast shortly after it appears.
useEffect(() => {
if (copiedTick === 0) return;
const timer = window.setTimeout(() => setCopiedTick(0), 1500);
return () => window.clearTimeout(timer);
}, [copiedTick]);
const isSkewed = statusState === "version_skew";
const daemonVersion = snapshot.handshake?.version ?? "—";
@@ -68,142 +81,307 @@ export function App() {
setCopiedInstanceId(true);
};
const recordReady = statusState === "connected" && Boolean(snapshot.instanceId);
const recordPurpose = purposeDraft.trim();
const recordStartUrl = startUrlDraft.trim();
const recordCommand = snapshot.instanceId
? [
`bsk record start --browser ${snapshot.instanceId}`,
recordStartUrl ? `--url ${recordStartUrl}` : "",
recordPurpose ? `--purpose "${recordPurpose}"` : "",
]
.filter(Boolean)
.join(" ")
: "";
const recordPrompt = snapshot.instanceId
? t("popup.record.promptTemplate", {
command: recordCommand,
})
: "";
const copyRecordPrompt = async () => {
if (!recordReady || !navigator.clipboard?.writeText) return;
await navigator.clipboard.writeText(recordPrompt);
setCopiedTick((tick) => tick + 1);
};
const headerTitle =
view === "features"
? t("popup.launcher.title")
: view === "record"
? t("popup.record.sectionTitle")
: t("popup.brandName");
return (
<main
className="min-w-[320px] max-w-[340px] space-y-3 bg-background p-3 text-foreground"
data-slot="popup-root"
data-view={view}
data-version-skew={isSkewed ? "true" : undefined}
>
<header className="flex items-center gap-2" data-slot="popup-brand-header">
<img
src={getLogoSrc()}
alt=""
className="size-7 rounded-lg"
data-slot="popup-brand-logo"
/>
<h1 className="text-sm font-semibold tracking-tight">{t("popup.brandName")}</h1>
</header>
<section
className="rounded-xl border border-border/80 bg-card/60 px-3 py-2.5"
data-slot="popup-connection-card"
<header
className="flex items-center gap-2"
data-slot={view === "main" ? "popup-brand-header" : "popup-subview-header"}
>
<div className="flex items-center justify-between gap-2" data-slot="popup-status">
<div className="flex min-w-0 items-center gap-2">
<ConnectionStatusIndicator state={statusState} />
<span
className="truncate text-sm font-medium"
data-slot="popup-state-label"
>
{t(STATE_LABEL_KEYS[statusState])}
</span>
</div>
<div className="flex shrink-0 items-center gap-2">
<Badge
variant="outline"
className="px-1.5 py-0 text-[10px] font-medium uppercase"
data-slot="popup-state-badge"
>
{t(STATE_BADGE_KEYS[statusState])}
</Badge>
<button
type="button"
role="switch"
aria-checked={snapshot.connectionEnabled}
aria-label={t("popup.connectionToggleTitle")}
data-slot="popup-connection-toggle"
className={cn(
"relative inline-flex h-5 w-9 shrink-0 cursor-pointer items-center rounded-full border border-transparent transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring/50",
snapshot.connectionEnabled ? "bg-primary" : "bg-muted",
)}
onClick={() => setConnectionEnabled(!snapshot.connectionEnabled)}
>
<span
className={cn(
"pointer-events-none block size-4 rounded-full bg-background shadow-sm transition-transform",
snapshot.connectionEnabled ? "translate-x-4" : "translate-x-0.5",
)}
aria-hidden
/>
</button>
</div>
</div>
{isSkewed && (
<p
className="mt-2 text-xs leading-snug text-amber-600 dark:text-amber-400"
data-slot="popup-version-skew-warning"
>
{t("popup.versionSkewWarning", {
extensionProtocol: PROTOCOL_VERSION,
daemonProtocol,
})}
</p>
)}
</section>
{snapshot.lastError && (
<div
className="rounded-lg border border-destructive/25 bg-destructive/10 px-3 py-2 text-xs leading-snug text-destructive"
data-slot="popup-error"
>
{snapshot.lastError}
</div>
)}
<div className="flex flex-col gap-3" data-slot="popup-label-field">
<Label htmlFor="bh-label" className="block text-xs text-muted-foreground">
{t("popup.labelTitle")}
</Label>
<Input
id="bh-label"
type="text"
value={labelDraft}
onChange={(event: ChangeEvent<HTMLInputElement>) => setLabelDraft(event.target.value)}
onBlur={onLabelBlur}
placeholder={t("popup.labelPlaceholder")}
className="mt-0 h-8 text-sm"
data-slot="popup-label-input"
/>
</div>
<section
className="flex min-w-0 items-center justify-between gap-2 border-t border-border/70 pt-2 text-[10px] leading-tight text-muted-foreground"
data-slot="popup-meta"
>
<div className="flex shrink-0 items-center gap-1">
<span title={t("popup.extensionVersionHint")}>{extensionVersion}</span>
<span aria-hidden>/</span>
<span title={t("popup.daemonVersionHint")}>{daemonVersion}</span>
</div>
<div className="flex min-w-0 items-center justify-end gap-1">
<span className="shrink-0">{t("popup.instanceTitle")}</span>
<code
className="max-w-[88px] truncate font-mono text-[10px] text-foreground/80"
data-slot="popup-instance-id"
>
{instanceId}
</code>
{view !== "main" ? (
<Button
type="button"
variant="ghost"
size="icon"
className="size-5 rounded-md"
disabled={!snapshot.instanceId}
aria-label={t("popup.copyInstanceId")}
title={copiedInstanceId ? t("popup.copied") : t("popup.copyInstanceId")}
onClick={() => {
void copyInstanceId();
}}
data-slot="popup-copy-instance-id"
className="size-7 shrink-0 rounded-md"
aria-label={t("popup.back")}
onClick={() => setView(view === "record" ? "features" : "main")}
data-slot="popup-back"
>
{copiedInstanceId ? (
<RiCheckLine className="size-3" aria-hidden />
) : (
<RiFileCopyLine className="size-3" aria-hidden />
)}
<RiArrowLeftLine className="size-4" aria-hidden />
</Button>
</div>
</section>
) : (
<img
src={getLogoSrc()}
alt=""
className="size-7 rounded-lg"
data-slot="popup-brand-logo"
/>
)}
<h1 className="min-w-0 flex-1 truncate text-sm font-semibold tracking-tight">
{headerTitle}
</h1>
{view === "main" && (
<Button
type="button"
variant="ghost"
size="icon"
className="group size-8 shrink-0 rounded-md"
aria-label={t("popup.launcher.title")}
onClick={() => setView("features")}
data-slot="popup-launcher"
>
<img
src={functionIconUrl}
alt=""
className="size-6 opacity-35 transition-opacity group-hover:opacity-100 group-focus-visible:opacity-100"
data-slot="popup-launcher-icon"
/>
</Button>
)}
</header>
{view === "main" && (
<>
<section
className="rounded-xl border border-border/80 bg-card/60 px-3 py-2.5"
data-slot="popup-connection-card"
>
<div className="flex items-center justify-between gap-2" data-slot="popup-status">
<div className="flex min-w-0 items-center gap-2">
<ConnectionStatusIndicator state={statusState} />
<span className="truncate text-sm font-medium" data-slot="popup-state-label">
{t(STATE_LABEL_KEYS[statusState])}
</span>
</div>
<div className="flex shrink-0 items-center gap-2">
<Badge
variant="outline"
className="px-1.5 py-0 text-[10px] font-medium uppercase"
data-slot="popup-state-badge"
>
{t(STATE_BADGE_KEYS[statusState])}
</Badge>
<button
type="button"
role="switch"
aria-checked={snapshot.connectionEnabled}
aria-label={t("popup.connectionToggleTitle")}
data-slot="popup-connection-toggle"
className={cn(
"relative inline-flex h-5 w-9 shrink-0 cursor-pointer items-center rounded-full border border-transparent transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring/50",
snapshot.connectionEnabled ? "bg-primary" : "bg-muted",
)}
onClick={() => setConnectionEnabled(!snapshot.connectionEnabled)}
>
<span
className={cn(
"pointer-events-none block size-4 rounded-full bg-background shadow-sm transition-transform",
snapshot.connectionEnabled ? "translate-x-4" : "translate-x-0.5",
)}
aria-hidden
/>
</button>
</div>
</div>
{isSkewed && (
<p
className="mt-2 text-xs leading-snug text-amber-600 dark:text-amber-400"
data-slot="popup-version-skew-warning"
>
{t("popup.versionSkewWarning", {
extensionProtocol: PROTOCOL_VERSION,
daemonProtocol,
})}
</p>
)}
</section>
{snapshot.lastError && (
<div
className="rounded-lg border border-destructive/25 bg-destructive/10 px-3 py-2 text-xs leading-snug text-destructive"
data-slot="popup-error"
>
{snapshot.lastError}
</div>
)}
<section
className="flex min-w-0 items-center justify-between gap-2 border-t border-border/70 pt-2 text-[10px] leading-tight text-muted-foreground"
data-slot="popup-meta"
>
<div className="flex shrink-0 items-center gap-1">
<span title={t("popup.extensionVersionHint")}>{extensionVersion}</span>
<span aria-hidden>/</span>
<span title={t("popup.daemonVersionHint")}>{daemonVersion}</span>
</div>
<div className="flex min-w-0 items-center justify-end gap-1">
<span className="shrink-0">{t("popup.instanceTitle")}</span>
<code
className="max-w-[88px] truncate font-mono text-[10px] text-foreground/80"
data-slot="popup-instance-id"
>
{instanceId}
</code>
<Button
type="button"
variant="ghost"
size="icon"
className="size-5 rounded-md"
disabled={!snapshot.instanceId}
aria-label={t("popup.copyInstanceId")}
title={copiedInstanceId ? t("popup.copied") : t("popup.copyInstanceId")}
onClick={() => {
void copyInstanceId();
}}
data-slot="popup-copy-instance-id"
>
{copiedInstanceId ? (
<RiCheckLine className="size-3" aria-hidden />
) : (
<RiFileCopyLine className="size-3" aria-hidden />
)}
</Button>
</div>
</section>
</>
)}
{view === "features" && (
<section className="space-y-2" data-slot="popup-feature-list">
{POPUP_FEATURES.map((feature) => {
const Icon = feature.icon;
return (
<button
key={feature.id}
type="button"
className="flex w-full items-center gap-3 rounded-xl border border-border/80 bg-card/60 px-3 py-2.5 text-left transition-colors hover:bg-card/80"
onClick={() => setView(feature.id)}
data-slot={`popup-feature-${feature.id}`}
>
<Icon className="size-4 shrink-0 text-muted-foreground" aria-hidden />
<span className="min-w-0 flex-1">
<span className="block text-xs font-medium text-foreground">
{t(feature.titleKey)}
</span>
<span className="mt-0.5 block text-[11px] leading-snug text-muted-foreground">
{t(feature.descKey)}
</span>
</span>
<RiArrowRightSLine className="size-4 shrink-0 text-muted-foreground" aria-hidden />
</button>
);
})}
</section>
)}
{view === "record" && (
<section className="space-y-2.5" data-slot="popup-record-body">
<p
className="text-[11px] leading-snug text-muted-foreground"
data-slot="popup-record-hint-desc"
>
{t("popup.record.sectionHint")}
</p>
<div className="flex flex-col gap-1.5" data-slot="popup-record-purpose-field">
<Label htmlFor="bh-record-purpose" className="block text-xs text-muted-foreground">
{t("popup.record.topicLabel")}
</Label>
<Input
id="bh-record-purpose"
type="text"
value={purposeDraft}
onChange={(event: ChangeEvent<HTMLInputElement>) =>
setPurposeDraft(event.target.value)
}
placeholder={t("popup.record.topicPlaceholder")}
className="mt-0 h-8 text-sm"
data-slot="popup-record-purpose-input"
/>
</div>
<div className="flex flex-col gap-1.5" data-slot="popup-record-start-url-field">
<Label htmlFor="bh-record-start-url" className="block text-xs text-muted-foreground">
{t("popup.record.startUrlLabel")}
</Label>
<Input
id="bh-record-start-url"
type="url"
value={startUrlDraft}
onChange={(event: ChangeEvent<HTMLInputElement>) =>
setStartUrlDraft(event.target.value)
}
placeholder={t("popup.record.startUrlPlaceholder")}
className="mt-0 h-8 text-sm"
data-slot="popup-record-start-url-input"
/>
</div>
<textarea
readOnly
value={recordPrompt}
rows={5}
className="w-full resize-none rounded-md border border-input bg-muted/40 px-2.5 py-2 font-mono text-[10px] leading-snug text-foreground/90 focus-visible:outline-none"
data-slot="popup-record-prompt"
/>
<div className="flex items-center justify-between gap-2">
{!recordReady && (
<p className="text-[10px] text-muted-foreground" data-slot="popup-record-hint">
{t("popup.record.hintDisconnected")}
</p>
)}
<div className="relative ml-auto shrink-0">
{showCopiedToast && (
<div
role="status"
className="absolute bottom-full right-0 mb-1.5 flex items-center gap-1 whitespace-nowrap rounded-md bg-foreground/65 px-2 py-1 text-[10px] font-medium text-background shadow-md backdrop-blur-sm"
data-slot="popup-record-copied-toast"
>
<RiCheckLine className="size-3" aria-hidden />
{t("popup.record.copied")}
</div>
)}
<Button
type="button"
variant="secondary"
size="sm"
className="h-7 px-2.5 text-xs"
disabled={!recordReady}
onClick={() => {
void copyRecordPrompt();
}}
data-slot="popup-record-copy"
>
<RiFileCopyLine className="size-3.5" aria-hidden />
{t("popup.record.copyButton")}
</Button>
</div>
</div>
</section>
)}
</main>
);
}
@@ -0,0 +1,22 @@
import type { RemixiconComponentType } from "@remixicon/react";
import { RiRecordCircleLine } from "@remixicon/react";
export type PopupFeatureId = "record";
export type PopupView = "main" | "features" | PopupFeatureId;
export type PopupFeature = {
id: PopupFeatureId;
icon: RemixiconComponentType;
titleKey: "popup.record.sectionTitle";
descKey: "popup.record.cardDesc";
};
export const POPUP_FEATURES: PopupFeature[] = [
{
id: "record",
icon: RiRecordCircleLine,
titleKey: "popup.record.sectionTitle",
descKey: "popup.record.cardDesc",
},
];
@@ -1,3 +1,13 @@
@import "@browser-skill/ui/styles";
@source "../../";
@source "../../../../../packages/ui/src";
/* The shared base sets html/body/#root to height:100%, which is meant for
full-page surfaces. In a popup that pins the window to the tallest content
it ever showed, so collapsing the record section leaves blank space. Size
to content instead so Chrome can grow and shrink the popup to fit. */
html,
body,
#root {
height: auto;
}
@@ -0,0 +1,77 @@
import { describe, expect, it, vi } from "vitest";
import {
CAPTURE_SUPPRESS,
type CaptureSuppressMessage,
withOverlaysHiddenForCapture,
} from "../capture-suppress-bridge";
function recordingSendToTab(events: string[]) {
return vi.fn(async (_tabId: number, message: CaptureSuppressMessage) => {
events.push(message.phase);
return { type: CAPTURE_SUPPRESS, ok: true };
});
}
describe("withOverlaysHiddenForCapture", () => {
it("sends begin → fn → end and returns fn's result", async () => {
const events: string[] = [];
const sendToTab = recordingSendToTab(events);
const result = await withOverlaysHiddenForCapture(
7,
async () => {
events.push("capture");
return "shot";
},
sendToTab,
);
expect(result).toBe("shot");
expect(events).toEqual(["begin", "capture", "end"]);
expect(sendToTab).toHaveBeenCalledWith(7, { type: CAPTURE_SUPPRESS, phase: "begin" });
expect(sendToTab).toHaveBeenCalledWith(7, { type: CAPTURE_SUPPRESS, phase: "end" });
});
it("still sends end when fn throws, then rethrows", async () => {
const events: string[] = [];
const sendToTab = recordingSendToTab(events);
await expect(
withOverlaysHiddenForCapture(
7,
async () => {
events.push("capture");
throw new Error("capture exploded");
},
sendToTab,
),
).rejects.toThrow("capture exploded");
expect(events).toEqual(["begin", "capture", "end"]);
});
it("captures directly when the tab has no content script", async () => {
const sendToTab = vi.fn(async () => {
throw new Error("Could not establish connection. Receiving end does not exist.");
});
const fn = vi.fn(async () => "shot");
const result = await withOverlaysHiddenForCapture(7, fn, sendToTab);
expect(result).toBe("shot");
expect(fn).toHaveBeenCalledTimes(1);
// begin failed → no matching end may be sent.
expect(sendToTab).toHaveBeenCalledTimes(1);
});
it("swallows end failures so the capture result stands", async () => {
const sendToTab = vi.fn(async (_tabId: number, message: CaptureSuppressMessage) => {
if (message.phase === "end") throw new Error("tab navigated away");
return { type: CAPTURE_SUPPRESS, ok: true };
});
const result = await withOverlaysHiddenForCapture(7, async () => "shot", sendToTab);
expect(result).toBe("shot");
expect(sendToTab).toHaveBeenCalledTimes(2);
});
});
@@ -1,10 +1,8 @@
import { ConnectionController, __testing__ } from "../connection-controller";
import type { Transport } from "../../transport/transport";
import type { ConnectionState } from "../../transport/types";
import type { ConnectionStateHandler } from "../../transport/transport";
import { describe, expect, it, vi, beforeEach } from "vitest";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { MIN_COMPATIBLE_PROTOCOL } from "../../transport/handshake";
import type { HandshakeResult } from "../../transport/types";
import type { ConnectionStateHandler, Transport } from "../../transport/transport";
import type { ConnectionState, HandshakeResult } from "../../transport/types";
import { __testing__, ConnectionController } from "../connection-controller";
vi.mock("../instance-id", () => ({
getOrCreateInstanceId: vi.fn(async () => "a1b2c3d4"),
@@ -0,0 +1,113 @@
import { beforeEach, describe, expect, it } from "vitest";
import { describeEventTarget, describeTarget, isMeaningfulClickTarget } from "../describe-target";
describe("describeTarget", () => {
beforeEach(() => {
document.body.innerHTML = "";
});
it("prefers aria-label and button role", () => {
document.body.innerHTML = `<button aria-label="发布">go</button>`;
const el = document.querySelector("button")!;
expect(describeTarget(el)).toEqual({
tag: "button",
role: "button",
name: "发布",
});
});
it("uses label text and name_attr for inputs", () => {
document.body.innerHTML = `
<label for="svc">服务名称</label>
<input id="svc" name="serviceName" />
`;
const el = document.querySelector("input")!;
expect(describeTarget(el)).toEqual({
tag: "input",
role: "textbox",
name: "服务名称",
name_attr: "serviceName",
});
});
it("never invents CSS selectors or @eN", () => {
document.body.innerHTML = `<a href="/x">详情</a>`;
const desc = describeTarget(document.querySelector("a")!);
expect(JSON.stringify(desc)).not.toMatch(/@e\d+|nth-of-type|#/);
expect(desc).toEqual({ tag: "a", role: "link", name: "详情" });
});
it("prefers heading text inside SERP-style links", () => {
document.body.innerHTML = `
<a href="https://www.tencent.com">
<h3>Tencent 腾讯</h3>
<cite>https://www.tencent.com</cite>
</a>
`;
expect(describeTarget(document.querySelector("a")!).name).toBe("Tencent 腾讯");
});
});
describe("LLM textbook click targets", () => {
beforeEach(() => {
document.body.innerHTML = "";
});
it("rejects tag-only targets the LLM cannot find", () => {
expect(isMeaningfulClickTarget({ tag: "div" })).toBe(false);
});
it("treats a short visible name as the teaching signal", () => {
expect(isMeaningfulClickTarget({ tag: "div", name: "发布" })).toBe(true);
});
it("ignores anonymous layout chrome and lifts child clicks to the named button", () => {
document.body.innerHTML = `
<div id="noise">background</div>
<button type="button"><span id="icon">发布</span></button>
`;
expect(describeEventTarget(document.querySelector("#noise"))).toBeNull();
expect(describeEventTarget(document.querySelector("#icon"))).toEqual({
tag: "button",
role: "button",
name: "发布",
});
});
it("does not attach layout sibling text as nearby_label on buttons", () => {
document.body.innerHTML = `
<div>background</div>
<button type="button" id="pub">发布</button>
`;
expect(describeTarget(document.querySelector("#pub")!)).toEqual({
tag: "button",
role: "button",
name: "发布",
});
});
it("keeps nearby_label for form fields from a preceding label-like sibling", () => {
document.body.innerHTML = `
<div>服务名称</div>
<input id="svc" />
`;
expect(describeTarget(document.querySelector("#svc")!)).toEqual(
expect.objectContaining({
tag: "input",
nearby_label: "服务名称",
}),
);
});
it("does not treat plain text blocks as clicks", () => {
document.body.innerHTML = `<p id="copy">说明文字</p>`;
expect(describeEventTarget(document.querySelector("#copy"))).toBeNull();
});
it("records custom button-like controls by their visible label", () => {
document.body.innerHTML = `<div class="btn" id="pub" style="cursor: pointer">发布</div>`;
expect(describeEventTarget(document.querySelector("#pub"))).toEqual(
expect.objectContaining({ tag: "div", name: "发布" }),
);
});
});
@@ -0,0 +1,130 @@
import { describe, expect, it, vi } from "vitest";
import type { ProtocolFrame } from "@/transport/types";
import { HEARTBEAT_EVENT, HEARTBEAT_INTERVAL_MS, startHeartbeat } from "../heartbeat";
function makeTimers() {
let nextId = 1;
const intervals = new Map<number, { cb: () => void; ms: number }>();
return {
intervals,
api: {
setInterval: (cb: () => void, ms: number) => {
const id = nextId++;
intervals.set(id, { cb, ms });
return id;
},
clearInterval: (id: number) => {
intervals.delete(id);
},
},
tick: (id: number) => intervals.get(id)?.cb(),
};
}
/** Drives `onActiveChange` subscribers so a test can flip the link state. */
function makeActiveSource() {
const listeners = new Set<(active: boolean) => void>();
return {
subscribe: (cb: (active: boolean) => void) => {
// Mirror the controller contract: fire immediately with the current
// value (starts inactive), then on every change.
cb(false);
listeners.add(cb);
return () => listeners.delete(cb);
},
set: (active: boolean) => {
for (const cb of listeners) cb(active);
},
listenerCount: () => listeners.size,
};
}
describe("startHeartbeat", () => {
it("does not beat while the link is inactive", () => {
const timers = makeTimers();
const active = makeActiveSource();
const send = vi.fn();
startHeartbeat({ send, onActiveChange: active.subscribe, timers: timers.api });
expect(timers.intervals.size).toBe(0);
expect(send).not.toHaveBeenCalled();
});
it("starts a 20s interval on activation and sends heartbeat events", () => {
const timers = makeTimers();
const active = makeActiveSource();
const send = vi.fn();
startHeartbeat({ send, onActiveChange: active.subscribe, timers: timers.api });
active.set(true);
expect(timers.intervals.size).toBe(1);
const [id, info] = [...timers.intervals.entries()][0];
expect(info.ms).toBe(HEARTBEAT_INTERVAL_MS);
timers.tick(id);
expect(send).toHaveBeenCalledWith({ event: HEARTBEAT_EVENT, payload: {} });
});
it("stops beating when the link goes inactive", () => {
const timers = makeTimers();
const active = makeActiveSource();
const send = vi.fn();
startHeartbeat({ send, onActiveChange: active.subscribe, timers: timers.api });
active.set(true);
expect(timers.intervals.size).toBe(1);
active.set(false);
expect(timers.intervals.size).toBe(0);
});
it("does not stack intervals on repeated activation", () => {
const timers = makeTimers();
const active = makeActiveSource();
startHeartbeat({ send: vi.fn(), onActiveChange: active.subscribe, timers: timers.api });
active.set(true);
active.set(true);
expect(timers.intervals.size).toBe(1);
});
it("swallows send errors so the interval keeps running", () => {
const timers = makeTimers();
const active = makeActiveSource();
const send = vi.fn(() => {
throw new Error("socket closed");
});
const handle = startHeartbeat({ send, onActiveChange: active.subscribe, timers: timers.api });
expect(() => handle.beatForTest()).not.toThrow();
});
it("dispose clears the interval and unsubscribes", () => {
const timers = makeTimers();
const active = makeActiveSource();
const handle = startHeartbeat({
send: vi.fn(),
onActiveChange: active.subscribe,
timers: timers.api,
});
active.set(true);
expect(timers.intervals.size).toBe(1);
handle.dispose();
expect(timers.intervals.size).toBe(0);
expect(active.listenerCount()).toBe(0);
});
it("sends a well-formed protocol frame", () => {
const timers = makeTimers();
const active = makeActiveSource();
let captured: ProtocolFrame | null = null;
const handle = startHeartbeat({
send: (f) => {
captured = f;
},
onActiveChange: active.subscribe,
timers: timers.api,
});
handle.beatForTest();
expect(captured).toEqual({ event: HEARTBEAT_EVENT, payload: {} });
});
});
@@ -1,10 +1,17 @@
import { describe, expect, it } from "vitest";
import {
HELP_ACK,
HELP_CANCEL,
HELP_FINISH,
HELP_QUERY,
HELP_REQUEST,
HELP_RESPONSE,
isHelpAckMessage,
isHelpCancelMessage,
isHelpFinishMessage,
isHelpQueryMessage,
isHelpRequestMessage,
isHelpResponseMessage,
} from "../help-bridge";
describe("help-bridge", () => {
@@ -12,6 +19,9 @@ describe("help-bridge", () => {
expect(HELP_REQUEST).toBe("bsk-help-request");
expect(HELP_RESPONSE).toBe("bsk-help-response");
expect(HELP_CANCEL).toBe("bsk-help-cancel");
expect(HELP_ACK).toBe("bsk-help-ack");
expect(HELP_QUERY).toBe("bsk-help-query");
expect(HELP_FINISH).toBe("bsk-help-finish");
});
it("type-guards a help-request message", () => {
@@ -20,6 +30,7 @@ describe("help-bridge", () => {
type: HELP_REQUEST,
requestId: "r1",
prompt: "log in",
displayMode: "compact",
selectors: ["#login"],
timeoutMs: 1000,
}),
@@ -51,12 +62,22 @@ describe("help-bridge", () => {
});
it("type-guards a help-cancel message", () => {
expect(
isHelpCancelMessage({ type: HELP_CANCEL, requestId: "r1" }),
).toBe(true);
expect(isHelpCancelMessage({ type: HELP_REQUEST, requestId: "r1" })).toBe(
false,
);
expect(isHelpCancelMessage({ type: HELP_CANCEL, requestId: "r1" })).toBe(true);
expect(isHelpCancelMessage({ type: HELP_REQUEST, requestId: "r1" })).toBe(false);
expect(isHelpCancelMessage(null)).toBe(false);
});
it("type-guards lifecycle messages", () => {
expect(isHelpAckMessage({ type: HELP_ACK, ok: true })).toBe(true);
expect(isHelpQueryMessage({ type: HELP_QUERY })).toBe(true);
expect(isHelpFinishMessage({ type: HELP_FINISH, requestId: "r1", outcome: "continued" })).toBe(
true,
);
expect(isHelpResponseMessage({ type: HELP_RESPONSE, outcome: "cancelled", note: "no" })).toBe(
true,
);
expect(isHelpFinishMessage({ type: HELP_FINISH, requestId: "r1", outcome: "weird" })).toBe(
false,
);
});
});
@@ -1,10 +1,10 @@
import { describe, it, expect } from "vitest";
import { describe, expect, it } from "vitest";
import {
isOverlayAgentOverlayResetMessage,
OVERLAY_AGENT_OVERLAY_RESET,
OVERLAY_MSG_INTERRUPT,
type OverlayInterruptRequest,
type OverlayInterruptResponse,
isOverlayAgentOverlayResetMessage,
} from "@/lib/overlay-bridge";
describe("OVERLAY_MSG_INTERRUPT", () => {
@@ -1,4 +1,4 @@
import { describe, it, expect, vi } from "vitest";
import { describe, expect, it, vi } from "vitest";
import { sendInterrupt } from "@/lib/overlay-interrupt-client";
describe("sendInterrupt", () => {
@@ -0,0 +1,80 @@
import { describe, expect, it } from "vitest";
import { appendRecordedPayload, observeRecordedNavigation } from "../recording-step-buffer";
describe("recording-step-buffer", () => {
it("stores semantic click without summary", () => {
const buffer = { steps: [], pendingNavigation: false };
appendRecordedPayload(buffer, {
op: "click",
target: { tag: "button", role: "button", name: "发布" },
expects_navigation: true,
});
expect(buffer.steps).toEqual([
{
op: "click",
target: { tag: "button", role: "button", name: "发布" },
},
]);
expect(buffer.pendingNavigation).toBe(true);
});
it("annotates navigated_to on action-caused navigation instead of wait_for_navigation", () => {
const buffer = {
steps: [
{
op: "click" as const,
target: { tag: "button", role: "button", name: "发布" },
},
],
currentUrl: "https://example.com/a",
pendingNavigation: true,
pendingNavigationDeadline: Date.now() + 5_000,
};
observeRecordedNavigation(buffer, "https://example.com/b", true);
expect(buffer.steps).toEqual([
{
op: "click",
target: { tag: "button", role: "button", name: "发布" },
navigated_to: "https://example.com/b",
},
]);
expect(JSON.stringify(buffer.steps)).not.toContain("wait_for_navigation");
});
it("annotates navigated_to onto a select that auto-submits a navigation", () => {
const buffer = {
steps: [
{
op: "select" as const,
target: { tag: "select", role: "combobox", name: "分类" },
values: ["tech"],
},
],
currentUrl: "https://example.com/list",
pendingNavigation: true,
pendingNavigationDeadline: Date.now() + 5_000,
};
observeRecordedNavigation(buffer, "https://example.com/list?cat=tech", true);
expect(buffer.steps).toEqual([
{
op: "select",
target: { tag: "select", role: "combobox", name: "分类" },
values: ["tech"],
navigated_to: "https://example.com/list?cat=tech",
},
]);
});
it("emits navigate for uncaused URL changes", () => {
const buffer = {
steps: [],
currentUrl: "https://example.com/a",
pendingNavigation: false,
};
observeRecordedNavigation(buffer, "https://example.com/b", false);
expect(buffer.steps[0]).toMatchObject({
op: "navigate",
url: "https://example.com/b",
});
});
});
@@ -0,0 +1,141 @@
import { describe, expect, it } from "vitest";
import type { DraftTraceStep } from "@/transport/types";
import { reduceTraceSteps, resolveTraceStartUrl, shouldRecordPress } from "../trace-reducer";
describe("shouldRecordPress", () => {
it("keeps Enter and Escape", () => {
expect(shouldRecordPress("Enter")).toBe(true);
expect(shouldRecordPress("Escape")).toBe(true);
});
it("drops modifiers, clipboard shortcuts, and bare typing", () => {
expect(shouldRecordPress("Meta")).toBe(false);
expect(shouldRecordPress("c", ["meta"])).toBe(false);
expect(shouldRecordPress("a", ["ctrl"])).toBe(false);
expect(shouldRecordPress("x")).toBe(false);
expect(shouldRecordPress("中")).toBe(false);
});
});
describe("reduceTraceSteps", () => {
it("builds steps with pages dictionary and page id references", () => {
const drafts: DraftTraceStep[] = [
{
op: "navigate",
url: "https://example.com/search?q=hello&utm_source=x",
page_url: "https://example.com/search?q=hello&utm_source=x",
},
{
op: "fill",
target: { tag: "input", role: "textbox", name: "搜索", name_attr: "q" },
value: "browser skill",
page_url: "https://example.com/search?q=hello&utm_source=x",
},
{
op: "press",
key: "Enter",
target: { tag: "input", role: "textbox", name: "搜索", name_attr: "q" },
navigated_to: "https://example.com/results/42",
page_url: "https://example.com/search?q=hello&utm_source=x",
},
{
op: "click",
target: { tag: "button", role: "button", name: "发布" },
navigated_to: "https://example.com/p/99",
page_url: "https://example.com/results/42",
},
{
op: "press",
key: "a",
page_url: "https://example.com/p/99",
},
];
const { pages, steps } = reduceTraceSteps(
drafts,
"https://example.com/search?q=hello&utm_source=x",
);
expect(JSON.stringify(steps)).not.toContain("parameters");
expect(JSON.stringify(steps)).not.toContain("intent");
expect(JSON.stringify(steps)).not.toContain("summary");
expect(steps.map((s) => s.op)).toEqual(["navigate", "fill", "press", "click"]);
expect(pages.map((p) => p.url)).toEqual([
"https://example.com/search?q=hello&utm_source=x",
"https://example.com/results/42",
"https://example.com/p/99",
]);
expect(steps[0]).toMatchObject({
id: 1,
op: "navigate",
page: "p1",
to: "https://example.com/search?q=hello&utm_source=x",
});
expect(steps[1]).toMatchObject({
op: "fill",
page: "p1",
value: "browser skill",
});
expect(steps[2]).toMatchObject({
op: "press",
key: "Enter",
page: "p1",
effect: { navigated_to: "p2" },
});
expect(steps[3]).toMatchObject({
op: "click",
page: "p2",
effect: { navigated_to: "p3" },
});
});
it("collapses consecutive navigations", () => {
const { steps } = reduceTraceSteps([
{ op: "navigate", url: "https://a.example/redirect1" },
{ op: "navigate", url: "https://a.example/final" },
]);
expect(steps).toHaveLength(1);
expect(steps[0]).toMatchObject({
op: "navigate",
to: "https://a.example/final",
});
});
it("maps select navigated_to onto effect.navigated_to (page id)", () => {
const { pages, steps } = reduceTraceSteps(
[
{
op: "select",
target: { tag: "select", role: "combobox", name: "分类" },
values: ["tech"],
labels: ["技术"],
navigated_to: "https://example.com/list?cat=tech",
page_url: "https://example.com/list",
},
],
"https://example.com/list",
);
expect(pages.map((p) => p.url)).toEqual([
"https://example.com/list",
"https://example.com/list?cat=tech",
]);
expect(steps[0]).toMatchObject({
op: "select",
page: "p1",
selection: [{ value: "tech", label: "技术" }],
effect: { navigated_to: "p2" },
});
});
it("resolveTraceStartUrl prefers explicit start URL", () => {
expect(
resolveTraceStartUrl(
[{ op: "navigate", url: "https://example.com/other" }],
"https://example.com/start",
),
).toBe("https://example.com/start");
});
});
@@ -0,0 +1,71 @@
/**
* Wire protocol and background-side helper for hiding the extension's
* in-page overlay while a screenshot is captured, so captured frames only
* contain the page itself.
*
* The background wraps every capture in `withOverlaysHiddenForCapture`,
* which sends `begin` (the content script hides the overlay host and only
* acks once the compositor has produced an overlay-free frame) and always
* follows up with `end`, even when the capture throws. Tabs without the
* content script (chrome://, the Web Store, ...) host no overlay, so a
* failed `begin` falls through to capturing directly.
*/
export const CAPTURE_SUPPRESS = "bsk/capture-suppress";
export type CaptureSuppressPhase = "begin" | "end";
export interface CaptureSuppressMessage {
type: typeof CAPTURE_SUPPRESS;
phase: CaptureSuppressPhase;
}
export interface CaptureSuppressAck {
type: typeof CAPTURE_SUPPRESS;
ok: true;
}
export function isCaptureSuppressMessage(msg: unknown): msg is CaptureSuppressMessage {
if (typeof msg !== "object" || msg === null) return false;
const m = msg as Record<string, unknown>;
return m.type === CAPTURE_SUPPRESS && (m.phase === "begin" || m.phase === "end");
}
/** Minimal `chrome.tabs.sendMessage` surface so tests can inject a fake. */
export type CaptureSuppressSendToTab = (
tabId: number,
message: CaptureSuppressMessage,
) => Promise<unknown>;
const defaultSendToTab: CaptureSuppressSendToTab = (tabId, message) =>
chrome.tabs.sendMessage(tabId, message);
/**
* Run `fn` with the target tab's overlay hidden. `end` is sent from a
* `finally` so the overlay always reappears, and a missing content script
* (or any other `begin` failure) simply skips suppression — there is no
* overlay to hide in that tab.
*/
export async function withOverlaysHiddenForCapture<T>(
tabId: number,
fn: () => Promise<T>,
sendToTab: CaptureSuppressSendToTab = defaultSendToTab,
): Promise<T> {
let began = false;
try {
await sendToTab(tabId, { type: CAPTURE_SUPPRESS, phase: "begin" });
began = true;
} catch {
// No content script in the target tab → no overlay can leak into the
// capture; proceed without suppression.
}
try {
return await fn();
} finally {
if (began) {
await sendToTab(tabId, { type: CAPTURE_SUPPRESS, phase: "end" }).catch((err) => {
console.debug("[bsk capture] overlay restore failed", err);
});
}
}
}
+345
View File
@@ -0,0 +1,345 @@
/**
* Build a semantic TargetDescriptor for an interacted element.
*
* Trace steps are an LLM *textbook*: each click must say what to look for
* on screen (usually a short visible name). Tag-only noise like
* `{ "tag": "div" }` / “点击div” is useless and must not be recorded.
*/
export interface TargetDescriptor {
role?: string;
name?: string;
tag: string;
name_attr?: string;
placeholder?: string;
nearby_label?: string;
}
/** Max length for a label that is still a useful “find this on screen” hint. */
const ACTIONABLE_LABEL_MAX = 48;
function normalizeWhitespace(value: string): string {
return value.replace(/\s+/g, " ").trim();
}
function truncate(value: string, max = 80): string {
const trimmed = normalizeWhitespace(value);
if (trimmed.length <= max) return trimmed;
return `${trimmed.slice(0, max - 1)}…`;
}
/** Short, human label an LLM can search for in a snapshot. */
export function isActionableLabel(name: string): boolean {
const trimmed = normalizeWhitespace(name);
if (!trimmed) return false;
if (trimmed.length > ACTIONABLE_LABEL_MAX) return false;
if (/^https?:\/\//i.test(trimmed)) return false;
return true;
}
function labelledByText(el: Element): string | null {
const ids = el.getAttribute("aria-labelledby");
if (!ids) return null;
const parts = ids
.split(/\s+/)
.map((id) => document.getElementById(id)?.textContent ?? "")
.map(normalizeWhitespace)
.filter(Boolean);
return parts.length ? truncate(parts.join(" ")) : null;
}
function associatedLabelText(el: Element): string | null {
if (!(el instanceof HTMLElement) || !el.id) return null;
const label = document.querySelector(`label[for="${CSS.escape(el.id)}"]`);
if (!label) return null;
const text = normalizeWhitespace(label.textContent ?? "");
return text ? truncate(text) : null;
}
function wrappingLabelText(el: Element): string | null {
const label = el.closest("label");
if (!label) return null;
const clone = label.cloneNode(true) as HTMLElement;
for (const control of clone.querySelectorAll("input,textarea,select,button")) {
control.remove();
}
const text = normalizeWhitespace(clone.textContent ?? "");
return text ? truncate(text) : null;
}
function placeholderOrTitle(el: Element): string | null {
if (el instanceof HTMLInputElement || el instanceof HTMLTextAreaElement) {
const ph = normalizeWhitespace(el.placeholder);
if (ph) return truncate(ph);
}
const title = normalizeWhitespace(el.getAttribute("title") ?? "");
return title ? truncate(title) : null;
}
function ownText(el: Element): string | null {
if (el instanceof HTMLInputElement || el instanceof HTMLTextAreaElement) {
return null;
}
const text = normalizeWhitespace(el.textContent ?? "");
return text ? truncate(text) : null;
}
/** Prefer a compact title inside links (e.g. SERP result `<h3>`), not the whole blob. */
function compactLinkName(el: Element): string | null {
const heading = el.querySelector("h1, h2, h3, h4");
if (heading) {
const text = normalizeWhitespace(heading.textContent ?? "");
if (text) return truncate(text);
}
const img = el.querySelector("img[alt]");
if (img instanceof HTMLImageElement) {
const alt = normalizeWhitespace(img.alt);
if (alt) return truncate(alt);
}
const clone = el.cloneNode(true) as HTMLElement;
for (const junk of clone.querySelectorAll("cite, script, style, svg, noscript")) {
junk.remove();
}
const text = normalizeWhitespace(clone.textContent ?? "");
if (!text) return null;
const withoutUrl = text.replace(/https?:\/\/\S+/g, "").trim();
return truncate(withoutUrl || text);
}
/** Visible accessible name preference order for LLM textbooks. */
export function accessibleName(el: Element): string | undefined {
const ariaLabel = normalizeWhitespace(el.getAttribute("aria-label") ?? "");
if (ariaLabel) return truncate(ariaLabel);
const labelledBy = labelledByText(el);
if (labelledBy) return labelledBy;
const forLabel = associatedLabelText(el);
if (forLabel) return forLabel;
const wrapLabel = wrappingLabelText(el);
if (wrapLabel) return wrapLabel;
const alt =
el instanceof HTMLImageElement
? normalizeWhitespace(el.alt)
: normalizeWhitespace(el.getAttribute("alt") ?? "");
if (alt) return truncate(alt);
const ph = placeholderOrTitle(el);
if (ph) return ph;
const tag = el.tagName.toLowerCase();
if (tag === "a" || el.getAttribute("role") === "link") {
const linkName = compactLinkName(el);
if (linkName) return linkName;
}
const text = ownText(el);
if (text) return text;
if (el instanceof HTMLInputElement || el instanceof HTMLButtonElement) {
const value = normalizeWhitespace(el.value);
if (
value &&
(el instanceof HTMLButtonElement || el.type === "submit" || el.type === "button")
) {
return truncate(value);
}
}
return undefined;
}
const INTERACTIVE_SELECTOR = [
"a[href]",
"button",
"summary",
"select",
'input:not([type="hidden"])',
"textarea",
'[role="button"]',
'[role="link"]',
'[role="tab"]',
'[role="menuitem"]',
'[role="menuitemcheckbox"]',
'[role="menuitemradio"]',
'[role="option"]',
'[role="checkbox"]',
'[role="radio"]',
'[role="switch"]',
'[role="treeitem"]',
'[role="combobox"]',
'[contenteditable]:not([contenteditable="false"])',
].join(",");
const INTERACTIVE_ROLES = new Set([
"button",
"link",
"tab",
"menuitem",
"menuitemcheckbox",
"menuitemradio",
"option",
"checkbox",
"radio",
"switch",
"treeitem",
"combobox",
"textbox",
"searchbox",
"slider",
]);
function looksClickable(el: Element): boolean {
if (!(el instanceof HTMLElement)) return false;
if (el.matches(INTERACTIVE_SELECTOR)) return true;
const role = normalizeWhitespace(el.getAttribute("role") ?? "").toLowerCase();
if (role && INTERACTIVE_ROLES.has(role)) return true;
if (el.tabIndex >= 0) return true;
try {
if (getComputedStyle(el).cursor === "pointer") return true;
} catch {
// happy-dom / detached nodes
}
const cls = typeof el.className === "string" ? el.className : "";
if (/\b(btn|button|link|clickable|action)\b/i.test(cls)) return true;
return false;
}
/**
* Resolve the element a click should describe for the LLM textbook.
* Never falls back to an anonymous layout `div`.
*/
export function resolveClickableElement(target: Element): Element | null {
const interactive = target.closest(INTERACTIVE_SELECTOR);
if (interactive instanceof Element) return interactive;
let node: Element | null = target;
let depth = 0;
while (node && node !== document.body && node !== document.documentElement && depth < 8) {
const name = accessibleName(node);
if (name && isActionableLabel(name) && looksClickable(node)) {
return node;
}
node = node.parentElement;
depth += 1;
}
return null;
}
/**
* Teachable clicks: the LLM must get a label it can later find via snapshot
* (visible name), or at least a form `name_attr` for checkbox/radio.
* Recording “点击div” with no name fails this bar.
*/
export function isMeaningfulClickTarget(target: TargetDescriptor): boolean {
const name = target.name?.trim();
if (name && isActionableLabel(name)) return true;
if (
(target.role === "checkbox" || target.role === "radio" || target.tag === "input") &&
target.name_attr
) {
return true;
}
return false;
}
export function inferRole(el: Element): string | undefined {
const explicit = normalizeWhitespace(el.getAttribute("role") ?? "").toLowerCase();
if (explicit) return explicit;
const tag = el.tagName.toLowerCase();
if (tag === "button") return "button";
if (tag === "a" && el.hasAttribute("href")) return "link";
if (tag === "select") return "combobox";
if (tag === "textarea") return "textbox";
if (tag === "img") return "img";
if (tag === "summary") return "button";
if (el instanceof HTMLElement && (el.isContentEditable || el.contentEditable === "true")) {
return "textbox";
}
if (el instanceof HTMLInputElement) {
const type = el.type.toLowerCase();
if (type === "checkbox") return "checkbox";
if (type === "radio") return "radio";
if (type === "submit" || type === "button" || type === "image" || type === "reset") {
return "button";
}
if (type === "range") return "slider";
return "textbox";
}
return undefined;
}
/**
* Nearby labels help the LLM name form fields. Only apply to fillable
* controls — never to buttons/links, where a previous layout sibling
* (e.g. a chrome `<div>`) would pollute the textbook target.
*/
function nearbyLabelText(el: Element): string | undefined {
if (
!(
el instanceof HTMLInputElement ||
el instanceof HTMLTextAreaElement ||
el instanceof HTMLSelectElement
)
) {
return undefined;
}
if (el.id) {
const byFor = document.querySelector(`label[for="${CSS.escape(el.id)}"]`);
const text = byFor ? normalizeWhitespace(byFor.textContent ?? "") : "";
if (text) return truncate(text, ACTIONABLE_LABEL_MAX);
}
const wrapped = el.closest("label");
if (wrapped) {
const clone = wrapped.cloneNode(true) as HTMLLabelElement;
for (const control of clone.querySelectorAll("input, textarea, select, button")) {
control.remove();
}
const text = normalizeWhitespace(clone.textContent ?? "");
if (text) return truncate(text, ACTIONABLE_LABEL_MAX);
}
const prev = el.previousElementSibling;
if (prev && /^(LABEL|SPAN|DIV|P|DT)$/i.test(prev.tagName)) {
const text = normalizeWhitespace(prev.textContent ?? "");
if (text && text.length <= ACTIONABLE_LABEL_MAX) return text;
}
return undefined;
}
export function describeTarget(el: Element): TargetDescriptor {
const tag = el.tagName.toLowerCase();
const role = inferRole(el);
const name = accessibleName(el);
const nameAttr =
el instanceof HTMLInputElement ||
el instanceof HTMLTextAreaElement ||
el instanceof HTMLSelectElement ||
el instanceof HTMLButtonElement
? normalizeWhitespace(el.name) || undefined
: normalizeWhitespace(el.getAttribute("name") ?? "") || undefined;
const placeholder =
el instanceof HTMLInputElement || el instanceof HTMLTextAreaElement
? normalizeWhitespace(el.placeholder) || undefined
: normalizeWhitespace(el.getAttribute("placeholder") ?? "") || undefined;
const nearby = nearbyLabelText(el);
return {
tag,
...(role ? { role } : {}),
...(name ? { name } : {}),
...(nameAttr ? { name_attr: nameAttr } : {}),
...(placeholder ? { placeholder } : {}),
...(nearby && nearby !== name ? { nearby_label: nearby } : {}),
};
}
export function describeEventTarget(target: EventTarget | null): TargetDescriptor | null {
if (!(target instanceof Element)) return null;
const clickable = resolveClickableElement(target);
if (!clickable) return null;
const desc = describeTarget(clickable);
return isMeaningfulClickTarget(desc) ? desc : null;
}
+110
View File
@@ -0,0 +1,110 @@
/**
* Application-level WebSocket heartbeat.
*
* Since Chrome 116, sending or receiving a message over a WebSocket
* resets the MV3 service-worker idle timer (see
* https://developer.chrome.com/docs/extensions/how-to/web-platform/websockets).
* By emitting a tiny `system.heartbeat` event every 20s — comfortably
* inside the 30s idle window — we keep the worker (and therefore the
* daemon link) alive for as long as the connection is up, instead of
* relying solely on the 30s keepalive alarm that sits right on the
* eviction boundary.
*
* The daemon also treats the heartbeat as a liveness signal, so a
* silently-dead browser can be reaped on its side.
*
* The heartbeat only runs while the *post-handshake* link is live: the
* daemon rejects any first frame that is not `system.handshake`, so
* beating before the handshake completes would get the connection
* kicked. Callers wire {@link HeartbeatOptions.onActiveChange} to the
* connection controller, which only reports `connected` / `version_skew`
* after the handshake has landed.
*/
import type { ProtocolFrame } from "@/transport/types";
export const HEARTBEAT_EVENT = "system.heartbeat";
/** 20s: the interval Chrome's own WebSocket keepalive sample recommends. */
export const HEARTBEAT_INTERVAL_MS = 20_000;
/** Subset of the timer API we use; lets tests inject fakes. */
export interface HeartbeatTimers {
setInterval(cb: () => void, ms: number): number;
clearInterval(handle: number): void;
}
export interface HeartbeatOptions {
/** Send a frame over the live transport. */
send: (frame: ProtocolFrame) => void;
/**
* Subscribe to "is the post-handshake link live" changes. Must invoke
* the callback with the current value immediately and again on every
* transition, and return an unsubscribe function.
*/
onActiveChange: (cb: (active: boolean) => void) => () => void;
/** Override the beat interval (tests). */
intervalMs?: number;
/** Inject fake timers (tests). */
timers?: HeartbeatTimers;
}
export interface HeartbeatHandle {
/** Stop beating and detach the active-state listener. */
dispose(): void;
/** Test hook: emit one heartbeat frame right now. */
beatForTest(): void;
}
function defaultTimers(): HeartbeatTimers {
return {
setInterval: (cb, ms) => setInterval(cb, ms) as unknown as number,
clearInterval: (handle) => clearInterval(handle),
};
}
/**
* Start a heartbeat that beats every `intervalMs` while the link is
* active and pauses entirely while it is not.
*/
export function startHeartbeat(options: HeartbeatOptions): HeartbeatHandle {
const intervalMs = options.intervalMs ?? HEARTBEAT_INTERVAL_MS;
const timers = options.timers ?? defaultTimers();
let timer: number | null = null;
const beat = () => {
try {
options.send({ event: HEARTBEAT_EVENT, payload: {} });
} catch (err) {
// The socket raced a close between the state change and this
// tick. The transport's own reconnect path recovers; swallow so
// the interval keeps running (it is cleared on the disconnect
// notification anyway).
console.debug("[bsk heartbeat] send failed", err);
}
};
const stop = () => {
if (timer !== null) {
timers.clearInterval(timer);
timer = null;
}
};
const start = () => {
if (timer !== null) return;
timer = timers.setInterval(beat, intervalMs);
};
const unsubscribe = options.onActiveChange((active) => {
if (active) start();
else stop();
});
return {
dispose: () => {
unsubscribe();
stop();
},
beatForTest: beat,
};
}
+61 -1
View File
@@ -6,12 +6,15 @@
* Mirrors the borrow-confirmation message style (see
* `@/tools/borrow-confirmation`). The background asks a specific tab to
* enter "help mode" (render HelpRequestOverlay + hide ControlOverlay);
* the content script replies once the user clicks Continue / Cancel.
* the content script acks display, then later reports Done / Cancel.
*/
export const HELP_REQUEST = "bsk-help-request";
export const HELP_RESPONSE = "bsk-help-response";
export const HELP_CANCEL = "bsk-help-cancel";
export const HELP_ACK = "bsk-help-ack";
export const HELP_QUERY = "bsk-help-query";
export const HELP_FINISH = "bsk-help-finish";
export interface HelpRequestMessage {
type: typeof HELP_REQUEST;
@@ -19,11 +22,18 @@ export interface HelpRequestMessage {
prompt: string;
/** Custom overlay title; omitted when the extension should use its default. */
title?: string;
/** Full task UI on the subject tab; compact status UI on related tabs. */
displayMode?: "full" | "compact";
/** CSS selectors to scroll to + flash-highlight (may be empty). */
selectors: string[];
timeoutMs: number;
}
export interface HelpAckMessage {
type: typeof HELP_ACK;
ok: true;
}
export interface HelpResponseMessage {
type: typeof HELP_RESPONSE;
outcome: "continued" | "cancelled";
@@ -35,6 +45,22 @@ export interface HelpCancelMessage {
requestId: string;
}
export interface HelpQueryMessage {
type: typeof HELP_QUERY;
}
export interface HelpQueryResponse {
active: boolean;
request?: Omit<HelpRequestMessage, "type">;
}
export interface HelpFinishMessage {
type: typeof HELP_FINISH;
requestId: string;
outcome: "continued" | "cancelled";
note?: string;
}
export function isHelpRequestMessage(msg: unknown): msg is HelpRequestMessage {
if (typeof msg !== "object" || msg === null) {
return false;
@@ -45,6 +71,7 @@ export function isHelpRequestMessage(msg: unknown): msg is HelpRequestMessage {
typeof m.requestId === "string" &&
typeof m.prompt === "string" &&
(m.title === undefined || typeof m.title === "string") &&
(m.displayMode === undefined || m.displayMode === "full" || m.displayMode === "compact") &&
Array.isArray(m.selectors) &&
m.selectors.every((selector) => typeof selector === "string") &&
typeof m.timeoutMs === "number"
@@ -58,3 +85,36 @@ export function isHelpCancelMessage(msg: unknown): msg is HelpCancelMessage {
const m = msg as Record<string, unknown>;
return m.type === HELP_CANCEL && typeof m.requestId === "string";
}
export function isHelpResponseMessage(msg: unknown): msg is HelpResponseMessage {
if (typeof msg !== "object" || msg === null) return false;
const m = msg as Record<string, unknown>;
return (
m.type === HELP_RESPONSE &&
(m.outcome === "continued" || m.outcome === "cancelled") &&
(m.note === undefined || typeof m.note === "string")
);
}
export function isHelpAckMessage(msg: unknown): msg is HelpAckMessage {
if (typeof msg !== "object" || msg === null) return false;
const m = msg as Record<string, unknown>;
return m.type === HELP_ACK && m.ok === true;
}
export function isHelpQueryMessage(msg: unknown): msg is HelpQueryMessage {
if (typeof msg !== "object" || msg === null) return false;
const m = msg as Record<string, unknown>;
return m.type === HELP_QUERY;
}
export function isHelpFinishMessage(msg: unknown): msg is HelpFinishMessage {
if (typeof msg !== "object" || msg === null) return false;
const m = msg as Record<string, unknown>;
return (
m.type === HELP_FINISH &&
typeof m.requestId === "string" &&
(m.outcome === "continued" || m.outcome === "cancelled") &&
(m.note === undefined || typeof m.note === "string")
);
}
+70 -4
View File
@@ -3,9 +3,8 @@
* between the content-script control overlay and the background SW.
*
* Content script → background:
* - `{ kind: "overlay.who_am_i", tabId, windowId }` → background
* replies `{ sessionId: string | null }` so the overlay knows
* whether to render itself.
* - `{ kind: "overlay.ready" }` → background replies with the authoritative
* overlay state for the sender's window.
* - `{ kind: "overlay.interrupt", sessionId }` → background asks the
* daemon (via a `session.user_interrupt` WS event) to cancel
* every inflight + queued tool call for that session with
@@ -14,8 +13,40 @@
*/
export const OVERLAY_MSG_WHO_AM_I = "overlay.who_am_i";
export const OVERLAY_MSG_READY = "overlay.ready";
export const OVERLAY_MSG_INTERRUPT = "overlay.interrupt";
/**
* WXT shadow-host element name (`createShadowRootUi({ name })`) and the marker
* attribute set on the host for the agent's own overlay UI. The VOM capture
* adapter uses these to skip the overlay host and its shadow subtree.
*/
export const OVERLAY_HOST_NAME = "browser-skill-overlay";
export const OVERLAY_HOST_MARKER_ATTR = "data-bsk-overlay";
/** CSS selector for the WXT shadow host (tag + marker attribute). */
export const OVERLAY_HOST_SELECTOR = `${OVERLAY_HOST_NAME}, [${OVERLAY_HOST_MARKER_ATTR}]`;
export function isOverlayHostElementName(tagName: string): boolean {
return tagName.toLowerCase() === OVERLAY_HOST_NAME;
}
export function isOverlayHostMarkerAttribute(attrName: string): boolean {
return attrName.toLowerCase() === OVERLAY_HOST_MARKER_ATTR;
}
export function isOverlayHostNode(tagName: string, attributeNames?: Iterable<string>): boolean {
if (isOverlayHostElementName(tagName)) return true;
if (!attributeNames) return false;
for (const name of attributeNames) {
if (isOverlayHostMarkerAttribute(name)) return true;
}
return false;
}
/** Page-world `document.querySelector(...)` for the overlay shadow host. */
export const OVERLAY_HOST_LOOKUP_EXPR = `document.querySelector(${JSON.stringify(OVERLAY_HOST_SELECTOR)})`;
export interface OverlayWhoAmIRequest {
kind: typeof OVERLAY_MSG_WHO_AM_I;
tabId?: number;
@@ -26,6 +57,22 @@ export interface OverlayWhoAmIResponse {
sessionId: string | null;
}
export interface OverlayReadyRequest {
kind: typeof OVERLAY_MSG_READY;
}
export type OverlayMode = "control" | "interrupting" | "paused" | "hidden";
/** Background → content: complete, authoritative control-overlay state. */
export const OVERLAY_AGENT_STATE = "bh-agent-overlay-state";
export interface OverlayAgentStateMessage {
type: typeof OVERLAY_AGENT_STATE;
sessionId: string | null;
mode: OverlayMode;
generation: number;
}
export interface OverlayInterruptRequest {
kind: typeof OVERLAY_MSG_INTERRUPT;
sessionId: string;
@@ -59,4 +106,23 @@ export function isOverlayAgentOverlayResetMessage(
return candidate.type === OVERLAY_AGENT_OVERLAY_RESET && typeof candidate.sessionId === "string";
}
export type OverlayMessage = OverlayWhoAmIRequest | OverlayInterruptRequest;
export function isOverlayAgentStateMessage(message: unknown): message is OverlayAgentStateMessage {
if (!message || typeof message !== "object") return false;
const candidate = message as {
type?: unknown;
sessionId?: unknown;
mode?: unknown;
generation?: unknown;
};
return (
candidate.type === OVERLAY_AGENT_STATE &&
(typeof candidate.sessionId === "string" || candidate.sessionId === null) &&
(candidate.mode === "control" ||
candidate.mode === "interrupting" ||
candidate.mode === "paused" ||
candidate.mode === "hidden") &&
typeof candidate.generation === "number"
);
}
export type OverlayMessage = OverlayWhoAmIRequest | OverlayReadyRequest | OverlayInterruptRequest;
+119
View File
@@ -0,0 +1,119 @@
/**
* Wire protocol for user-action recording, sent between the background
* service worker and a tab's content script.
*/
import type { TargetDescriptor } from "./describe-target";
export const RECORD_START = "bsk-record-start";
export const RECORD_STEP = "bsk-record-step";
export const RECORD_STOP = "bsk-record-stop";
export const RECORD_CANCEL = "bsk-record-cancel";
export const RECORD_FINISH = "bsk-record-finish";
export const RECORD_QUERY = "bsk-record-query";
export interface RecordStartAck {
ok: true;
}
export type RecordStopAck = { ok: true } | { ok: false; error: string };
export interface RecordQueryMessage {
type: typeof RECORD_QUERY;
}
export interface RecordQueryResponse {
active: boolean;
requestId?: string;
/** Epoch ms when the recording began; see `RecordStartMessage.startedAtMs`. */
startedAtMs?: number;
}
export interface RecordStartMessage {
type: typeof RECORD_START;
requestId: string;
/**
* Epoch ms when the whole recording began, not when this tab was armed.
* The overlay timer must span the session, so it survives navigations and
* content-script remounts instead of restarting per page.
*/
startedAtMs?: number;
}
export interface RecordStepPayload {
op: "click" | "fill" | "press" | "select" | "navigate";
target?: TargetDescriptor;
value?: string;
key?: string;
modifiers?: Array<"alt" | "ctrl" | "meta" | "shift">;
values?: string[];
labels?: string[];
url?: string;
redacted?: boolean;
/** Page URL when the step was captured. */
page_url?: string;
/** Capture-only hint; never persisted unless converted to navigated_to. */
expects_navigation?: boolean;
/** Whether an observed URL change was synchronously caused by the action. */
navigation_caused_by_action?: boolean;
}
export interface RecordStepMessage {
type: typeof RECORD_STEP;
requestId: string;
step: RecordStepPayload;
}
export interface RecordStopMessage {
type: typeof RECORD_STOP;
requestId: string;
}
export interface RecordCancelMessage {
type: typeof RECORD_CANCEL;
requestId: string;
}
export interface RecordFinishMessage {
type: typeof RECORD_FINISH;
requestId: string;
}
export function isRecordStartMessage(msg: unknown): msg is RecordStartMessage {
if (typeof msg !== "object" || msg === null) return false;
const m = msg as Record<string, unknown>;
return m.type === RECORD_START && typeof m.requestId === "string";
}
export function isRecordStopMessage(msg: unknown): msg is RecordStopMessage {
if (typeof msg !== "object" || msg === null) return false;
const m = msg as Record<string, unknown>;
return m.type === RECORD_STOP && typeof m.requestId === "string";
}
export function isRecordCancelMessage(msg: unknown): msg is RecordCancelMessage {
if (typeof msg !== "object" || msg === null) return false;
const m = msg as Record<string, unknown>;
return m.type === RECORD_CANCEL && typeof m.requestId === "string";
}
export function isRecordFinishMessage(msg: unknown): msg is RecordFinishMessage {
if (typeof msg !== "object" || msg === null) return false;
const m = msg as Record<string, unknown>;
return m.type === RECORD_FINISH && typeof m.requestId === "string";
}
export function isRecordQueryMessage(msg: unknown): msg is RecordQueryMessage {
if (typeof msg !== "object" || msg === null) return false;
const m = msg as Record<string, unknown>;
return m.type === RECORD_QUERY;
}
export function isRecordStepMessage(msg: unknown): msg is RecordStepMessage {
if (typeof msg !== "object" || msg === null) return false;
const m = msg as Record<string, unknown>;
if (m.type !== RECORD_STEP || typeof m.requestId !== "string") return false;
const step = m.step;
if (typeof step !== "object" || step === null) return false;
return typeof (step as RecordStepPayload).op === "string";
}
@@ -0,0 +1,128 @@
import type { DraftTraceStep } from "@/transport/types";
import type { RecordStepPayload } from "./record-bridge";
export interface RecordingStepBuffer {
steps: DraftTraceStep[];
currentUrl?: string;
pendingNavigation: boolean;
pendingNavigationDeadline?: number;
}
const NAVIGATION_TRIGGER_WINDOW_MS = 3_000;
function toDraftStep(payload: RecordStepPayload): DraftTraceStep | null {
const pageUrl = payload.page_url;
switch (payload.op) {
case "click":
return payload.target
? {
op: "click",
target: payload.target,
...(pageUrl ? { page_url: pageUrl } : {}),
}
: null;
case "fill":
return payload.target
? {
op: "fill",
target: payload.target,
value: payload.value ?? "",
...(payload.redacted ? { redacted: true } : {}),
...(pageUrl ? { page_url: pageUrl } : {}),
}
: null;
case "press":
return payload.key
? {
op: "press",
key: payload.key,
...(payload.target ? { target: payload.target } : {}),
...(payload.modifiers?.length ? { modifiers: payload.modifiers } : {}),
...(pageUrl ? { page_url: pageUrl } : {}),
}
: null;
case "select":
return payload.target && payload.values
? {
op: "select",
target: payload.target,
values: payload.values,
...(payload.labels?.length ? { labels: payload.labels } : {}),
...(pageUrl ? { page_url: pageUrl } : {}),
}
: null;
case "navigate":
return null;
}
}
function annotateLastStepNavigation(buffer: RecordingStepBuffer, url: string): boolean {
for (let i = buffer.steps.length - 1; i >= 0; i -= 1) {
const step = buffer.steps[i];
if (!step) continue;
if (step.op === "click" || step.op === "press" || step.op === "select") {
buffer.steps[i] = { ...step, navigated_to: url };
return true;
}
break;
}
return false;
}
export function observeRecordedNavigation(
buffer: RecordingStepBuffer,
url: string,
causedByAction?: boolean,
): void {
if (!url || url === buffer.currentUrl) return;
buffer.currentUrl = url;
const pendingIsCurrent =
buffer.pendingNavigation &&
(buffer.pendingNavigationDeadline === undefined ||
buffer.pendingNavigationDeadline >= Date.now());
if (causedByAction === true || (causedByAction === undefined && pendingIsCurrent)) {
buffer.pendingNavigation = false;
buffer.pendingNavigationDeadline = undefined;
if (!annotateLastStepNavigation(buffer, url)) {
buffer.steps.push({
op: "navigate",
url,
page_url: url,
});
}
return;
}
buffer.pendingNavigation = false;
buffer.pendingNavigationDeadline = undefined;
buffer.steps.push({
op: "navigate",
url,
page_url: url,
});
}
export function appendRecordedPayload(
buffer: RecordingStepBuffer,
payload: RecordStepPayload,
): void {
if (payload.op === "navigate") {
if (payload.url) {
observeRecordedNavigation(buffer, payload.url, payload.navigation_caused_by_action);
}
return;
}
const step = toDraftStep({
...payload,
page_url: payload.page_url ?? buffer.currentUrl,
});
if (!step) return;
buffer.steps.push(step);
if (step.op === "click" || step.op === "press" || step.op === "select" || step.op === "fill") {
buffer.pendingNavigation = payload.expects_navigation === true;
buffer.pendingNavigationDeadline = buffer.pendingNavigation
? Date.now() + NAVIGATION_TRIGGER_WINDOW_MS
: undefined;
}
}
+218
View File
@@ -0,0 +1,218 @@
import type { DraftTraceStep, PageRef, SelectedOption, Step } from "@/transport/types";
const CLIPBOARD_KEYS = new Set(["a", "c", "v", "x", "A", "C", "V", "X"]);
const MODIFIER_ONLY_KEYS = new Set(["Meta", "Control", "Alt", "Shift", "OS", "Hyper", "Super"]);
export function shouldRecordPress(
key: string,
modifiers?: Array<"alt" | "ctrl" | "meta" | "shift">,
): boolean {
if (MODIFIER_ONLY_KEYS.has(key)) return false;
const mods = modifiers ?? [];
const hasCtrlOrMeta = mods.includes("ctrl") || mods.includes("meta");
if (hasCtrlOrMeta && CLIPBOARD_KEYS.has(key)) return false;
if (key === "Enter" || key === "Escape") return true;
// Drop bare character typing — FillSession already records the value.
if (key.length === 1 && !hasCtrlOrMeta && !mods.includes("alt")) return false;
return false;
}
function shouldIncludeDraft(step: DraftTraceStep): boolean {
if (step.op === "fill" && !(step.value ?? "").trim() && !step.redacted) return false;
if (step.op === "press" && !shouldRecordPress(step.key, step.modifiers)) return false;
return true;
}
/** Collapse consecutive navigations to the last hop. */
function collapseNavigations(steps: DraftTraceStep[]): DraftTraceStep[] {
const out: DraftTraceStep[] = [];
for (const step of steps) {
const prev = out[out.length - 1];
if (step.op === "navigate" && prev?.op === "navigate") {
out[out.length - 1] = step;
continue;
}
out.push(step);
}
return out;
}
function collectUrls(steps: DraftTraceStep[], startUrl?: string): string[] {
const urls: string[] = [];
if (startUrl) urls.push(startUrl);
for (const step of steps) {
if (step.op === "navigate") {
urls.push(step.url);
continue;
}
if ("page_url" in step && step.page_url) urls.push(step.page_url);
if ("navigated_to" in step && step.navigated_to) urls.push(step.navigated_to);
}
const seen = new Set<string>();
const unique: string[] = [];
for (const url of urls) {
if (!url || seen.has(url)) continue;
seen.add(url);
unique.push(url);
}
return unique;
}
function buildPageRegistry(
steps: DraftTraceStep[],
startUrl?: string,
): { pages: PageRef[]; urlToId: Map<string, string> } {
const urls = collectUrls(steps, startUrl);
const urlToId = new Map<string, string>();
const pages = urls.map((url, index) => {
const id = `p${index + 1}`;
urlToId.set(url, id);
return { id, url };
});
return { pages, urlToId };
}
function pageIdFor(
url: string | undefined,
urlToId: Map<string, string>,
fallbackUrl?: string,
): string {
if (url && urlToId.has(url)) return urlToId.get(url)!;
if (fallbackUrl && urlToId.has(fallbackUrl)) return urlToId.get(fallbackUrl)!;
return urlToId.values().next().value ?? "p1";
}
function pageUrlForDraft(step: DraftTraceStep, fallbackUrl?: string): string | undefined {
if (step.op === "navigate") return step.page_url ?? step.url;
if ("page_url" in step && step.page_url) return step.page_url;
return fallbackUrl;
}
function effectForNavigation(
navigatedTo: string | undefined,
urlToId: Map<string, string>,
): Step["effect"] {
if (!navigatedTo) return undefined;
const pageId = urlToId.get(navigatedTo);
if (!pageId) return undefined;
return { navigated_to: pageId };
}
function withEffect(step: Step, effect: Step["effect"]): Step {
if (!effect) return step;
return { ...step, effect };
}
function toSelection(values: string[], labels?: string[]): SelectedOption[] {
return values.map((value, index) => ({
value,
...(labels?.[index] ? { label: labels[index] } : {}),
}));
}
function toV2Step(
step: DraftTraceStep,
id: number,
urlToId: Map<string, string>,
fallbackUrl?: string,
): Step | null {
if (!shouldIncludeDraft(step)) return null;
const pageUrl = pageUrlForDraft(step, fallbackUrl);
const page = pageIdFor(pageUrl, urlToId, fallbackUrl);
switch (step.op) {
case "navigate":
return {
op: "navigate",
id,
page: pageIdFor(step.url, urlToId, fallbackUrl),
to: step.url,
};
case "click":
return withEffect(
{
op: "click",
id,
page,
target: step.target,
},
effectForNavigation(step.navigated_to, urlToId),
);
case "fill":
return {
op: "fill",
id,
page,
target: step.target,
value: step.value,
...(step.redacted ? { redacted: true } : {}),
};
case "press":
return withEffect(
{
op: "press",
id,
page,
key: step.key,
...(step.target ? { target: step.target } : {}),
...(step.modifiers?.length ? { modifiers: step.modifiers } : {}),
},
effectForNavigation(step.navigated_to, urlToId),
);
case "select":
return withEffect(
{
op: "select",
id,
page,
target: step.target,
selection: toSelection(step.values, step.labels),
},
effectForNavigation(step.navigated_to, urlToId),
);
}
}
export interface ReducedTrace {
pages: PageRef[];
steps: Step[];
}
/**
* Compile capture drafts into record-only trace v2 steps.
* Variable inputs are NOT classified here — executing agents infer that at run time.
*/
export function reduceTraceSteps(steps: DraftTraceStep[], startUrl?: string): ReducedTrace {
const collapsed = collapseNavigations(steps);
const { pages, urlToId } = buildPageRegistry(collapsed, startUrl);
const out: Step[] = [];
let id = 1;
let lastUrl = startUrl;
for (const draft of collapsed) {
if (draft.op === "navigate") lastUrl = draft.url;
else if ("navigated_to" in draft && draft.navigated_to) lastUrl = draft.navigated_to;
else if ("page_url" in draft && draft.page_url) lastUrl = draft.page_url;
const step = toV2Step(draft, id, urlToId, lastUrl);
if (!step) continue;
out.push(step);
id += 1;
}
return { pages, steps: out };
}
export function resolveTraceStartUrl(
drafts: DraftTraceStep[],
startUrl?: string,
pages?: PageRef[],
): string {
if (startUrl) return startUrl;
const navigate = drafts.find((step): step is Extract<DraftTraceStep, { op: "navigate" }> => {
return step.op === "navigate";
});
if (navigate) return navigate.url;
for (const draft of drafts) {
if ("page_url" in draft && draft.page_url) return draft.page_url;
}
return pages?.[0]?.url ?? "about:blank";
}
@@ -39,6 +39,13 @@ describe("SessionManager", () => {
expect(ctx.refStore.isEmpty()).toBe(true);
expect(ctx.borrowedTabs.size).toBe(0);
});
it("forwards an optional window size when starting a session", async () => {
const aw = fakeAgentWindow();
const sm = new SessionManager({ agentWindow: aw });
const ctx = await sm.start("aa11", { width: 1280, height: 800 });
expect(aw.createMock).toHaveBeenCalledWith("about:blank", { width: 1280, height: 800 });
expect(ctx.agentWindowId).toBe(100);
});
it("indexes the session by sessionId and agent window id", async () => {
const aw = fakeAgentWindow();
@@ -8,7 +8,7 @@
*/
export interface AgentWindowApi {
create(url: string): Promise<number>;
create(url: string, size?: { width: number; height: number }): Promise<number>;
remove(windowId: number): Promise<void>;
/**
* Guarantee the Agent Window has an active, CDP-navigable tab.
@@ -22,11 +22,12 @@ export interface AgentWindowApi {
export const AGENT_WINDOW_HOME = "about:blank";
export const chromeAgentWindowApi: AgentWindowApi = {
async create(url: string): Promise<number> {
async create(url: string, size?: { width: number; height: number }): Promise<number> {
const win = await chrome.windows.create({
type: "normal",
focused: true,
url,
...(size ? { width: size.width, height: size.height } : {}),
});
if (typeof win?.id !== "number") {
throw new Error("[bh] chrome.windows.create returned no window id");
@@ -128,11 +128,18 @@ export class SessionManager {
* Returns the created window id so callers can echo it back to the
* daemon in the `tool.session_start` reply.
*/
async start(sessionId: string): Promise<SessionContext> {
async start(
sessionId: string,
size?: { width: number; height: number },
): Promise<SessionContext> {
if (this.sessions.has(sessionId)) {
throw new Error(`[bh] session ${sessionId} already exists`);
}
const windowId = await this.agentWindow.create(AGENT_WINDOW_HOME);
// Only pass `size` when given so the no-size call shape (and its
// chrome.windows.create payload) stays exactly as before.
const windowId = size
? await this.agentWindow.create(AGENT_WINDOW_HOME, size)
: await this.agentWindow.create(AGENT_WINDOW_HOME);
await this.agentWindow.ensureActiveTab(windowId, AGENT_WINDOW_HOME);
const ctx: SessionContext = {
sessionId,
@@ -162,6 +162,22 @@ describe("requestBorrowConfirmation", () => {
expect(await pending).toBe(false);
});
it("denies malformed or missing confirmation responses", async () => {
tabs.get.mockResolvedValue({ id: 42, title: "Tab" });
windows.getLastFocused.mockResolvedValue(
userWindowWithActiveTab({ windowId: 11, tabId: 42, url: "https://app.example/" }),
);
for (const response of [undefined, {}, { allowed: true }, { type: "borrow-response" }]) {
tabs.sendMessage.mockResolvedValueOnce(response);
const pending = requestBorrowConfirmation(42, {
deps: { tabs, windows, notifications },
});
await vi.runAllTimersAsync();
expect(await pending).toBe(false);
}
});
it("skips the Agent Window when it is lastFocusedWindow and falls back to a real user window", async () => {
tabs.get.mockResolvedValueOnce({
id: 42,
@@ -250,7 +266,7 @@ describe("requestBorrowConfirmation", () => {
expect(tabs.sendMessage.mock.calls[1]?.[0]).toBe(301);
});
it("does NOT fail-open immediately when every candidate's sendMessage fails — waits for notification button or timeout", async () => {
it("waits for a notification decision and denies when every UI path times out", async () => {
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => undefined);
tabs.get.mockResolvedValueOnce({ id: 42, title: "Tab" });
windows.getLastFocused.mockResolvedValueOnce(
@@ -282,13 +298,13 @@ describe("requestBorrowConfirmation", () => {
expect(tabs.sendMessage).toHaveBeenCalledTimes(2);
// CRITICAL: previously this would already be settled(true). Now we
// require an explicit user choice (notification button) or timeout.
// require an explicit user choice (notification button) or safe timeout.
expect(resolved).toBe(false);
// Allow the BACKGROUND_TIMEOUT_MS fail-open to fire so the test resolves.
// The background timeout must resolve the request without authorizing it.
await vi.advanceTimersByTimeAsync(BACKGROUND_TIMEOUT_MS);
expect(await pending).toBe(true);
expect(resolvedValue).toBe(true);
expect(await pending).toBe(false);
expect(resolvedValue).toBe(false);
const exhaustedWarn = warnSpy.mock.calls.find(
(call) =>
@@ -375,7 +391,7 @@ describe("requestBorrowConfirmation", () => {
expect(await pending).toBe(false);
});
it("fail-opens when no injectable user window exists at all", async () => {
it("denies when no injectable user window exists at all", async () => {
tabs.get.mockResolvedValueOnce({ id: 42, title: "Stranded Tab" });
windows.getLastFocused.mockResolvedValueOnce(
userWindowWithActiveTab({ windowId: 500, tabId: 5001, url: "about:blank" }),
@@ -394,7 +410,7 @@ describe("requestBorrowConfirmation", () => {
});
await vi.runAllTimersAsync();
expect(await pending).toBe(true);
expect(await pending).toBe(false);
expect(tabs.sendMessage).not.toHaveBeenCalled();
// No notification either — there's no actionable target window.
expect(notifications.create).not.toHaveBeenCalled();
@@ -421,6 +437,7 @@ describe("requestBorrowConfirmation", () => {
expect(notificationId.startsWith(BORROW_NOTIFICATION_PREFIX)).toBe(true);
expect(options.type).toBe("basic");
expect(options.title).toMatch(/borrow/i);
expect(options.requireInteraction).toBe(true);
expect(notifications.clear).toHaveBeenCalledWith(notificationId);
});
@@ -447,10 +464,12 @@ describe("requestBorrowConfirmation", () => {
userWindowWithActiveTab({ windowId: 77, tabId: 4242, url: "https://docs.example/" }),
);
// Keep sendMessage pending so the request stays live while we click.
// A second call may arrive if the background timeout dismisses the overlay.
let resolveSend: (v: { type: string; allowed: boolean }) => void = () => undefined;
tabs.sendMessage.mockImplementationOnce(
() => new Promise((res) => (resolveSend = res as never)),
);
tabs.sendMessage.mockResolvedValue(undefined);
const pending = requestBorrowConfirmation(42, {
deps: { tabs, windows, notifications },
@@ -460,7 +479,7 @@ describe("requestBorrowConfirmation", () => {
// Simulate user clicking the OS notification.
for (const l of listeners) l(notificationId);
await vi.runAllTimersAsync();
await Promise.resolve();
expect(windows.update).toHaveBeenCalledWith(77, { focused: true });
// Finish the borrow so the test resolves cleanly.
@@ -469,21 +488,29 @@ describe("requestBorrowConfirmation", () => {
await pending;
});
it("fail-opens after background timeout when content script never responds", async () => {
it("denies after background timeout when content script never responds", async () => {
tabs.get.mockResolvedValueOnce({ id: 42, title: "Tab" });
windows.getLastFocused.mockResolvedValueOnce(
userWindowWithActiveTab({ windowId: 11, tabId: 42, url: "https://app.example/" }),
);
tabs.sendMessage.mockImplementationOnce(() => new Promise(() => undefined));
tabs.sendMessage.mockResolvedValueOnce(undefined);
const pending = requestBorrowConfirmation(42, {
deps: { tabs, windows, notifications },
});
await vi.advanceTimersByTimeAsync(BACKGROUND_TIMEOUT_MS);
expect(await pending).toBe(true);
expect(await pending).toBe(false);
// Timeout must dismiss any in-flight overlay so Allow cannot linger.
expect(tabs.sendMessage).toHaveBeenCalledTimes(2);
const cancelMessage = tabs.sendMessage.mock.calls[1][1] as {
type: string;
requestId: string;
};
expect(cancelMessage.type).toBe("borrow-cancel");
});
it("dismisses pending overlay and fail-opens when aborted", async () => {
it("dismisses the pending overlay and denies when aborted", async () => {
tabs.get.mockResolvedValueOnce({ id: 42, title: "Borrow Target" });
windows.getLastFocused.mockResolvedValueOnce(
userWindowWithActiveTab({ windowId: 11, tabId: 7, url: "https://app.example/" }),
@@ -500,7 +527,7 @@ describe("requestBorrowConfirmation", () => {
controller.abort();
await vi.runAllTimersAsync();
expect(await pending).toBe(true);
expect(await pending).toBe(false);
expect(tabs.sendMessage).toHaveBeenCalledTimes(2);
const requestMessage = tabs.sendMessage.mock.calls[0][1] as { requestId: string };
const cancelMessage = tabs.sendMessage.mock.calls[1][1] as {
@@ -510,6 +537,48 @@ describe("requestBorrowConfirmation", () => {
expect(cancelMessage.type).toBe("borrow-cancel");
expect(cancelMessage.requestId).toBe(requestMessage.requestId);
});
it("does not fail-open when tabs.get rejects — still surfaces confirmation to a candidate", async () => {
// Regression: a transient tabs.get failure (SW teardown / page hiccup)
// used to return true immediately, approving the borrow with no
// confirmation UI. The tab was already fetched by the borrow pipeline
// moments earlier, so the failure is transient — we must fall back to a
// generic title and still ask the user.
tabs.get.mockRejectedValueOnce(new Error("tab info unavailable"));
windows.getLastFocused.mockResolvedValueOnce(
userWindowWithActiveTab({ windowId: 50, tabId: 7, url: "https://app.example/" }),
);
tabs.sendMessage.mockResolvedValueOnce({ type: "borrow-response", allowed: false });
const pending = requestBorrowConfirmation(42, {
deps: { tabs, windows, notifications },
});
await vi.runAllTimersAsync();
const allowed = await pending;
// The user's explicit deny must be honoured — NOT auto-allowed by the
// earlier tabs.get rejection.
expect(allowed).toBe(false);
expect(tabs.sendMessage).toHaveBeenCalledTimes(1);
expect(notifications.create).toHaveBeenCalledTimes(1);
});
it("still allows an explicit overlay allow after a tabs.get rejection", async () => {
tabs.get.mockRejectedValueOnce(new Error("tab info unavailable"));
windows.getLastFocused.mockResolvedValueOnce(
userWindowWithActiveTab({ windowId: 50, tabId: 7, url: "https://app.example/" }),
);
tabs.sendMessage.mockResolvedValueOnce({ type: "borrow-response", allowed: true });
const pending = requestBorrowConfirmation(42, {
deps: { tabs, windows, notifications },
});
await vi.runAllTimersAsync();
const allowed = await pending;
expect(allowed).toBe(true);
expect(tabs.sendMessage).toHaveBeenCalledTimes(1);
});
});
describe("isInjectableContentScriptUrl", () => {
@@ -0,0 +1,112 @@
import { describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import type { CdpRunner } from "@/tools/shared";
import type { ConsoleResult } from "@/transport/types";
import { handleConsole } from "../console";
function fakeAgentWindow(ids: number[]) {
let i = 0;
return {
create: vi.fn(async () => {
const id = ids[i++];
if (id === undefined) throw new Error("ran out of fake ids");
return id;
}),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
};
}
function makeDeps(
opts: {
get?: (tabId: number) => Promise<chrome.tabs.Tab>;
query?: (query: chrome.tabs.QueryInfo) => Promise<chrome.tabs.Tab[]>;
result?: ConsoleResult;
} = {},
) {
const result =
opts.result ??
({
tab_id: 7,
entries: [],
next_since: 0,
truncated: false,
} satisfies ConsoleResult);
const ensureConsoleCapture = vi.fn(async () => {});
const consoleEntriesSince = vi.fn(() => result);
const cdp = {
send: vi.fn(),
ensureConsoleCapture,
consoleEntriesSince,
} as unknown as CdpRunner;
const tabsApi = {
get:
opts.get ??
vi.fn(
async (tabId: number) => ({ id: tabId, windowId: 100, active: true }) as chrome.tabs.Tab,
),
query:
opts.query ?? vi.fn(async () => [{ id: 7, windowId: 100, active: true } as chrome.tabs.Tab]),
};
return { cdp, tabsApi, ensureConsoleCapture, consoleEntriesSince };
}
describe("handleConsole", () => {
it("reads the Agent Window active tab with safe defaults", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
const deps = makeDeps({
result: {
tab_id: 7,
entries: [{ sequence: 3, kind: "console", level: "log", text: "hello", truncated: false }],
next_since: 3,
truncated: false,
},
});
const res = await handleConsole(sm, { session_id: "aa11" }, deps);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.entries[0].text).toBe("hello");
expect(deps.ensureConsoleCapture).toHaveBeenCalledWith(7);
expect(deps.consoleEntriesSince).toHaveBeenCalledWith(7, undefined, 50, 1000, false);
});
it("reads an explicit tab and forwards cursor options", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
const deps = makeDeps({
get: vi.fn(async () => ({ id: 9, windowId: 200, active: true }) as chrome.tabs.Tab),
});
await handleConsole(
sm,
{
session_id: "aa11",
tab_id: 9,
since: 12,
limit: 75,
max_text_chars: 1500,
include_stack: true,
},
deps,
);
expect(deps.ensureConsoleCapture).toHaveBeenCalledWith(9);
expect(deps.consoleEntriesSince).toHaveBeenCalledWith(9, 12, 75, 1500, true);
});
it("hides other sessions' Agent Window tabs", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100, 101]) });
await sm.start("aa11");
await sm.start("bb22");
const deps = makeDeps({
get: vi.fn(async () => ({ id: 9, windowId: 101, active: true }) as chrome.tabs.Tab),
});
const res = await handleConsole(sm, { session_id: "aa11", tab_id: 9 }, deps);
expect(res).toMatchObject({ code: "not_found" });
expect(deps.ensureConsoleCapture).not.toHaveBeenCalled();
});
});
@@ -1,7 +1,12 @@
import { describe, expect, it, vi } from "vitest";
import { afterEach, describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import type { ConnectionStateHandler, FrameHandler, Transport } from "@/transport/transport";
import type { ConnectionState, ProtocolFrame, RequestFrame } from "@/transport/types";
import type {
ConnectionState,
ConsoleResult,
ProtocolFrame,
RequestFrame,
} from "@/transport/types";
import { ToolDispatcher } from "../dispatcher";
function fakeTransport() {
@@ -36,6 +41,10 @@ function makeRequest(method: string, params: unknown): RequestFrame {
}
describe("ToolDispatcher", () => {
afterEach(() => {
vi.unstubAllGlobals();
});
it("routes tool.session_start to the SessionManager and replies with the window id", async () => {
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
@@ -78,6 +87,66 @@ describe("ToolDispatcher", () => {
expect(sent[0]).toEqual({ id: "r-1", result: {} });
});
it("routes tool.console through the CDP console buffer", async () => {
vi.stubGlobal("chrome", {
tabs: {
query: vi.fn(async () => [{ id: 7, windowId: 4242, active: true }]),
},
});
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
},
});
await sessions.start("aa11");
const cdp = {
send: vi.fn(),
detachSession: vi.fn(async () => {}),
ensureConsoleCapture: vi.fn(async () => {}),
ensureNetworkCapture: vi.fn(async () => {}),
networkEntriesSince: vi.fn(() => ({
tab_id: 7,
entries: [],
next_since: 0,
truncated: false,
})),
consoleEntriesSince: vi.fn(
() =>
({
tab_id: 7,
entries: [
{ sequence: 1, kind: "console", level: "log", text: "hello", truncated: false },
],
next_since: 1,
truncated: false,
}) satisfies ConsoleResult,
),
setDeviceMetricsOverride: vi.fn(async () => {}),
clearDeviceMetricsOverride: vi.fn(async () => {}),
setUserAgentOverride: vi.fn(async () => {}),
setTouchEmulationEnabled: vi.fn(async () => {}),
};
const dispatcher = new ToolDispatcher({ transport, sessions, cdp });
dispatcher.start();
deliver(makeRequest("tool.console", { session_id: "aa11" }));
await flushMicrotasks();
expect(cdp.ensureConsoleCapture).toHaveBeenCalledWith(7);
expect(sent[0]).toEqual({
id: "r-1",
result: {
tab_id: 7,
entries: [{ sequence: 1, kind: "console", level: "log", text: "hello", truncated: false }],
next_since: 1,
truncated: false,
},
});
});
it("detaches CDP state before stopping a session", async () => {
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
@@ -91,6 +160,17 @@ describe("ToolDispatcher", () => {
const cdp = {
send: vi.fn(),
detachSession: vi.fn(async () => {}),
ensureNetworkCapture: vi.fn(async () => {}),
networkEntriesSince: vi.fn(() => ({
tab_id: 7,
entries: [],
next_since: 0,
truncated: false,
})),
setDeviceMetricsOverride: vi.fn(async () => {}),
clearDeviceMetricsOverride: vi.fn(async () => {}),
setUserAgentOverride: vi.fn(async () => {}),
setTouchEmulationEnabled: vi.fn(async () => {}),
};
const dispatcher = new ToolDispatcher({ transport, sessions, cdp });
dispatcher.start();
@@ -202,6 +282,35 @@ describe("ToolDispatcher", () => {
expect(onSessionsChanged).toHaveBeenCalledTimes(2);
});
it("invokes onBrowserControlResumed for browser-control tools but not passive reads", async () => {
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
},
});
const onBrowserControlResumed = vi.fn();
const dispatcher = new ToolDispatcher({ transport, sessions, onBrowserControlResumed });
dispatcher.start();
deliver(makeRequest("tool.session_start", { session_id: "aa11" }));
await flushMicrotasks();
sent.length = 0;
deliver({ ...makeRequest("tool.snapshot", { session_id: "aa11" }), id: "r-passive" });
await flushMicrotasks();
expect(onBrowserControlResumed).not.toHaveBeenCalled();
deliver({
...makeRequest("tool.tab_create", { session_id: "aa11", url: "https://example.test/" }),
id: "r-control",
});
await flushMicrotasks();
expect(onBrowserControlResumed).toHaveBeenCalledWith("aa11");
});
it("disconnects the transport when send() fails so keepalive can rebuild", async () => {
const { transport, deliver } = fakeTransport();
const sessions = new SessionManager({
@@ -0,0 +1,416 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import type { DeviceMetricsOverride, UserAgentOverride } from "@/browser-driver/chromium-cdp";
import { SessionManager } from "@/session-manager/manager";
import type { EmulateOverrides, RpcError } from "@/transport/types";
import {
EMULATE_SCOPE_NOTE,
type EmulateCdpRunner,
handleEmulate,
resetEmulateStatesForTests,
toCdpUserAgentMetadata,
validateOverrides,
} from "../emulate";
function fakeAgentWindow(ids: number[]) {
let i = 0;
return {
create: vi.fn(async () => {
const id = ids[i++];
if (id === undefined) throw new Error("ran out of fake ids");
return id;
}),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
};
}
interface CdpCalls {
metrics: Array<{ tabId: number; metrics: DeviceMetricsOverride }>;
clearedMetrics: number[];
ua: Array<{ tabId: number; override: UserAgentOverride }>;
touch: Array<{ tabId: number; enabled: boolean; maxTouchPoints?: number }>;
}
function makeDeps(opts: { throwOn?: string; tabWindowId?: number } = {}) {
const calls: CdpCalls = { metrics: [], clearedMetrics: [], ua: [], touch: [] };
const maybeThrow = (kind: string) => {
if (opts.throwOn === kind) throw new Error(`simulated ${kind} failure`);
};
const cdp: EmulateCdpRunner = {
setDeviceMetricsOverride: vi.fn(async (tabId, metrics) => {
maybeThrow("metrics");
calls.metrics.push({ tabId, metrics });
}),
clearDeviceMetricsOverride: vi.fn(async (tabId) => {
maybeThrow("metrics");
calls.clearedMetrics.push(tabId);
}),
setUserAgentOverride: vi.fn(async (tabId, override) => {
maybeThrow("ua");
calls.ua.push({ tabId, override });
}),
setTouchEmulationEnabled: vi.fn(async (tabId, enabled, maxTouchPoints) => {
maybeThrow("touch");
calls.touch.push({ tabId, enabled, maxTouchPoints });
}),
trackSessionTab: vi.fn(),
};
const tabsApi = {
get: vi.fn(
async (tabId: number) =>
({ id: tabId, windowId: opts.tabWindowId ?? 100, active: true }) as chrome.tabs.Tab,
),
query: vi.fn(async () => [
{ id: 7, windowId: opts.tabWindowId ?? 100, active: true } as chrome.tabs.Tab,
]),
};
return { cdp, tabsApi, calls };
}
async function makeManager(): Promise<SessionManager> {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
return sm;
}
const fullOverrides: EmulateOverrides = {
width: 390,
height: 844,
device_scale_factor: 3,
mobile: true,
user_agent: "Mozilla/5.0 (iPhone)",
accept_language: "zh-CN",
touch: true,
max_touch_points: 5,
};
describe("handleEmulate", () => {
beforeEach(() => {
resetEmulateStatesForTests();
});
it("applies viewport, UA and touch overrides to the active tab", async () => {
const sm = await makeManager();
const deps = makeDeps();
const result = await handleEmulate(sm, { session_id: "aa11", overrides: fullOverrides }, deps);
expect(result).toEqual({
tab_id: 7,
cleared: false,
applied: fullOverrides,
note: EMULATE_SCOPE_NOTE,
});
expect(deps.calls.metrics).toEqual([
{ tabId: 7, metrics: { width: 390, height: 844, deviceScaleFactor: 3, mobile: true } },
]);
expect(deps.calls.ua).toEqual([
{
tabId: 7,
override: { userAgent: "Mozilla/5.0 (iPhone)", acceptLanguage: "zh-CN" },
},
]);
expect(deps.calls.touch).toEqual([{ tabId: 7, enabled: true, maxTouchPoints: 5 }]);
expect(deps.cdp.trackSessionTab).toHaveBeenCalledWith("aa11", 7);
});
it("defaults deviceScaleFactor to 0 and mobile to false when omitted", async () => {
const sm = await makeManager();
const deps = makeDeps();
const result = await handleEmulate(
sm,
{ session_id: "aa11", overrides: { width: 800, height: 1280 } },
deps,
);
expect(result).toMatchObject({ tab_id: 7, cleared: false });
expect(deps.calls.metrics).toEqual([
{ tabId: 7, metrics: { width: 800, height: 1280, deviceScaleFactor: 0, mobile: false } },
]);
expect(deps.calls.ua).toEqual([]);
expect(deps.calls.touch).toEqual([]);
});
it("applies a UA-only override", async () => {
const sm = await makeManager();
const deps = makeDeps();
const result = await handleEmulate(
sm,
{ session_id: "aa11", overrides: { user_agent: "custom-ua" } },
deps,
);
expect(result).toMatchObject({ tab_id: 7, cleared: false });
expect(deps.calls.metrics).toEqual([]);
expect(deps.calls.ua).toEqual([{ tabId: 7, override: { userAgent: "custom-ua" } }]);
});
it("converts user_agent_metadata to CDP camelCase", async () => {
const sm = await makeManager();
const deps = makeDeps();
const result = await handleEmulate(
sm,
{
session_id: "aa11",
overrides: {
user_agent: "custom-ua",
user_agent_metadata: {
brands: [{ brand: "Chromium", version: "120" }],
full_version: "120.0.0.0",
platform: "Android",
platform_version: "14",
mobile: true,
},
},
},
deps,
);
expect(result).toMatchObject({ tab_id: 7 });
expect(deps.calls.ua[0].override.userAgentMetadata).toEqual({
brands: [{ brand: "Chromium", version: "120" }],
fullVersion: "120.0.0.0",
platform: "Android",
platformVersion: "14",
mobile: true,
});
});
it("max_touch_points without touch enables touch emulation", async () => {
const sm = await makeManager();
const deps = makeDeps();
const result = await handleEmulate(
sm,
{ session_id: "aa11", overrides: { max_touch_points: 5 } },
deps,
);
expect(result).toMatchObject({ tab_id: 7, cleared: false });
expect(deps.calls.touch).toEqual([{ tabId: 7, enabled: true, maxTouchPoints: 5 }]);
});
it("touch: false only disables touch emulation", async () => {
const sm = await makeManager();
const deps = makeDeps();
const result = await handleEmulate(
sm,
{ session_id: "aa11", overrides: { touch: false } },
deps,
);
expect(result).toMatchObject({ tab_id: 7, cleared: false });
expect(deps.calls.touch).toEqual([{ tabId: 7, enabled: false, maxTouchPoints: undefined }]);
});
it("targets an explicit tab_id", async () => {
const sm = await makeManager();
const deps = makeDeps();
const result = await handleEmulate(
sm,
{ session_id: "aa11", tab_id: 9, overrides: { user_agent: "custom-ua" } },
deps,
);
expect(result).toMatchObject({ tab_id: 9 });
expect(deps.calls.ua).toEqual([{ tabId: 9, override: { userAgent: "custom-ua" } }]);
});
it("off clears metrics, touch and UA overrides", async () => {
const sm = await makeManager();
const deps = makeDeps();
const result = await handleEmulate(sm, { session_id: "aa11", off: true }, deps);
expect(result).toEqual({ tab_id: 7, cleared: true, note: EMULATE_SCOPE_NOTE });
expect(deps.calls.clearedMetrics).toEqual([7]);
expect(deps.calls.touch).toEqual([{ tabId: 7, enabled: false, maxTouchPoints: undefined }]);
expect(deps.calls.ua).toEqual([{ tabId: 7, override: { userAgent: "" } }]);
});
it("rejects off combined with overrides", async () => {
const sm = await makeManager();
const deps = makeDeps();
const result = await handleEmulate(
sm,
{ session_id: "aa11", off: true, overrides: { touch: true } },
deps,
);
expect(result).toMatchObject({ code: "invalid_params" });
expect(deps.calls.clearedMetrics).toEqual([]);
});
it("rejects missing overrides and missing off", async () => {
const sm = await makeManager();
const deps = makeDeps();
const result = await handleEmulate(sm, { session_id: "aa11" }, deps);
expect(result).toMatchObject({ code: "invalid_params" });
});
it("rejects an unknown session", async () => {
const sm = await makeManager();
const deps = makeDeps();
const result = await handleEmulate(sm, { session_id: "zz99", off: true }, deps);
expect(result).toMatchObject({ code: "not_found" });
});
it("rejects a tab outside the Agent Window", async () => {
const sm = await makeManager();
const deps = makeDeps({ tabWindowId: 200 });
const result = await handleEmulate(sm, { session_id: "aa11", tab_id: 9, off: true }, deps);
expect(result).toMatchObject({ code: "permission_denied" });
expect(deps.calls.clearedMetrics).toEqual([]);
});
it("merges onto the tab's stored state: later width/height keep the earlier dpr/mobile", async () => {
const sm = await makeManager();
const deps = makeDeps();
await handleEmulate(sm, { session_id: "aa11", overrides: fullOverrides }, deps);
const result = await handleEmulate(
sm,
{ session_id: "aa11", overrides: { width: 500, height: 900 } },
deps,
);
expect(result).toMatchObject({
tab_id: 7,
cleared: false,
applied: { width: 500, height: 900, device_scale_factor: 3, mobile: true },
});
expect(deps.calls.metrics).toEqual([
{ tabId: 7, metrics: { width: 390, height: 844, deviceScaleFactor: 3, mobile: true } },
{ tabId: 7, metrics: { width: 500, height: 900, deviceScaleFactor: 3, mobile: true } },
]);
// The merged state is applied as a whole: UA and touch are re-applied too.
expect(deps.calls.ua).toHaveLength(2);
expect(deps.calls.touch).toHaveLength(2);
});
it("keeps emulation state per tab", async () => {
const sm = await makeManager();
const deps = makeDeps();
await handleEmulate(sm, { session_id: "aa11", overrides: fullOverrides }, deps);
await handleEmulate(
sm,
{ session_id: "aa11", tab_id: 9, overrides: { width: 500, height: 900 } },
deps,
);
// Tab 9 has no stored state, so dpr/mobile fall back to the CDP defaults.
expect(deps.calls.metrics.at(-1)).toEqual({
tabId: 9,
metrics: { width: 500, height: 900, deviceScaleFactor: 0, mobile: false },
});
});
it("off clears the stored state: a later request starts from scratch", async () => {
const sm = await makeManager();
const deps = makeDeps();
await handleEmulate(sm, { session_id: "aa11", overrides: fullOverrides }, deps);
await handleEmulate(sm, { session_id: "aa11", off: true }, deps);
const result = await handleEmulate(
sm,
{ session_id: "aa11", overrides: { width: 500, height: 900 } },
deps,
);
expect(result).toEqual({
tab_id: 7,
cleared: false,
applied: { width: 500, height: 900 },
note: EMULATE_SCOPE_NOTE,
});
// dpr/mobile fell back to the CDP defaults instead of the cleared values,
// and the cleared UA/touch were not re-applied.
expect(deps.calls.metrics.at(-1)).toEqual({
tabId: 7,
metrics: { width: 500, height: 900, deviceScaleFactor: 0, mobile: false },
});
expect(deps.calls.ua).toHaveLength(2); // preset + the clearing empty UA
expect(deps.calls.touch).toHaveLength(2); // preset + the disabling call
});
it("maps CDP failures to cdp_failed and points at --off", async () => {
const sm = await makeManager();
const deps = makeDeps({ throwOn: "ua" });
const result = await handleEmulate(
sm,
{ session_id: "aa11", overrides: { user_agent: "custom-ua" } },
deps,
);
expect(result).toMatchObject({ code: "cdp_failed" });
const message = (result as RpcError).message;
expect(message).toContain("simulated ua failure");
expect(message).toContain("not rolled back");
expect(message).toContain("--off");
});
it("reports an off-branch CDP failure with the same not-rolled-back note", async () => {
const sm = await makeManager();
const deps = makeDeps({ throwOn: "metrics" });
const result = await handleEmulate(sm, { session_id: "aa11", off: true }, deps);
expect(result).toMatchObject({ code: "cdp_failed" });
const message = (result as RpcError).message;
expect(message).toContain("simulated metrics failure");
expect(message).toContain("not rolled back");
expect(message).toContain("--off");
});
});
describe("validateOverrides", () => {
it("accepts a full override set", () => {
expect(validateOverrides(fullOverrides)).toBeNull();
});
it("rejects width without height and vice versa", () => {
expect(validateOverrides({ width: 390 })).toMatchObject({ code: "invalid_params" });
expect(validateOverrides({ height: 844 })).toMatchObject({ code: "invalid_params" });
});
it("rejects non-positive or non-integer dimensions", () => {
expect(validateOverrides({ width: 0, height: 844 })).toMatchObject({
code: "invalid_params",
});
expect(validateOverrides({ width: 390.5, height: 844 })).toMatchObject({
code: "invalid_params",
});
});
it("rejects metrics extras without dimensions", () => {
expect(validateOverrides({ device_scale_factor: 3 })).toMatchObject({
code: "invalid_params",
});
expect(validateOverrides({ mobile: true })).toMatchObject({ code: "invalid_params" });
});
it("rejects a non-positive device_scale_factor", () => {
expect(validateOverrides({ width: 390, height: 844, device_scale_factor: 0 })).toMatchObject({
code: "invalid_params",
});
expect(
validateOverrides({ width: 390, height: 844, device_scale_factor: Number.NaN }),
).toMatchObject({ code: "invalid_params" });
});
it("rejects accept_language / user_agent_metadata without user_agent", () => {
expect(validateOverrides({ accept_language: "zh-CN" })).toMatchObject({
code: "invalid_params",
});
expect(validateOverrides({ user_agent_metadata: { platform: "Android" } })).toMatchObject({
code: "invalid_params",
});
});
it("rejects invalid max_touch_points", () => {
expect(validateOverrides({ touch: true, max_touch_points: 0 })).toMatchObject({
code: "invalid_params",
});
});
it("rejects an empty override set", () => {
expect(validateOverrides({})).toMatchObject({ code: "invalid_params" });
});
});
describe("toCdpUserAgentMetadata", () => {
it("maps snake_case fields to camelCase and drops absent ones", () => {
expect(
toCdpUserAgentMetadata({
brands: [{ brand: "Chromium", version: "120" }],
platform: "Android",
mobile: true,
}),
).toEqual({
brands: [{ brand: "Chromium", version: "120" }],
platform: "Android",
mobile: true,
});
expect(toCdpUserAgentMetadata({})).toEqual({});
});
});
@@ -284,6 +284,9 @@ describe("handleEvaluate", () => {
{ session_id: "aa11", expression: "1+1", tab_id: 11 },
{ cdp: fake.cdp, tabsApi: fake.tabsApi },
);
expect(res).toMatchObject({ code: "permission_denied", data: { reason: "agent_window_scope" } });
expect(res).toMatchObject({
code: "permission_denied",
data: { reason: "agent_window_scope" },
});
});
});
@@ -1,7 +1,53 @@
import { describe, expect, it, vi } from "vitest";
import { afterEach, describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import type { RequestHelpParams } from "@/transport/types";
import { handleRequestHelp, type RequestHelpDeps } from "../human-loop";
import { handleRequestHelp, type RequestHelpDeps, resetHelpLifecycleForTests } from "../human-loop";
function chromeEvent<T extends (...args: never[]) => unknown>() {
const listeners = new Set<T>();
return {
addListener: vi.fn((listener: T) => {
listeners.add(listener);
}),
removeListener: vi.fn((listener: T) => {
listeners.delete(listener);
}),
emit: (...args: Parameters<T>) => {
for (const listener of [...listeners]) listener(...args);
},
};
}
function installHelpLifecycleChrome() {
const runtimeOnMessage =
chromeEvent<
(
message: unknown,
sender: chrome.runtime.MessageSender,
sendResponse: (response: unknown) => void,
) => unknown
>();
const tabsOnActivated = chromeEvent<(activeInfo: chrome.tabs.TabActiveInfo) => unknown>();
const tabsOnCreated = chromeEvent<(tab: chrome.tabs.Tab) => unknown>();
const tabsOnUpdated =
chromeEvent<(tabId: number, changeInfo: chrome.tabs.TabChangeInfo) => unknown>();
const webNavigationOnCompleted =
chromeEvent<(details: chrome.webNavigation.WebNavigationFramedCallbackDetails) => unknown>();
vi.stubGlobal("chrome", {
runtime: { onMessage: runtimeOnMessage },
tabs: { onActivated: tabsOnActivated, onCreated: tabsOnCreated, onUpdated: tabsOnUpdated },
webNavigation: { onCompleted: webNavigationOnCompleted },
});
return {
runtimeOnMessage,
tabsOnActivated,
tabsOnCreated,
tabsOnUpdated,
webNavigationOnCompleted,
};
}
function fakeManager(sessionId: string, agentWindowId: number, tabId: number) {
const mgr = {
@@ -27,27 +73,64 @@ function baseDeps(over: Partial<RequestHelpDeps> = {}): RequestHelpDeps {
windows: { update: vi.fn(async () => ({}) as never) },
activateTab: vi.fn(async () => {}),
sendToTab: vi.fn(async () => ({ type: "bsk-help-response", outcome: "continued", note: "ok" })),
watchTabNavigation: () => () => {},
cdp: { send: vi.fn(async () => ({})) } as unknown as RequestHelpDeps["cdp"],
notifications: null,
autoAttachLifecycle: false,
...over,
};
}
describe("handleRequestHelp", () => {
afterEach(() => {
resetHelpLifecycleForTests();
vi.unstubAllGlobals();
});
it("rejects unknown session", async () => {
const res = await handleRequestHelp(fakeManager("abcd", 99, 5), baseParams({ session_id: "zzzz" }), baseDeps());
const res = await handleRequestHelp(
fakeManager("abcd", 99, 5),
baseParams({ session_id: "zzzz" }),
baseDeps(),
);
expect("code" in res && res.code).toBe("not_found");
});
it("brings the tab to the foreground and returns the user outcome", async () => {
const deps = baseDeps();
const res = await handleRequestHelp(fakeManager("abcd", 99, 5), baseParams({ tab_id: 5 }), deps);
const res = await handleRequestHelp(
fakeManager("abcd", 99, 5),
baseParams({ tab_id: 5 }),
deps,
);
expect(deps.windows.update).toHaveBeenCalledWith(99, { focused: true });
expect(deps.activateTab).toHaveBeenCalledWith(5);
expect(res).toMatchObject({ outcome: "continued", note: "ok", tab_id: 5 });
});
it("does not refresh observations after the user returns control", async () => {
const cdpSend = vi.fn(async () => ({}));
const deps = baseDeps({
cdp: { send: cdpSend } as unknown as RequestHelpDeps["cdp"],
});
const res = await handleRequestHelp(
fakeManager("abcd", 99, 5),
baseParams({ tab_id: 5 }),
deps,
);
expect(res).toMatchObject({ outcome: "continued", tab_id: 5 });
expect(cdpSend).not.toHaveBeenCalledWith(
expect.any(Number),
"Accessibility.getFullAXTree",
expect.anything(),
);
expect(cdpSend).not.toHaveBeenCalledWith(
expect.any(Number),
"DOMSnapshot.captureSnapshot",
expect.anything(),
);
});
it("forwards title into the help request message when provided", async () => {
const deps = baseDeps();
await handleRequestHelp(
@@ -70,18 +153,29 @@ describe("handleRequestHelp", () => {
expect(sentMsg.title).toBeUndefined();
});
it("returns navigated when the tab navigates during the wait", async () => {
const unwatch = vi.fn();
it("does not complete merely because the tab navigates during the wait", async () => {
vi.useFakeTimers();
const deps = baseDeps({
sendToTab: vi.fn(() => new Promise(() => {})),
watchTabNavigation: vi.fn((_tabId, cb) => {
queueMicrotask(() => cb());
return unwatch;
}),
});
const res = await handleRequestHelp(fakeManager("abcd", 99, 5), baseParams({ tab_id: 5 }), deps);
expect(res).toMatchObject({ outcome: "navigated", tab_id: 5 });
expect(unwatch).toHaveBeenCalled();
try {
const pending = handleRequestHelp(
fakeManager("abcd", 99, 5),
baseParams({ tab_id: 5, timeout_ms: 10 }),
deps,
);
await vi.advanceTimersByTimeAsync(5);
let settled = false;
void pending.then(() => {
settled = true;
});
await Promise.resolve();
expect(settled).toBe(false);
await vi.advanceTimersByTimeAsync(10);
await expect(pending).resolves.toMatchObject({ outcome: "timed_out", tab_id: 5 });
} finally {
vi.useRealTimers();
}
});
it("returns timed_out when the wait expires", async () => {
@@ -103,16 +197,177 @@ describe("handleRequestHelp", () => {
}
});
it("returns completed when explicit completion criteria match", async () => {
const deps = baseDeps({
sendToTab: vi.fn(async () => ({ type: "bsk-help-ack", ok: true })),
cdp: {
send: vi.fn(async (_tabId: number, method: string) => {
if (method === "Runtime.evaluate") return { result: { value: true } };
return {};
}),
} as unknown as RequestHelpDeps["cdp"],
});
const res = await handleRequestHelp(
fakeManager("abcd", 99, 5),
baseParams({
tab_id: 5,
completion_criteria: {
any: [{ selector_exists: "#account-menu" }],
stable_for_ms: 0,
},
}),
deps,
);
expect(res).toMatchObject({ outcome: "completed", completed_by: "system", tab_id: 5 });
});
it("keeps user control active on new tabs without moving completion off the primary tab", async () => {
vi.useFakeTimers();
const chromeEvents = installHelpLifecycleChrome();
const sendToTab = vi.fn(async () => ({ type: "bsk-help-ack", ok: true }));
const deps = baseDeps({
autoAttachLifecycle: undefined,
sendToTab,
tabsApi: {
get: vi.fn(async (tabId: number) => ({
id: tabId,
windowId: 99,
active: tabId === 6,
url: tabId === 6 ? "https://app.example/reset/success" : "https://app.example/login",
})) as never,
query: vi.fn(async () => [{ id: 5, windowId: 99, active: true }] as never),
},
});
try {
const pending = handleRequestHelp(
fakeManager("abcd", 99, 5),
baseParams({
tab_id: 5,
timeout_ms: 1_000,
completion_criteria: {
any: [{ url_contains: "/reset/success" }],
stable_for_ms: 0,
},
}),
deps,
);
await vi.waitFor(() => expect(sendToTab).toHaveBeenCalledWith(5, expect.anything()));
chromeEvents.tabsOnActivated.emit({ tabId: 6, windowId: 99 });
await vi.advanceTimersByTimeAsync(200);
await vi.waitFor(() =>
expect(sendToTab).toHaveBeenCalledWith(
6,
expect.objectContaining({
type: "bsk-help-request",
displayMode: "compact",
selectors: [],
}),
),
);
await vi.advanceTimersByTimeAsync(1_000);
await expect(pending).resolves.toMatchObject({
outcome: "timed_out",
tab_id: 5,
});
expect(sendToTab).toHaveBeenCalledWith(
6,
expect.objectContaining({ type: "bsk-help-cancel" }),
);
} finally {
vi.useRealTimers();
}
});
it("lets same-window tabs query active help after content-script load", async () => {
const chromeEvents = installHelpLifecycleChrome();
const ac = new AbortController();
const cdpSend = vi.fn(async () => ({ root: { nodeId: 1 }, nodeIds: [] }));
const deps = baseDeps({
autoAttachLifecycle: undefined,
signal: ac.signal,
sendToTab: vi.fn(async () => ({ type: "bsk-help-ack", ok: true })),
cdp: {
send: cdpSend,
} as unknown as RequestHelpDeps["cdp"],
});
const pending = handleRequestHelp(
fakeManager("abcd", 99, 5),
baseParams({ tab_id: 5, timeout_ms: 60_000 }),
deps,
);
await vi.waitFor(() => expect(deps.sendToTab).toHaveBeenCalledWith(5, expect.anything()));
const sameWindowResponse = vi.fn();
chromeEvents.runtimeOnMessage.emit(
{ type: "bsk-help-query" },
{ tab: { id: 6 } as chrome.tabs.Tab },
sameWindowResponse,
);
await vi.waitFor(() =>
expect(sameWindowResponse).toHaveBeenCalledWith({
active: true,
request: expect.objectContaining({
requestId: expect.any(String),
prompt: "log in",
displayMode: "compact",
selectors: [],
}),
}),
);
const subjectResponse = vi.fn();
chromeEvents.runtimeOnMessage.emit(
{ type: "bsk-help-query" },
{ tab: { id: 5 } as chrome.tabs.Tab },
subjectResponse,
);
await vi.waitFor(() =>
expect(subjectResponse).toHaveBeenCalledWith({
active: true,
request: expect.objectContaining({
requestId: expect.any(String),
prompt: "log in",
displayMode: "full",
}),
}),
);
ac.abort();
await expect(pending).resolves.toMatchObject({ code: "cancelled" });
expect(cdpSend).toHaveBeenCalledWith(
5,
"DOM.querySelectorAll",
expect.objectContaining({ selector: "[data-bsk-help]" }),
);
expect(cdpSend).toHaveBeenCalledWith(
6,
"DOM.querySelectorAll",
expect.objectContaining({ selector: "[data-bsk-help]" }),
);
});
it("tags ref targets via CDP and reports them matched", async () => {
const mgr = {
get: (id: string) =>
id === "abcd"
? { sessionId: "abcd", agentWindowId: 99, refStore: { resolve: () => 42 }, borrowedTabs: new Map() }
? {
sessionId: "abcd",
agentWindowId: 99,
refStore: { resolve: () => 42 },
borrowedTabs: new Map(),
}
: null,
findByWindowId: (wid: number) => (wid === 99 ? { sessionId: "abcd" } : null),
} as unknown as SessionManager;
const deps = baseDeps({
cdp: { send: vi.fn(async () => ({ object: { objectId: "obj-1" } })) } as unknown as RequestHelpDeps["cdp"],
cdp: {
send: vi.fn(async () => ({ object: { objectId: "obj-1" } })),
} as unknown as RequestHelpDeps["cdp"],
});
const res = await handleRequestHelp(
mgr,
@@ -171,34 +426,40 @@ describe("handleRequestHelp", () => {
expect("code" in res).toBe(false);
});
it("returns protocol_error when sendToTab rejects", async () => {
it("keeps waiting when the initial overlay delivery rejects", async () => {
vi.useFakeTimers();
const deps = baseDeps({
sendToTab: vi.fn(async () => {
throw new Error("no receiver");
}),
});
const res = await handleRequestHelp(fakeManager("abcd", 99, 5), baseParams({ tab_id: 5 }), deps);
expect("code" in res && res.code).toBe("protocol_error");
try {
const res = handleRequestHelp(
fakeManager("abcd", 99, 5),
baseParams({ tab_id: 5, timeout_ms: 10 }),
deps,
);
await vi.advanceTimersByTimeAsync(20);
await expect(res).resolves.toMatchObject({ outcome: "timed_out" });
} finally {
vi.useRealTimers();
}
});
it("returns protocol_error for a malformed help response", async () => {
const undefinedReply = baseDeps({ sendToTab: vi.fn(async () => undefined) });
const res1 = await handleRequestHelp(
fakeManager("abcd", 99, 5),
baseParams({ tab_id: 5 }),
undefinedReply,
);
expect("code" in res1 && res1.code).toBe("protocol_error");
const badOutcome = baseDeps({
sendToTab: vi.fn(async () => ({ type: "bsk-help-response", outcome: "weird" })),
});
const res2 = await handleRequestHelp(
fakeManager("abcd", 99, 5),
baseParams({ tab_id: 5 }),
badOutcome,
);
expect("code" in res2 && res2.code).toBe("protocol_error");
it("keeps waiting for explicit completion after a malformed help response", async () => {
vi.useFakeTimers();
const deps = baseDeps({ sendToTab: vi.fn(async () => undefined) });
try {
const res = handleRequestHelp(
fakeManager("abcd", 99, 5),
baseParams({ tab_id: 5, timeout_ms: 10 }),
deps,
);
await vi.advanceTimersByTimeAsync(20);
await expect(res).resolves.toMatchObject({ outcome: "timed_out" });
} finally {
vi.useRealTimers();
}
});
it("reports selector match status from CDP", async () => {
@@ -254,7 +515,11 @@ describe("handleRequestHelp", () => {
it("returns cancelled when the signal aborts", async () => {
const ac = new AbortController();
const deps = baseDeps({ signal: ac.signal, sendToTab: vi.fn(() => new Promise(() => {})) });
const p = handleRequestHelp(fakeManager("abcd", 99, 5), baseParams({ tab_id: 5, timeout_ms: 60_000 }), deps);
const p = handleRequestHelp(
fakeManager("abcd", 99, 5),
baseParams({ tab_id: 5, timeout_ms: 60_000 }),
deps,
);
ac.abort();
const res = await p;
expect("code" in res ? res.code : (res as { outcome: string }).outcome).toMatch(/cancelled/);
@@ -982,4 +982,3 @@ describe("handleSelect", () => {
expect(fake.sent.some((c) => c.method === "Runtime.callFunctionOn")).toBe(false);
});
});
@@ -2,12 +2,12 @@ import { describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import type { CdpRunner } from "@/tools/shared";
import {
eventLoaderIsRelevant,
handleNavigate,
handleNavigateBack,
handleNavigateForward,
handleReload,
lifecycleAlreadyReached,
eventLoaderIsRelevant,
lifecycleMeetsOrExceeds,
shouldTrustReadyStateProbe,
} from "../navigation";
@@ -60,10 +60,7 @@ function makeFakeCdp(opts?: {
};
const events: EventListener[] = [];
const sent: Array<{ tabId: number; method: string; params?: object }> = [];
const methodHandlers: Record<
string,
(tabId: number, params?: object) => object
> = {
const methodHandlers: Record<string, (tabId: number, params?: object) => object> = {
"Page.enable": () => ({}),
"Page.setLifecycleEventsEnabled": () => ({}),
"Page.navigate": () => {
@@ -72,9 +69,7 @@ function makeFakeCdp(opts?: {
name: opts.fireLifecycleDuringNavigate,
frameId: opts.fireLifecycleDuringNavigateFrameId ?? opts.navigateFrameId ?? "frame-1",
loaderId:
opts.fireLifecycleDuringNavigateLoaderId ??
opts.navigateLoaderId ??
"loader-after",
opts.fireLifecycleDuringNavigateLoaderId ?? opts.navigateLoaderId ?? "loader-after",
};
for (const listener of [...events]) {
listener({ tabId: 4 }, "Page.lifecycleEvent", payload);
@@ -208,9 +203,7 @@ describe("lifecycleMeetsOrExceeds", () => {
describe("eventLoaderIsRelevant", () => {
it("rejects events from the pre-navigation loader", () => {
expect(
eventLoaderIsRelevant("loader-before", { beforeLoaderId: "loader-before" }),
).toBe(false);
expect(eventLoaderIsRelevant("loader-before", { beforeLoaderId: "loader-before" })).toBe(false);
expect(
eventLoaderIsRelevant("loader-after", {
beforeLoaderId: "loader-before",
@@ -0,0 +1,128 @@
import { describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import type { NetworkResult } from "@/transport/types";
import { handleNetwork, type NetworkCdpRunner } from "../network";
function fakeAgentWindow(ids: number[]) {
let i = 0;
return {
create: vi.fn(async () => {
const id = ids[i++];
if (id === undefined) throw new Error("ran out of fake ids");
return id;
}),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
};
}
function makeDeps(
opts: {
get?: (tabId: number) => Promise<chrome.tabs.Tab>;
query?: (query: chrome.tabs.QueryInfo) => Promise<chrome.tabs.Tab[]>;
result?: NetworkResult;
captureError?: Error;
} = {},
) {
const result =
opts.result ??
({
tab_id: 7,
entries: [],
next_since: 0,
truncated: false,
} satisfies NetworkResult);
const ensureNetworkCapture = vi.fn(async () => {
if (opts.captureError) throw opts.captureError;
});
const networkEntriesSince = vi.fn(() => result);
const trackSessionTab = vi.fn();
const cdp = {
trackSessionTab,
ensureNetworkCapture,
networkEntriesSince,
} satisfies NetworkCdpRunner;
const tabsApi = {
get:
opts.get ??
vi.fn(
async (tabId: number) => ({ id: tabId, windowId: 100, active: true }) as chrome.tabs.Tab,
),
query:
opts.query ?? vi.fn(async () => [{ id: 7, windowId: 100, active: true } as chrome.tabs.Tab]),
};
return { cdp, tabsApi, ensureNetworkCapture, networkEntriesSince, trackSessionTab };
}
describe("handleNetwork", () => {
it("reads the Agent Window active tab with safe defaults", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
const deps = makeDeps({
result: {
tab_id: 7,
entries: [
{ sequence: 3, kind: "response", url: "https://x/api", status: 404, truncated: false },
],
next_since: 3,
truncated: false,
},
});
const res = await handleNetwork(sm, { session_id: "aa11" }, deps);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.entries[0].status).toBe(404);
expect(deps.ensureNetworkCapture).toHaveBeenCalledWith(7);
expect(deps.networkEntriesSince).toHaveBeenCalledWith(7, undefined, 50, 1000);
});
it("reads an explicit tab and forwards cursor options", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
const deps = makeDeps({
get: vi.fn(async () => ({ id: 9, windowId: 200, active: true }) as chrome.tabs.Tab),
});
await handleNetwork(
sm,
{
session_id: "aa11",
tab_id: 9,
since: 12,
limit: 75,
max_text_chars: 1500,
},
deps,
);
expect(deps.ensureNetworkCapture).toHaveBeenCalledWith(9);
expect(deps.networkEntriesSince).toHaveBeenCalledWith(9, 12, 75, 1500);
});
it("surfaces network capture failures as cdp_failed", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
const deps = makeDeps({ captureError: new Error("Network.enable denied") });
const res = await handleNetwork(sm, { session_id: "aa11" }, deps);
expect(res).toEqual({ code: "cdp_failed", message: "Network.enable denied" });
expect(deps.trackSessionTab).toHaveBeenCalledWith("aa11", 7);
expect(deps.networkEntriesSince).not.toHaveBeenCalled();
});
it("hides other sessions' Agent Window tabs", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100, 101]) });
await sm.start("aa11");
await sm.start("bb22");
const deps = makeDeps({
get: vi.fn(async () => ({ id: 9, windowId: 101, active: true }) as chrome.tabs.Tab),
});
const res = await handleNetwork(sm, { session_id: "aa11", tab_id: 9 }, deps);
expect(res).toMatchObject({ code: "not_found" });
expect(deps.ensureNetworkCapture).not.toHaveBeenCalled();
});
});
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,59 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import {
ensureBrowserObservationListeners,
isBrowserObservationAttachedForTests,
RECORD_DEFAULT_START_URL,
releaseBrowserObservationListenersIfIdle,
resetBrowserObservationForTests,
setBrowserObservationAttachForTests,
} from "../record";
describe("record defaults", () => {
it("uses example.com as the default injectable start URL", () => {
expect(RECORD_DEFAULT_START_URL).toBe("https://example.com/");
});
});
describe("browser observation lifecycle", () => {
afterEach(() => {
resetBrowserObservationForTests();
});
it("does not attach listeners until ensure is called", () => {
const tab = vi.fn(() => () => undefined);
const nav = vi.fn(() => () => undefined);
setBrowserObservationAttachForTests(tab, nav);
expect(isBrowserObservationAttachedForTests()).toBe(false);
expect(tab).not.toHaveBeenCalled();
expect(nav).not.toHaveBeenCalled();
});
it("attaches once on ensure and detaches when idle", () => {
const detachTab = vi.fn();
const detachNav = vi.fn();
const tab = vi.fn(() => detachTab);
const nav = vi.fn(() => detachNav);
setBrowserObservationAttachForTests(tab, nav);
ensureBrowserObservationListeners({
tabsApi: { get: vi.fn(), query: vi.fn(), create: vi.fn() } as never,
sendToTab: vi.fn(),
});
expect(isBrowserObservationAttachedForTests()).toBe(true);
expect(tab).toHaveBeenCalledTimes(1);
expect(nav).toHaveBeenCalledTimes(1);
// Idempotent.
ensureBrowserObservationListeners({
tabsApi: { get: vi.fn(), query: vi.fn(), create: vi.fn() } as never,
sendToTab: vi.fn(),
});
expect(tab).toHaveBeenCalledTimes(1);
releaseBrowserObservationListenersIfIdle();
expect(isBrowserObservationAttachedForTests()).toBe(false);
expect(detachTab).toHaveBeenCalledTimes(1);
expect(detachNav).toHaveBeenCalledTimes(1);
});
});
@@ -139,9 +139,13 @@ describe("handleSessionStop with auto-return", () => {
resetAgentOverlays: vi.fn(async () => {}),
} satisfies AgentOverlayResetApi;
const res = await handleSessionStop(sm, { session_id: "aa11" }, {
tabManagement: { tabs, windows, agentOverlayReset },
});
const res = await handleSessionStop(
sm,
{ session_id: "aa11" },
{
tabManagement: { tabs, windows, agentOverlayReset },
},
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(agentOverlayReset.resetAgentOverlays).toHaveBeenCalledWith(7, "aa11");
@@ -1,6 +1,12 @@
import { describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import { lookupSession } from "../shared";
import {
cdpBlockedUrlReason,
enforceCdpAccessibleTarget,
lookupSession,
parseBufferedReadBounds,
resolveCdpAccessibleTargetTab,
} from "../shared";
function fakeAgentWindow() {
return {
@@ -20,9 +26,7 @@ describe("lookupSession", () => {
expect(lookupSession(sm, { session_id: "" }, "tool.test")).toMatchObject({
code: "invalid_params",
});
expect(
lookupSession(sm, { session_id: 42 as unknown as string }, "tool.test"),
).toMatchObject({
expect(lookupSession(sm, { session_id: 42 as unknown as string }, "tool.test")).toMatchObject({
code: "invalid_params",
});
});
@@ -42,3 +46,128 @@ describe("lookupSession", () => {
expect(result).toBe(ctx);
});
});
describe("parseBufferedReadBounds", () => {
it("applies shared defaults and caps", () => {
expect(parseBufferedReadBounds({})).toEqual({
since: undefined,
limit: 50,
maxTextChars: 1000,
});
expect(parseBufferedReadBounds({ since: 12, limit: 500, max_text_chars: 9999 })).toEqual({
since: 12,
limit: 200,
maxTextChars: 4096,
});
});
it.each([
[{ since: -1 }, "since"],
[{ since: 1.5 }, "since"],
[{ limit: 0 }, "limit"],
[{ max_text_chars: 0 }, "max_text_chars"],
])("rejects invalid bounds in %o", (params, field) => {
expect(parseBufferedReadBounds(params)).toMatchObject({
code: "invalid_params",
message: expect.stringContaining(field),
});
});
});
describe("CDP target URL guard", () => {
it("allows ordinary pages and about:blank", () => {
expect(cdpBlockedUrlReason("https://example.test/path")).toBeNull();
expect(cdpBlockedUrlReason("http://example.test/path")).toBeNull();
expect(cdpBlockedUrlReason("about:blank")).toBeNull();
});
it("blocks browser and extension internal pages", () => {
expect(cdpBlockedUrlReason("chrome-extension://abc/options.html")).toBe("chrome-extension:");
expect(cdpBlockedUrlReason("chrome://extensions")).toBe("chrome:");
expect(cdpBlockedUrlReason("about:newtab")).toBe("about:");
});
it("returns a structured permission error before page CDP access", () => {
expect(
enforceCdpAccessibleTarget(
{
tabId: 7,
windowId: 100,
active: true,
url: "chrome-extension://other-extension/page.html",
},
"snapshot",
),
).toMatchObject({
code: "permission_denied",
data: { reason: "restricted_tab_url" },
message: expect.stringContaining("snapshot cannot access tab 7"),
});
});
it("falls back from a restricted default active tab to an accessible agent tab", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow() });
const ctx = await sm.start("aa11");
const tabsApi = {
get: vi.fn(),
query: vi.fn(async (query: chrome.tabs.QueryInfo) => {
if (query.active) {
return [
{
id: 7,
windowId: ctx.agentWindowId,
active: true,
url: "chrome-extension://other-extension/page.html",
} as chrome.tabs.Tab,
];
}
return [
{
id: 7,
windowId: ctx.agentWindowId,
active: true,
url: "chrome-extension://other-extension/page.html",
} as chrome.tabs.Tab,
{
id: 8,
windowId: ctx.agentWindowId,
active: false,
url: "https://example.test/",
} as chrome.tabs.Tab,
];
}),
};
await expect(
resolveCdpAccessibleTargetTab(sm, ctx, undefined, tabsApi, "snapshot"),
).resolves.toMatchObject({
tabId: 8,
url: "https://example.test/",
});
});
it("does not replace an explicitly requested restricted tab", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow() });
const ctx = await sm.start("aa11");
const tabsApi = {
get: vi.fn(
async () =>
({
id: 7,
windowId: ctx.agentWindowId,
active: true,
url: "chrome-extension://other-extension/page.html",
}) as chrome.tabs.Tab,
),
query: vi.fn(),
};
await expect(
resolveCdpAccessibleTargetTab(sm, ctx, 7, tabsApi, "snapshot"),
).resolves.toMatchObject({
code: "permission_denied",
data: { reason: "restricted_tab_url" },
});
expect(tabsApi.query).not.toHaveBeenCalled();
});
});
@@ -292,7 +292,10 @@ describe("handleTabClose", () => {
};
const { api } = makeTabMutationApi(state);
const res = await handleTabClose(sm, { session_id: "aa11", tab_id: 9 }, { tabs: api });
expect(res).toMatchObject({ code: "permission_denied", data: { reason: "agent_window_scope" } });
expect(res).toMatchObject({
code: "permission_denied",
data: { reason: "agent_window_scope" },
});
});
it("rejects when another session has borrowed the tab", async () => {
@@ -338,7 +341,10 @@ describe("handleTabSelect", () => {
};
const { api } = makeTabMutationApi(state);
const res = await handleTabSelect(sm, { session_id: "aa11", tab_id: 3 }, { tabs: api });
expect(res).toMatchObject({ code: "permission_denied", data: { reason: "agent_window_scope" } });
expect(res).toMatchObject({
code: "permission_denied",
data: { reason: "agent_window_scope" },
});
});
it("rejects a borrowed tab that is no longer in the Agent Window", async () => {
@@ -352,7 +358,10 @@ describe("handleTabSelect", () => {
};
const { api } = makeTabMutationApi(state);
const res = await handleTabSelect(sm, { session_id: "aa11", tab_id: 3 }, { tabs: api });
expect(res).toMatchObject({ code: "permission_denied", data: { reason: "agent_window_scope" } });
expect(res).toMatchObject({
code: "permission_denied",
data: { reason: "agent_window_scope" },
});
});
});
@@ -444,7 +453,10 @@ describe("handleTabBorrow", () => {
};
const { api } = makeTabMutationApi(state);
const res = await handleTabBorrow(sm, { session_id: "aa11", tab_id: 7 }, { tabs: api });
expect(res).toMatchObject({ code: "permission_denied", data: { reason: "agent_window_scope" } });
expect(res).toMatchObject({
code: "permission_denied",
data: { reason: "agent_window_scope" },
});
});
it("returns cancelled when the approver declines", async () => {
@@ -613,6 +625,65 @@ describe("handleTabReturn", () => {
expect(winSpies.create).toHaveBeenCalledOnce();
});
it("does not fall back into another session's Agent Window (creates a new window instead)", async () => {
// Regression: a returned borrowed tab whose original window is gone
// must never be parked in *another* session's Agent Window. Agent
// Windows are `type: "normal"`, so getLastFocused({windowTypes:
// ["normal"]}) can return session bb22's window (200). Without the
// isAgentWindowId guard the tab would be moved into window 200,
// letting bb22 write to it and destroying it when bb22 stops.
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100, 200]) });
const ctx = await sm.start("aa11");
await sm.start("bb22"); // owns Agent Window 200
ctx.borrowedTabs.set(7, { tabId: 7, originalWindowId: 300, originalIndex: 4 });
const state: FakeTabState = {
tabs: new Map([[7, { id: 7, windowId: 100 } as chrome.tabs.Tab]]),
nextTabId: 50,
windowsClosed: new Set([300]),
};
const { api, spies } = makeTabMutationApi(state);
const { api: windowsApi, spies: winSpies } = makeWindowsApi(state, {
lastFocused: 200, // session bb22's Agent Window
createWindowId: 777,
});
const res = await handleTabReturn(
sm,
{ session_id: "aa11", tab_id: 7 },
{ tabs: api, windows: windowsApi },
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.returned_to_window_id).toBe(777);
expect(res.fallback).toBe(true);
expect(winSpies.create).toHaveBeenCalledOnce();
expect(spies.move).not.toHaveBeenCalledWith(7, expect.objectContaining({ windowId: 200 }));
});
it("returnBorrowedTab honours an explicit isAgentWindowId predicate", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
ctx.borrowedTabs.set(7, { tabId: 7, originalWindowId: 300, originalIndex: 4 });
const state: FakeTabState = {
tabs: new Map([[7, { id: 7, windowId: 100 } as chrome.tabs.Tab]]),
nextTabId: 50,
windowsClosed: new Set([300]),
};
const { api, spies } = makeTabMutationApi(state);
const { api: windowsApi, spies: winSpies } = makeWindowsApi(state, {
lastFocused: 555,
createWindowId: 777,
});
const res = await returnBorrowedTab(ctx, 7, {
tabs: api,
windows: windowsApi,
isAgentWindowId: (windowId) => windowId === 555,
});
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.toWindowId).toBe(777);
expect(res.fallback).toBe(true);
expect(winSpies.create).toHaveBeenCalledOnce();
expect(spies.move).not.toHaveBeenCalledWith(7, expect.objectContaining({ windowId: 555 }));
});
it("returns not_found when the tab is not borrowed by this session", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
+138
View File
@@ -0,0 +1,138 @@
import { describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import { handleSessionStart } from "../session";
import { handleWindowResize, type WindowResizeApi } from "../window";
function fakeAgentWindow(ids: number[]) {
let i = 0;
const create = vi.fn(async () => {
const id = ids[i++];
if (id === undefined) throw new Error("ran out of fake ids");
return id;
});
const remove = vi.fn(async () => {});
const ensureActiveTab = vi.fn(async () => {});
return { create, remove, ensureActiveTab };
}
async function makeManager(): Promise<SessionManager> {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
return sm;
}
function fakeUpdateApi(opts?: { throw?: boolean }) {
const calls: Array<{ windowId: number; width: number; height: number }> = [];
const api: WindowResizeApi = {
update: vi.fn(async (windowId: number, updateInfo: { width: number; height: number }) => {
if (opts?.throw) throw new Error("simulated chrome failure");
calls.push({ windowId, width: updateInfo.width, height: updateInfo.height });
return undefined;
}),
};
return { api, calls };
}
describe("handleWindowResize", () => {
it("resizes the session's Agent Window", async () => {
const sm = await makeManager();
const { api, calls } = fakeUpdateApi();
const result = await handleWindowResize(
sm,
{ session_id: "aa11", width: 1280, height: 800 },
api,
);
expect(result).toEqual({ window_id: 100, width: 1280, height: 800 });
expect(calls).toEqual([{ windowId: 100, width: 1280, height: 800 }]);
});
it("rejects an unknown session", async () => {
const sm = await makeManager();
const { api } = fakeUpdateApi();
const result = await handleWindowResize(
sm,
{ session_id: "zz99", width: 1280, height: 800 },
api,
);
expect(result).toMatchObject({ code: "not_found" });
});
it("rejects missing session_id", async () => {
const sm = await makeManager();
const { api } = fakeUpdateApi();
const result = await handleWindowResize(sm, { session_id: "", width: 1280, height: 800 }, api);
expect(result).toMatchObject({ code: "invalid_params" });
});
it("rejects missing dimensions", async () => {
const sm = await makeManager();
const { api, calls } = fakeUpdateApi();
const result = await handleWindowResize(
sm,
{ session_id: "aa11" } as unknown as { session_id: string; width: number; height: number },
api,
);
expect(result).toMatchObject({ code: "invalid_params" });
expect(calls).toEqual([]);
});
it.each([
[99, 800],
[100, 7681],
[1280.5, 800],
[Number.NaN, 800],
])("rejects out-of-range or non-integer dimensions (%s, %s)", async (width, height) => {
const sm = await makeManager();
const { api, calls } = fakeUpdateApi();
const result = await handleWindowResize(sm, { session_id: "aa11", width, height }, api);
expect(result).toMatchObject({ code: "invalid_params" });
expect(calls).toEqual([]);
});
it("maps chrome API failures to protocol_error", async () => {
const sm = await makeManager();
const { api } = fakeUpdateApi({ throw: true });
const result = await handleWindowResize(
sm,
{ session_id: "aa11", width: 1280, height: 800 },
api,
);
expect(result).toMatchObject({ code: "protocol_error", message: "simulated chrome failure" });
});
});
describe("handleSessionStart window size", () => {
it("passes width/height through to Agent Window creation", async () => {
const aw = fakeAgentWindow([100]);
const sm = new SessionManager({ agentWindow: aw });
const result = await handleSessionStart(sm, { session_id: "aa11", width: 1280, height: 800 });
expect(result).toEqual({ agent_window_id: 100 });
expect(aw.create).toHaveBeenCalledWith("about:blank", { width: 1280, height: 800 });
});
it("creates the window without size when width/height are omitted", async () => {
const aw = fakeAgentWindow([100]);
const sm = new SessionManager({ agentWindow: aw });
const result = await handleSessionStart(sm, { session_id: "aa11" });
expect(result).toEqual({ agent_window_id: 100 });
expect(aw.create).toHaveBeenCalledWith("about:blank");
});
it("rejects a lone width without height", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const result = await handleSessionStart(sm, { session_id: "aa11", width: 1280 });
expect(result).toMatchObject({ code: "invalid_params" });
expect(sm.has("aa11")).toBe(false);
});
it.each([
[99, 800],
[1280, 7681],
[1280.5, 800],
])("rejects invalid dimensions (%s, %s)", async (width, height) => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const result = await handleSessionStart(sm, { session_id: "aa11", width, height });
expect(result).toMatchObject({ code: "invalid_params" });
expect(sm.has("aa11")).toBe(false);
});
});
+53 -50
View File
@@ -28,8 +28,9 @@ export const CONFIRMATION_TIMEOUT_MS = 5000;
const EXIT_ANIMATION_MS = 150;
/** Matches BorrowConfirmationOverlay progress ring/bar transition (duration-1000). */
export const PROGRESS_TRANSITION_MS = 1000;
// UI auto-allows after countdown + progress transition + exit fade. Background
// timeout must not fire before that chain completes or deny is ignored.
// UI auto-denies after countdown + progress transition + exit fade. Background
// timeout must not fire before that chain completes or the UI decision races
// with an already-settled deny.
export const BACKGROUND_TIMEOUT_MS =
CONFIRMATION_TIMEOUT_MS + PROGRESS_TRANSITION_MS + EXIT_ANIMATION_MS + 500;
@@ -114,9 +115,8 @@ const pendingBorrowNotifications = new Map<string, PendingBorrowNotification>();
*
* This is the *fallback* authorization path that fires when no candidate
* user window could host the in-page overlay (every `chrome.tabs.sendMessage`
* was rejected because the content script was not present). Without it the
* background-task fail-open at the end of `tryCandidate` would silently
* allow the borrow without any user-visible authorization step.
* was rejected because the content script was not present). It preserves an
* explicit authorization path without weakening the fail-closed timeout.
*/
const pendingBorrowDecisions = new Map<string, (allowed: boolean) => void>();
@@ -170,7 +170,7 @@ export function attachBorrowNotificationClickHandler(deps: {
* Allow / Deny buttons on the OS notification can resolve the matching
* pending `requestBorrowConfirmation`. This is the explicit-authorization
* fallback path used when every candidate user window's content script was
* missing — without it we'd be back to silently fail-opening the borrow.
* missing.
*
* Button index convention (matches `BorrowNotificationCopy`):
* - 0 → Allow → `resolve(true)`
@@ -326,8 +326,7 @@ async function listConfirmationCandidates(
if (typeof w.id !== "number") continue;
// Never bounce the overlay back into a session's Agent Window — those
// pages can't host a content script when they're on the about:blank
// bootstrap URL, so sendMessage would fail-open and silently allow the
// borrow with no UI shown.
// bootstrap URL, so they cannot surface an authorization decision.
if (isAgentWindowId(w.id)) continue;
const activeTab = w.tabs?.find((t) => t.active === true) ?? null;
if (!activeTab || typeof activeTab.id !== "number") continue;
@@ -360,16 +359,14 @@ async function listConfirmationCandidates(
* *explicit-authorization fallback* when every candidate fails sendMessage:
* we no longer silently allow the borrow in that case, the promise stays
* pending until the user clicks a notification button (resolves true/false)
* or `BACKGROUND_TIMEOUT_MS` fires (last-resort fail-open so the agent
* doesn't hang forever when even the notification path is broken).
* or `BACKGROUND_TIMEOUT_MS` fires and safely denies the request.
*
* Clicking the notification body (vs. its buttons) focuses the chosen user
* window so the overlay becomes visible — the previous behaviour.
*
* Returns `true` on user-allow (overlay or notification) OR last-resort
* fail-open (no candidate window at all; abort signal; timeout). Returns
* `false` only on explicit user-deny (overlay button or notification Deny
* button).
* Returns `true` only on an explicit user allow from the overlay or
* notification. Missing UI, aborts, malformed responses, and timeouts all
* fail closed and return `false`.
*/
export async function requestBorrowConfirmation(
tabId: number,
@@ -383,24 +380,29 @@ export async function requestBorrowConfirmation(
const isAgentWindowId = deps.isAgentWindowId ?? (() => false);
const copy = deps.notificationCopy ?? DEFAULT_NOTIFICATION_COPY;
let tab: chrome.tabs.Tab;
// The borrow pipeline already fetched this tab moments earlier
// (validateBorrowTarget in tabs.ts), so a failure here is a transient
// SW/page hiccup, not a missing tab. Fall back to a generic title and still
// surface the overlay + notification so the user gets a real chance to
// approve or deny.
let tabTitle: string;
try {
tab = await tabsApi.get(tabId);
const tab = await tabsApi.get(tabId);
tabTitle = tab.title ?? tab.url ?? String(tabId);
} catch (err) {
console.warn("[bsk borrow] could not get tab info — proceeding", {
console.warn("[bsk borrow] could not get tab info — using fallback title", {
tabId,
error: err instanceof Error ? err.message : String(err),
});
return true;
tabTitle = String(tabId);
}
const tabTitle = tab.title ?? tab.url ?? String(tabId);
const candidates = await listConfirmationCandidates(windowsApi, isAgentWindowId);
if (candidates.length === 0) {
console.warn("[bsk borrow] no injectable user window available — proceeding without overlay", {
console.warn("[bsk borrow] no injectable user window available — denying borrow", {
tabId,
});
return true;
return false;
}
const requestId = createBorrowRequestId(tabId);
@@ -411,7 +413,7 @@ export async function requestBorrowConfirmation(
// overlay immediately.
const notificationAnchor = candidates[0];
if (!notificationAnchor) {
return true;
return false;
}
return new Promise<boolean>((resolve) => {
@@ -440,22 +442,25 @@ export async function requestBorrowConfirmation(
resolve(allowed);
};
const onAbort = () => {
if (activeMessageTabId !== null) {
const cancelMessage: BorrowCancelMessage = {
type: "borrow-cancel",
const dismissPendingOverlay = () => {
if (activeMessageTabId === null) return;
const cancelMessage: BorrowCancelMessage = {
type: "borrow-cancel",
requestId,
};
void Promise.resolve(tabsApi.sendMessage(activeMessageTabId, cancelMessage)).catch((err) => {
console.debug("[bsk borrow] cancel message failed", {
tabId,
messageTabId: activeMessageTabId,
requestId,
};
void tabsApi.sendMessage(activeMessageTabId, cancelMessage).catch((err) => {
console.debug("[bsk borrow] cancel message failed", {
tabId,
messageTabId: activeMessageTabId,
requestId,
error: err instanceof Error ? err.message : String(err),
});
error: err instanceof Error ? err.message : String(err),
});
}
settle(true);
});
};
const onAbort = () => {
dismissPendingOverlay();
settle(false);
};
if (signal?.aborted) {
@@ -465,19 +470,18 @@ export async function requestBorrowConfirmation(
signal?.addEventListener("abort", onAbort, { once: true });
timeout = setTimeout(() => {
console.info("[bsk borrow] confirmation timed out — proceeding", {
console.info("[bsk borrow] confirmation timed out — denying borrow", {
tabId,
messageTabId: activeMessageTabId,
});
settle(true);
dismissPendingOverlay();
settle(false);
}, BACKGROUND_TIMEOUT_MS);
// Surface the OS notification *before* messaging any candidate so the
// user has a parallel signal even if every content script is missing.
// The Allow / Deny buttons let the user authorize explicitly when the
// in-page overlay never reached them — without those buttons we'd have
// to fail-open after `tryCandidate` exhausts its candidates, which is
// exactly the silent-allow bug we're fixing here.
// in-page overlay never reached them.
if (notificationsApi) {
pendingBorrowNotifications.set(notificationId, {
windowId: notificationAnchor.windowId,
@@ -491,7 +495,7 @@ export async function requestBorrowConfirmation(
title: copy.title,
message: copy.body(tabTitle),
priority: 2,
requireInteraction: false,
requireInteraction: true,
silent: false,
buttons: [{ title: copy.allowButton }, { title: copy.denyButton }],
})
@@ -515,15 +519,12 @@ export async function requestBorrowConfirmation(
): void => {
if (settled) return;
if (index >= candidates.length) {
// Every candidate rejected sendMessage. We deliberately do NOT
// fail-open here anymore: silently allowing a borrow with zero
// user-visible authorization UI is the bug. Instead we keep the
// promise pending and rely on:
// Every candidate rejected sendMessage. Keep the promise pending and
// rely on:
// • the OS notification's Allow / Deny buttons for an explicit
// user choice (preferred), or
// • the BACKGROUND_TIMEOUT_MS fail-open as the last-resort
// soft-fail so the agent doesn't hang forever when even the
// notification API is unavailable.
// • the BACKGROUND_TIMEOUT_MS fail-closed result so the agent does
// not hang forever when even the notification API is unavailable.
// Keep the diagnostics so real-world frequency is still observable.
console.warn(
"[bsk borrow] every candidate user window failed sendMessage — awaiting notification button or timeout",
@@ -552,7 +553,9 @@ export async function requestBorrowConfirmation(
.sendMessage(candidate.tabId, message)
.then((response) => {
if (settled) return;
const allowed = (response as BorrowResponseMessage | undefined)?.allowed !== false;
const candidateResponse = response as BorrowResponseMessage | undefined;
const allowed =
candidateResponse?.type === "borrow-response" && candidateResponse.allowed === true;
console.info("[bsk borrow] confirmation response", { tabId, allowed });
settle(allowed);
})
+57
View File
@@ -0,0 +1,57 @@
import { ChromiumCdp } from "@/browser-driver/chromium-cdp";
import type { SessionManager } from "@/session-manager/manager";
import type { ConsoleParams, ConsoleResult, RpcError } from "@/transport/types";
import {
type CdpRunner,
type ChromeTabsApi,
chromeTabsApi,
isRpcError,
lookupSession,
parseBufferedReadBounds,
resolveTargetTab,
} from "./shared";
export interface ConsoleDeps {
cdp: CdpRunner;
tabsApi: ChromeTabsApi;
}
function defaultConsoleDeps(): ConsoleDeps {
return {
cdp: new ChromiumCdp(),
tabsApi: chromeTabsApi,
};
}
export async function handleConsole(
manager: SessionManager,
params: ConsoleParams,
deps: ConsoleDeps = defaultConsoleDeps(),
): Promise<ConsoleResult | RpcError> {
const ctxOrErr = lookupSession(manager, params, "console");
if (isRpcError(ctxOrErr)) return ctxOrErr;
const bounds = parseBufferedReadBounds(params);
if (isRpcError(bounds)) return bounds;
const target = await resolveTargetTab(manager, ctxOrErr, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
if (!deps.cdp.ensureConsoleCapture || !deps.cdp.consoleEntriesSince) {
return { code: "cdp_failed", message: "console capture requires CDP console support" };
}
try {
await deps.cdp.ensureConsoleCapture(target.tabId);
deps.cdp.trackSessionTab?.(ctxOrErr.sessionId, target.tabId);
return deps.cdp.consoleEntriesSince(
target.tabId,
bounds.since,
bounds.limit,
bounds.maxTextChars,
params.include_stack === true,
);
} catch (err) {
return {
code: "cdp_failed",
message: err instanceof Error ? err.message : String(err),
};
}
}
+126 -12
View File
@@ -1,27 +1,38 @@
import { OVERLAY_AUTOMATION_BYPASS } from "@/lib/overlay-bridge";
import type { SessionManager } from "@/session-manager/manager";
import type { Transport } from "@/transport/transport";
import type {
ClickParams,
ConsoleParams,
EmulateParams,
EvaluateParams,
FillParams,
GetHtmlParams,
NavigateBackParams,
NavigateForwardParams,
NavigateParams,
NetworkParams,
ObserveParams,
PressParams,
ProtocolFrame,
RecordAwaitParams,
RecordStartParams,
RecordStopParams,
ReloadParams,
RequestHelpParams,
RequestFrame,
RequestHelpParams,
ResponseFrame,
RpcError,
SelectParams,
ScreenshotParams,
SelectParams,
SnapshotParams,
WaitForNavigationParams,
} from "@/transport/types";
import { isRequestFrame } from "@/transport/types";
import { handleConsole } from "./console";
import { type EmulateCdpRunner, handleEmulate } from "./emulate";
import { handleEvaluate } from "./evaluate";
import { handleRequestHelp } from "./human-loop";
import { handleClick, handleFill, handlePress, handleSelect } from "./interaction";
import {
handleNavigate,
@@ -29,29 +40,31 @@ import {
handleNavigateForward,
handleReload,
} from "./navigation";
import { handleNetwork, type NetworkCdpRunner } from "./network";
import {
type CdpRunner,
chromeTabsCaptureApi,
handleGetHtml,
handleObserve,
handleScreenshot,
handleSnapshot,
} from "./observation";
import { handleRecordAwait, handleRecordStart, handleRecordStop } from "./record";
import {
handleSessionStart,
handleSessionStop,
type SessionStartParams,
type SessionStopParams,
} from "./session";
import { OVERLAY_AUTOMATION_BYPASS } from "@/lib/overlay-bridge";
import { chromeTabsApi } from "./shared";
import {
type BorrowConfirmationApprover,
handleTabBorrow,
handleTabClose,
handleTabCreate,
handleTabList,
handleTabReturn,
handleTabSelect,
type BorrowConfirmationApprover,
type TabBorrowParams,
type TabCloseParams,
type TabCreateParams,
@@ -59,15 +72,19 @@ import {
type TabReturnParams,
type TabSelectParams,
} from "./tabs";
import { defaultWatchTabNavigation, handleRequestHelp } from "./human-loop";
import { handleWaitForNavigation } from "./waits";
import { handleWindowResize, type WindowResizeParams } from "./window";
type DispatcherCdpRunner = CdpRunner &
NetworkCdpRunner &
EmulateCdpRunner & {
detachSession(sessionId: string): Promise<void>;
};
export interface DispatcherDeps {
transport: Transport;
sessions: SessionManager;
cdp?: CdpRunner & {
detachSession(sessionId: string): Promise<void>;
};
cdp?: DispatcherCdpRunner;
/**
* Invoked whenever a dispatched RPC may have changed the live
* session set (currently `tool.session_start` and
@@ -75,6 +92,8 @@ export interface DispatcherDeps {
* `chrome.storage.session` "sessions live" flag (review M4/M5 I3).
*/
onSessionsChanged?: () => void;
/** Invoked before a tool that dispatches page input or mutates browser state is forwarded. */
onBrowserControlResumed?: (sessionId: string) => void;
/** User approval for `tool.tab_borrow` (overlay in content script). */
approveBorrow?: BorrowConfirmationApprover;
/** i18n notification copy for `tool.request_help` (resolved per-call). */
@@ -101,10 +120,9 @@ export interface DispatcherDeps {
export class ToolDispatcher {
private readonly transport: Transport;
private readonly sessions: SessionManager;
private readonly cdp?: CdpRunner & {
detachSession(sessionId: string): Promise<void>;
};
private readonly cdp?: DispatcherCdpRunner;
private readonly onSessionsChanged?: () => void;
private readonly onBrowserControlResumed?: (sessionId: string) => void;
private readonly approveBorrow?: BorrowConfirmationApprover;
private readonly helpNotificationCopy?: () => { title: string; body: string };
private subscription: { dispose(): void } | null = null;
@@ -121,6 +139,7 @@ export class ToolDispatcher {
this.sessions = deps.sessions;
this.cdp = deps.cdp;
this.onSessionsChanged = deps.onSessionsChanged;
this.onBrowserControlResumed = deps.onBrowserControlResumed;
this.approveBorrow = deps.approveBorrow;
this.helpNotificationCopy = deps.helpNotificationCopy;
}
@@ -184,6 +203,8 @@ export class ToolDispatcher {
let body: ResponseFrame;
let startedSession: string | null = null;
try {
const sessionId = sessionIdForBrowserControlMethod(req);
if (sessionId) this.onBrowserControlResumed?.(sessionId);
const result = await Promise.race([this.invoke(req, ac.signal), abortPromise(ac.signal)]);
if (isRpcError(result)) {
body = { id: req.id, error: result };
@@ -269,6 +290,14 @@ export class ToolDispatcher {
});
case "tool.tab_return":
return handleTabReturn(this.sessions, req.params as TabReturnParams);
case "tool.window_resize":
return handleWindowResize(this.sessions, req.params as WindowResizeParams);
case "tool.emulate":
return handleEmulate(
this.sessions,
req.params as EmulateParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi } : undefined,
);
case "tool.screenshot":
return handleScreenshot(
this.sessions,
@@ -277,12 +306,30 @@ export class ToolDispatcher {
? { cdp: this.cdp, tabsApi: chromeTabsCaptureApi, captureApi: chromeTabsCaptureApi }
: undefined,
);
case "tool.console":
return handleConsole(
this.sessions,
req.params as ConsoleParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi } : undefined,
);
case "tool.network":
return handleNetwork(
this.sessions,
req.params as NetworkParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi } : undefined,
);
case "tool.snapshot":
return handleSnapshot(
this.sessions,
req.params as SnapshotParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsCaptureApi } : undefined,
);
case "tool.observe":
return handleObserve(
this.sessions,
req.params as ObserveParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsCaptureApi } : undefined,
);
case "tool.get_html":
return handleGetHtml(
this.sessions,
@@ -373,12 +420,50 @@ export class ToolDispatcher {
await chrome.tabs.update(tabId, { active: true });
},
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
watchTabNavigation: defaultWatchTabNavigation,
...(this.cdp ? { cdp: this.cdp } : {}),
notifications: makeHelpNotifications(),
notificationCopy: this.helpNotificationCopy?.(),
signal,
});
case "tool.record_start":
return handleRecordStart(this.sessions, req.params as RecordStartParams, {
tabsApi: chromeTabsApi,
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
bypassOverlay: async (tabId, enabled) => {
try {
await chrome.tabs.sendMessage(tabId, {
type: OVERLAY_AUTOMATION_BYPASS,
enabled,
});
} catch {
// Content script may be unavailable on restricted pages.
}
},
...(this.cdp ? { cdp: this.cdp } : {}),
signal,
});
case "tool.record_stop":
return handleRecordStop(this.sessions, req.params as RecordStopParams, {
tabsApi: chromeTabsApi,
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
bypassOverlay: async (tabId, enabled) => {
try {
await chrome.tabs.sendMessage(tabId, {
type: OVERLAY_AUTOMATION_BYPASS,
enabled,
});
} catch {
// Content script may be unavailable on restricted pages.
}
},
signal,
});
case "tool.record_await":
return handleRecordAwait(this.sessions, req.params as RecordAwaitParams, {
tabsApi: chromeTabsApi,
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
signal,
});
default:
return {
code: "unknown_method",
@@ -398,6 +483,35 @@ function isRpcError(v: unknown): v is RpcError {
);
}
function sessionIdForBrowserControlMethod(req: RequestFrame): string | null {
switch (req.method) {
case "tool.tab_create":
case "tool.tab_close":
case "tool.tab_select":
case "tool.tab_borrow":
case "tool.tab_return":
case "tool.window_resize":
case "tool.emulate":
case "tool.navigate":
case "tool.navigate_back":
case "tool.navigate_forward":
case "tool.reload":
case "tool.click":
case "tool.fill":
case "tool.press":
case "tool.select":
case "tool.evaluate":
case "tool.observe":
case "tool.request_help":
case "tool.record_start": {
const sessionId = (req.params as { session_id?: unknown } | undefined)?.session_id;
return typeof sessionId === "string" && sessionId.length > 0 ? sessionId : null;
}
default:
return null;
}
}
/**
* Resolves never; rejects with `AbortLikeError` as soon as the signal
* fires (or immediately if it is already aborted). Used by the
+285
View File
@@ -0,0 +1,285 @@
// `tool.emulate` — apply or clear CDP Emulation-domain overrides
// (viewport metrics, user agent, touch) on a session tab, so agents can
// debug mobile page behaviour. Overrides are per-target: new tabs do
// not inherit them, and `off` restores the tab's real environment.
//
// Device presets are resolved CLI-side; this handler only executes the
// concrete override parameters it receives. Each request is merged
// field by field onto the tab's remembered emulation state and the
// merged state is applied as a whole, so a partial request does not
// reset fields set by an earlier one.
import type { DeviceMetricsOverride, UserAgentOverride } from "@/browser-driver/chromium-cdp";
import { ChromiumCdp } from "@/browser-driver/chromium-cdp";
import type { SessionManager } from "@/session-manager/manager";
import type {
EmulateOverrides,
EmulateParams,
EmulateResult,
RpcError,
UserAgentMetadata,
} from "@/transport/types";
import {
type ChromeTabsApi,
chromeTabsApi,
enforceAgentWindow,
isRpcError,
lookupSession,
resolveTargetTab,
} from "./shared";
/**
* Emulation-domain surface the handler needs. Backed by `ChromiumCdp`
* in production; tests inject a fake. `trackSessionTab` is optional so
* test doubles need not implement it.
*/
export interface EmulateCdpRunner {
setDeviceMetricsOverride(tabId: number, metrics: DeviceMetricsOverride): Promise<void>;
clearDeviceMetricsOverride(tabId: number): Promise<void>;
setUserAgentOverride(tabId: number, override: UserAgentOverride): Promise<void>;
setTouchEmulationEnabled(tabId: number, enabled: boolean, maxTouchPoints?: number): Promise<void>;
trackSessionTab?(sessionId: string, tabId: number): void;
}
export interface EmulateDeps {
cdp: EmulateCdpRunner;
tabsApi: ChromeTabsApi;
}
function defaultEmulateDeps(): EmulateDeps {
return {
cdp: new ChromiumCdp(),
tabsApi: chromeTabsApi,
};
}
/**
* Result note explaining the per-target scope of CDP emulation
* overrides, echoed on every successful call so agents know to
* re-apply emulation after opening a new tab.
*/
export const EMULATE_SCOPE_NOTE =
"emulation overrides are per-tab (CDP target): new tabs do not inherit them — re-run `bsk emulate` on each new tab";
/** Convert the wire (snake_case) UA metadata to CDP's camelCase shape. */
export function toCdpUserAgentMetadata(meta: UserAgentMetadata): Record<string, unknown> {
const out: Record<string, unknown> = {};
if (meta.brands !== undefined) out.brands = meta.brands;
if (meta.full_version !== undefined) out.fullVersion = meta.full_version;
if (meta.platform !== undefined) out.platform = meta.platform;
if (meta.platform_version !== undefined) out.platformVersion = meta.platform_version;
if (meta.architecture !== undefined) out.architecture = meta.architecture;
if (meta.model !== undefined) out.model = meta.model;
if (meta.mobile !== undefined) out.mobile = meta.mobile;
return out;
}
function invalidParams(message: string): RpcError {
return { code: "invalid_params", message };
}
/**
* Per-tab record of the emulation state applied so far. New requests
* are merged onto it field by field (see `mergeEmulateOverrides`) and
* the merged state is applied as a whole, so e.g. a later
* `--width/--height` does not silently reset the dpr/mobile of an
* earlier `--device` preset.
*
* The record is best-effort: an entry left behind by a closed tab is
* simply overwritten by the next emulate call that targets the same tab
* id, and the whole map is lost when the extension service worker
* reloads — after a reload the next emulate is equivalent to a full
* (re)set of just the fields it carries.
*/
const tabEmulationStates = new Map<number, EmulateOverrides>();
/** Test hook: drop every per-tab emulation state record. */
export function resetEmulateStatesForTests(): void {
tabEmulationStates.clear();
}
/**
* Merge one validated request onto the tab's stored state: fields
* present in `overrides` overwrite the stored value, absent fields keep
* it. Returns a fresh object; `stored` is not mutated.
*/
export function mergeEmulateOverrides(
stored: EmulateOverrides | undefined,
overrides: EmulateOverrides,
): EmulateOverrides {
const merged: EmulateOverrides = { ...stored };
if (overrides.width !== undefined) merged.width = overrides.width;
if (overrides.height !== undefined) merged.height = overrides.height;
if (overrides.device_scale_factor !== undefined) {
merged.device_scale_factor = overrides.device_scale_factor;
}
if (overrides.mobile !== undefined) merged.mobile = overrides.mobile;
if (overrides.user_agent !== undefined) merged.user_agent = overrides.user_agent;
if (overrides.accept_language !== undefined) {
merged.accept_language = overrides.accept_language;
}
if (overrides.user_agent_metadata !== undefined) {
merged.user_agent_metadata = overrides.user_agent_metadata;
}
if (overrides.touch !== undefined) merged.touch = overrides.touch;
if (overrides.max_touch_points !== undefined) {
merged.max_touch_points = overrides.max_touch_points;
}
return merged;
}
/**
* CDP calls run in sequence without rollback: on a mid-way failure the
* overrides already applied stay in effect, so the error says how to
* reset them.
*/
function cdpFailed(err: unknown): RpcError {
const cause = err instanceof Error ? err.message : String(err);
return {
code: "cdp_failed",
message: `${cause} (overrides applied before the failure were not rolled back — use --off to reset)`,
};
}
/**
* Cross-field validation for one overrides set. The CLI performs the
* same checks; the extension re-validates because the wire format is
* public.
*/
export function validateOverrides(overrides: EmulateOverrides): RpcError | null {
const hasWidth = overrides.width !== undefined;
const hasHeight = overrides.height !== undefined;
if (hasWidth !== hasHeight) {
return invalidParams("emulate overrides require width and height together");
}
if (hasWidth) {
if (!Number.isSafeInteger(overrides.width) || (overrides.width as number) <= 0) {
return invalidParams("width must be a positive integer");
}
if (!Number.isSafeInteger(overrides.height) || (overrides.height as number) <= 0) {
return invalidParams("height must be a positive integer");
}
}
if (
(overrides.device_scale_factor !== undefined || overrides.mobile !== undefined) &&
!hasWidth
) {
return invalidParams("device_scale_factor and mobile require width and height");
}
if (
overrides.device_scale_factor !== undefined &&
(!Number.isFinite(overrides.device_scale_factor) || overrides.device_scale_factor <= 0)
) {
return invalidParams("device_scale_factor must be a positive finite number");
}
if (
(overrides.accept_language !== undefined || overrides.user_agent_metadata !== undefined) &&
overrides.user_agent === undefined
) {
return invalidParams("accept_language and user_agent_metadata require user_agent");
}
if (
overrides.max_touch_points !== undefined &&
(!Number.isSafeInteger(overrides.max_touch_points) || overrides.max_touch_points < 1)
) {
return invalidParams("max_touch_points must be a positive integer");
}
const anyOverride =
hasWidth ||
overrides.user_agent !== undefined ||
overrides.touch !== undefined ||
overrides.max_touch_points !== undefined;
if (!anyOverride) {
return invalidParams(
"emulate requires at least one override (viewport, user agent, or touch) or off: true",
);
}
return null;
}
/**
* Handler for `tool.emulate` (called by the daemon over WS).
*
* Merges the requested overrides onto the tab's remembered state and
* applies the merged state in a fixed order — viewport metrics, then
* UA, then touch — or clears everything (state included) when `off` is
* set. Raw CDP failures surface as `cdp_failed` (§4.5).
*/
export async function handleEmulate(
manager: SessionManager,
params: EmulateParams,
deps: EmulateDeps = defaultEmulateDeps(),
): Promise<EmulateResult | RpcError> {
const ctxOrErr = lookupSession(manager, params, "emulate");
if (isRpcError(ctxOrErr)) return ctxOrErr;
const ctx = ctxOrErr;
const target = await resolveTargetTab(manager, ctx, params?.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "emulate");
if (denied) return denied;
if (params.off === true) {
if (params.overrides !== undefined) {
return invalidParams("emulate off cannot be combined with overrides");
}
try {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
await deps.cdp.clearDeviceMetricsOverride(target.tabId);
await deps.cdp.setTouchEmulationEnabled(target.tabId, false);
// An empty userAgent string clears the override (CDP convention).
await deps.cdp.setUserAgentOverride(target.tabId, { userAgent: "" });
} catch (err) {
return cdpFailed(err);
}
tabEmulationStates.delete(target.tabId);
return { tab_id: target.tabId, cleared: true, note: EMULATE_SCOPE_NOTE };
}
const overrides = params.overrides;
if (!overrides) {
return invalidParams("emulate requires overrides or off: true");
}
const invalid = validateOverrides(overrides);
if (invalid) return invalid;
// Fields absent from this request keep their previously applied
// values; the merged state is what gets applied (and echoed back).
const merged = mergeEmulateOverrides(tabEmulationStates.get(target.tabId), overrides);
try {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
if (merged.width !== undefined && merged.height !== undefined) {
await deps.cdp.setDeviceMetricsOverride(target.tabId, {
width: merged.width,
height: merged.height,
deviceScaleFactor: merged.device_scale_factor ?? 0,
mobile: merged.mobile ?? false,
});
}
if (merged.user_agent !== undefined) {
await deps.cdp.setUserAgentOverride(target.tabId, {
userAgent: merged.user_agent,
...(merged.accept_language !== undefined ? { acceptLanguage: merged.accept_language } : {}),
...(merged.user_agent_metadata !== undefined
? { userAgentMetadata: toCdpUserAgentMetadata(merged.user_agent_metadata) }
: {}),
});
}
if (merged.touch !== undefined || merged.max_touch_points !== undefined) {
await deps.cdp.setTouchEmulationEnabled(
target.tabId,
merged.touch ?? true,
merged.max_touch_points,
);
}
} catch (err) {
return cdpFailed(err);
}
// Record the merged state only once it was fully applied.
tabEmulationStates.set(target.tabId, merged);
return {
tab_id: target.tabId,
cleared: false,
applied: merged,
note: EMULATE_SCOPE_NOTE,
};
}
+1 -1
View File
@@ -18,6 +18,7 @@
import { ChromiumCdp } from "@/browser-driver/chromium-cdp";
import type { SessionManager } from "@/session-manager/manager";
import type { EvaluateError, EvaluateParams, EvaluateResult, RpcError } from "@/transport/types";
import { attachDialogs, markDialogCursor } from "./dialogs";
import {
type CdpRunner,
type ChromeTabsApi,
@@ -27,7 +28,6 @@ import {
lookupSession,
resolveTargetTab,
} from "./shared";
import { attachDialogs, markDialogCursor } from "./dialogs";
export interface EvaluateDeps {
cdp: CdpRunner;
+583 -162
View File
@@ -1,28 +1,30 @@
// `tool.request_help` — pause and ask the human to act on a tab.
//
// Brings the target tab to the foreground, resolves `ref` targets to a
// temporary attribute selector via CDP, asks the content script to enter
// help mode (HelpRequestOverlay + hidden ControlOverlay), and blocks
// until the user clicks Continue / Cancel, the wait times out, or the
// daemon cancels the RPC.
// `tool.request_help` — pause automation and let the human control a tab.
import type { SessionManager } from "@/session-manager/manager";
import {
HELP_ACK,
HELP_CANCEL,
HELP_FINISH,
HELP_QUERY,
HELP_REQUEST,
HELP_RESPONSE,
type HelpCancelMessage,
type HelpFinishMessage,
type HelpQueryResponse,
type HelpRequestMessage,
type HelpResponseMessage,
isHelpAckMessage,
isHelpFinishMessage,
isHelpQueryMessage,
isHelpResponseMessage,
} from "@/lib/help-bridge";
import type { SessionContext, SessionManager } from "@/session-manager/manager";
import type {
HelpCompletionCondition,
HelpCompletionCriteria,
HelpTarget,
RequestHelpParams,
RequestHelpResult,
ResolvedTarget,
RpcError,
} from "@/transport/types";
import { lookupSnapshotRef } from "./snapshot-ref";
import {
type CdpRunner,
type ChromeTabsApi,
@@ -32,56 +34,62 @@ import {
lookupSession,
resolveTargetTab,
} from "./shared";
import { lookupSnapshotRef } from "./snapshot-ref";
const DEFAULT_HELP_TIMEOUT_MS = 300_000;
/** Attribute the overlay highlights for resolved `ref` targets. */
const HELP_ATTR = "data-bsk-help";
const HELP_SEND_RETRIES = 3;
const HELP_SEND_RETRY_DELAY_MS = 350;
const HELP_REARM_DEBOUNCE_MS = 150;
const HELP_REARM_MAX_ATTEMPTS = 12;
const HELP_REARM_RETRY_DELAY_MS = 400;
const DEFAULT_COMPLETION_STABLE_MS = 1_000;
const COMPLETION_POLL_MS = 500;
export interface RequestHelpNotifications {
create(id: string, options: chrome.notifications.NotificationOptions<true>): Promise<string>;
clear(id: string): Promise<boolean>;
}
export type TabNavigationUnsubscribe = () => void;
export interface RequestHelpDeps {
tabsApi: ChromeTabsApi;
windows: { update(windowId: number, info: { focused?: boolean }): Promise<chrome.windows.Window> };
/** Activate a tab inside its window. */
windows: {
update(windowId: number, info: { focused?: boolean }): Promise<chrome.windows.Window>;
};
activateTab(tabId: number): Promise<void>;
/** Send the help-request to the tab's content script and await reply. */
sendToTab(tabId: number, msg: HelpRequestMessage | HelpCancelMessage): Promise<unknown>;
/**
* Watch for navigation on `tabId` (full page load or SPA URL change).
* Invokes `onNavigated` once; returns unsubscribe.
*/
watchTabNavigation: (tabId: number, onNavigated: () => void) => TabNavigationUnsubscribe;
cdp?: CdpRunner;
/** Pass `null` to skip OS notifications (tests). */
notifications: RequestHelpNotifications | null;
notificationCopy?: { title: string; body: string };
signal?: AbortSignal;
autoAttachLifecycle?: boolean;
}
/** Default navigation watcher: full reload (`loading`) and SPA (`url`). */
export function defaultWatchTabNavigation(
tabId: number,
onNavigated: () => void,
): TabNavigationUnsubscribe {
const listener = (
updatedTabId: number,
changeInfo: chrome.tabs.TabChangeInfo,
) => {
if (updatedTabId !== tabId) return;
if (changeInfo.url !== undefined || changeInfo.status === "loading") {
chrome.tabs.onUpdated.removeListener(listener);
onNavigated();
}
};
chrome.tabs.onUpdated.addListener(listener);
return () => chrome.tabs.onUpdated.removeListener(listener);
interface ActiveHelpRequest {
ctx: SessionContext;
requestId: string;
primaryTabId: number;
overlayTabIds: Set<number>;
prompt: string;
title?: string;
targets: HelpTarget[];
selectors: string[];
timeoutMs: number;
notificationId: string;
resolvedTargets?: ResolvedTarget[];
completionCriteria?: HelpCompletionCriteria;
deps: RequestHelpDeps;
settled: boolean;
resolve: (value: RequestHelpResult | RpcError) => void;
timeoutTimer: ReturnType<typeof setTimeout>;
completionTimer: ReturnType<typeof setInterval> | null;
completionMatchedSince: number | null;
abortHandler: (() => void) | null;
}
const activeHelpRequests = new Map<string, ActiveHelpRequest>();
const helpRearmTimers = new Map<number, ReturnType<typeof setTimeout>>();
let defaultDeps: RequestHelpDeps | null = null;
function getDefaultDeps(): RequestHelpDeps {
if (!defaultDeps) {
@@ -92,7 +100,6 @@ function getDefaultDeps(): RequestHelpDeps {
await chrome.tabs.update(tabId, { active: true });
},
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
watchTabNavigation: defaultWatchTabNavigation,
notifications: {
create: (id, opts) =>
new Promise((resolve, reject) =>
@@ -102,7 +109,8 @@ function getDefaultDeps(): RequestHelpDeps {
else resolve(rid ?? id);
}),
),
clear: (id) => new Promise((resolve) => chrome.notifications.clear(id, (c) => resolve(c ?? false))),
clear: (id) =>
new Promise((resolve) => chrome.notifications.clear(id, (c) => resolve(c ?? false))),
},
};
}
@@ -113,11 +121,50 @@ function makeRequestId(tabId: number): string {
return `${tabId}-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 8)}`;
}
/**
* Tag a `ref` target's DOM node with `data-bsk-help="<i>"` via CDP so the
* content overlay can highlight it with a plain attribute selector.
* Returns the selector on success, or null when the ref can't resolve.
*/
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function helpRequestMessage(help: ActiveHelpRequest, tabId: number): HelpRequestMessage {
const isPrimaryTab = tabId === help.primaryTabId;
return {
type: HELP_REQUEST,
requestId: help.requestId,
prompt: help.prompt,
...(help.title ? { title: help.title } : {}),
displayMode: isPrimaryTab ? "full" : "compact",
selectors: isPrimaryTab ? help.selectors : [],
timeoutMs: help.timeoutMs,
};
}
async function sendHelpRequestWithAck(
tabId: number,
msg: HelpRequestMessage,
deps: RequestHelpDeps,
): Promise<HelpFinishMessage | null> {
let lastError: unknown;
for (let attempt = 0; attempt < HELP_SEND_RETRIES; attempt += 1) {
try {
const response = await deps.sendToTab(tabId, msg);
if (isHelpAckMessage(response)) return null;
if (isHelpResponseMessage(response)) {
return {
type: HELP_FINISH,
requestId: msg.requestId,
outcome: response.outcome,
...(response.note ? { note: response.note } : {}),
};
}
lastError = new Error("content script did not ack HELP_REQUEST");
} catch (err) {
lastError = err;
}
if (attempt + 1 < HELP_SEND_RETRIES) await sleep(HELP_SEND_RETRY_DELAY_MS);
}
throw lastError ?? new Error("failed to show help overlay");
}
async function tagRefTarget(
cdp: CdpRunner,
tabId: number,
@@ -145,7 +192,6 @@ async function tagRefTarget(
}
}
/** Returns true when `selector` matches a live element in the page. */
async function selectorExists(
cdp: CdpRunner,
tabId: number,
@@ -163,11 +209,12 @@ async function selectorExists(
}
}
/** Remove every help attribute we added (best-effort cleanup). */
async function clearRefTags(cdp: CdpRunner | undefined, tabId: number): Promise<void> {
if (!cdp) return;
try {
const doc = await cdp.send<{ root?: { nodeId?: number } }>(tabId, "DOM.getDocument", { depth: 0 });
const doc = await cdp.send<{ root?: { nodeId?: number } }>(tabId, "DOM.getDocument", {
depth: 0,
});
const rootId = doc.root?.nodeId;
if (rootId === undefined) return;
const { nodeIds } = await cdp.send<{ nodeIds: number[] }>(tabId, "DOM.querySelectorAll", {
@@ -178,34 +225,16 @@ async function clearRefTags(cdp: CdpRunner | undefined, tabId: number): Promise<
await cdp.send(tabId, "DOM.removeAttribute", { nodeId, name: HELP_ATTR }).catch(() => {});
}
} catch {
// best-effort
// Best-effort cleanup.
}
}
export async function handleRequestHelp(
manager: SessionManager,
params: RequestHelpParams,
deps: RequestHelpDeps = getDefaultDeps(),
): Promise<RequestHelpResult | RpcError> {
const ctxOrErr = lookupSession(manager, params, "request_help");
if (isRpcError(ctxOrErr)) return ctxOrErr;
const ctx = ctxOrErr;
if (!params.prompt || typeof params.prompt !== "string") {
return { code: "invalid_params", message: "request_help requires a prompt" };
}
if (deps.signal?.aborted) {
return { code: "cancelled", message: "request_help aborted" };
}
const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "request_help");
if (denied) return denied;
const tabId = target.tabId;
// Resolve targets → selectors. `selector` passes through; `ref` is
// tagged with an attribute via CDP and reported as its attr selector.
const targets = params.targets ?? [];
async function resolveHelpTargets(
ctx: ReturnType<SessionManager["get"]>,
tabId: number,
targets: HelpTarget[],
deps: RequestHelpDeps,
): Promise<{ selectors: string[]; resolvedTargets?: ResolvedTarget[] }> {
const selectors: string[] = [];
const resolved: ResolvedTarget[] = [];
let rootNodeId: number | undefined;
@@ -219,22 +248,20 @@ export async function handleRequestHelp(
rootNodeId = undefined;
}
}
for (let i = 0; i < targets.length; i++) {
const tgt = targets[i] as HelpTarget;
for (let i = 0; i < targets.length; i += 1) {
const tgt = targets[i];
if (tgt.selector) {
selectors.push(tgt.selector);
let matched: boolean;
if (deps.cdp) {
matched =
rootNodeId !== undefined
? await selectorExists(deps.cdp, tabId, rootNodeId, tgt.selector)
: false;
} else {
matched = true;
}
const matched =
deps.cdp && rootNodeId !== undefined
? await selectorExists(deps.cdp, tabId, rootNodeId, tgt.selector)
: deps.cdp
? false
: true;
resolved.push({ matched, selector: tgt.selector });
} else if (tgt.ref) {
const looked = lookupSnapshotRef(ctx, tgt.ref, tabId);
const looked = ctx ? lookupSnapshotRef(ctx, tgt.ref, tabId) : null;
const backendNodeId = looked?.backendNodeId ?? null;
let sel: string | null = null;
if (backendNodeId !== null && deps.cdp) {
@@ -245,7 +272,422 @@ export async function handleRequestHelp(
}
}
// Bring the tab to the foreground.
return { selectors, resolvedTargets: resolved.length > 0 ? resolved : undefined };
}
async function refreshHelpTargets(help: ActiveHelpRequest): Promise<void> {
await clearRefTags(help.deps.cdp, help.primaryTabId);
const { selectors, resolvedTargets } = await resolveHelpTargets(
help.ctx,
help.primaryTabId,
help.targets,
help.deps,
);
help.selectors = selectors;
help.resolvedTargets = resolvedTargets;
}
async function cleanupHelp(help: ActiveHelpRequest): Promise<void> {
if (help.deps.notifications) {
await help.deps.notifications.clear(help.notificationId).catch(() => {});
}
const tabsToCancel = new Set([help.primaryTabId, ...help.overlayTabIds]);
await Promise.all(
[...tabsToCancel].map((tabId) =>
Promise.all([
clearRefTags(help.deps.cdp, tabId),
help.deps
.sendToTab(tabId, { type: HELP_CANCEL, requestId: help.requestId })
.catch(() => {}),
]),
),
);
}
async function finishHelp(
help: ActiveHelpRequest,
value: RequestHelpResult | RpcError,
notifyContent = true,
waitForCleanup = false,
): Promise<void> {
if (help.settled) return;
help.settled = true;
activeHelpRequests.delete(help.requestId);
clearTimeout(help.timeoutTimer);
if (help.completionTimer) clearInterval(help.completionTimer);
if (help.abortHandler) help.deps.signal?.removeEventListener("abort", help.abortHandler);
clearRearmTimer(help.primaryTabId);
for (const tabId of help.overlayTabIds) clearRearmTimer(tabId);
if (notifyContent) {
const cleanup = cleanupHelp(help);
if (waitForCleanup) await cleanup;
else void cleanup;
}
help.resolve(value);
}
function findHelpByKnownTabId(tabId: number): ActiveHelpRequest | null {
for (const help of activeHelpRequests.values()) {
if (!help.settled && (help.primaryTabId === tabId || help.overlayTabIds.has(tabId))) {
return help;
}
}
return null;
}
async function findHelpForTab(
tabId: number,
deps: RequestHelpDeps,
): Promise<ActiveHelpRequest | null> {
const direct = findHelpByKnownTabId(tabId);
if (direct) return direct;
try {
const tab = await deps.tabsApi.get(tabId);
const windowId = tab.windowId;
if (typeof windowId !== "number") return null;
for (const help of activeHelpRequests.values()) {
if (!help.settled && help.ctx.agentWindowId === windowId) return help;
}
} catch {
return null;
}
return null;
}
function clearRearmTimer(tabId: number): void {
const timer = helpRearmTimers.get(tabId);
if (!timer) return;
clearTimeout(timer);
helpRearmTimers.delete(tabId);
}
async function sendCurrentHelpOverlay(help: ActiveHelpRequest): Promise<HelpFinishMessage | null> {
const legacyFinish = await sendHelpRequestWithAck(
help.primaryTabId,
helpRequestMessage(help, help.primaryTabId),
help.deps,
);
help.overlayTabIds.add(help.primaryTabId);
return legacyFinish;
}
async function refreshAndSendHelpOverlay(
help: ActiveHelpRequest,
tabId: number,
): Promise<HelpFinishMessage | null> {
if (tabId === help.primaryTabId) await refreshHelpTargets(help);
const legacyFinish = await sendHelpRequestWithAck(
tabId,
helpRequestMessage(help, tabId),
help.deps,
);
help.overlayTabIds.add(tabId);
return legacyFinish;
}
async function rearmHelp(help: ActiveHelpRequest, tabId: number): Promise<boolean> {
for (let attempt = 0; attempt < HELP_REARM_MAX_ATTEMPTS; attempt += 1) {
try {
const legacyFinish = await refreshAndSendHelpOverlay(help, tabId);
if (legacyFinish && !help.settled) {
await finishHelp(
help,
{
outcome: legacyFinish.outcome,
...(legacyFinish.note ? { note: legacyFinish.note } : {}),
tab_id: help.primaryTabId,
resolved_targets: help.resolvedTargets,
},
true,
true,
);
}
return true;
} catch {
if (attempt + 1 < HELP_REARM_MAX_ATTEMPTS) await sleep(HELP_REARM_RETRY_DELAY_MS);
}
}
return false;
}
function scheduleRearmForTab(tabId: number, deps: RequestHelpDeps): void {
const existing = helpRearmTimers.get(tabId);
if (existing) clearTimeout(existing);
helpRearmTimers.set(
tabId,
setTimeout(() => {
helpRearmTimers.delete(tabId);
void (async () => {
const help = await findHelpForTab(tabId, deps);
if (!help) return;
await rearmHelp(help, tabId);
})();
}, HELP_REARM_DEBOUNCE_MS),
);
}
let detachHelpObservation: (() => void) | null = null;
export function ensureHelpLifecycleListeners(deps: RequestHelpDeps = getDefaultDeps()): void {
if (detachHelpObservation) return;
const detachRuntime = attachHelpRuntimeListener(deps);
const detachTabs = attachHelpTabListener(deps);
const detachNav = attachHelpNavigationListener(deps);
detachHelpObservation = () => {
detachRuntime();
detachTabs();
detachNav();
};
}
export function resetHelpLifecycleForTests(): void {
detachHelpObservation?.();
detachHelpObservation = null;
for (const timer of helpRearmTimers.values()) clearTimeout(timer);
helpRearmTimers.clear();
for (const help of activeHelpRequests.values()) {
clearTimeout(help.timeoutTimer);
if (help.completionTimer) clearInterval(help.completionTimer);
}
activeHelpRequests.clear();
}
function attachHelpRuntimeListener(deps: RequestHelpDeps): () => void {
const listener = (
message: unknown,
sender: chrome.runtime.MessageSender,
sendResponse: (response: HelpQueryResponse) => void,
) => {
if (isHelpFinishMessage(message)) {
void finishHelpFromContent(message, sender, deps);
return false;
}
if (isHelpQueryMessage(message)) {
const tabId = sender.tab?.id;
if (tabId === undefined) {
sendResponse({ active: false });
return false;
}
void (async () => {
const help = await findHelpForTab(tabId, deps);
if (!help) {
sendResponse({ active: false });
return;
}
if (tabId === help.primaryTabId) await refreshHelpTargets(help);
help.overlayTabIds.add(tabId);
sendResponse({
active: true,
request: {
requestId: help.requestId,
prompt: help.prompt,
...(help.title ? { title: help.title } : {}),
displayMode: tabId === help.primaryTabId ? "full" : "compact",
selectors: tabId === help.primaryTabId ? help.selectors : [],
timeoutMs: help.timeoutMs,
},
});
})();
return true;
}
return false;
};
chrome.runtime.onMessage.addListener(listener);
return () => chrome.runtime.onMessage.removeListener(listener);
}
async function finishHelpFromContent(
message: HelpFinishMessage,
sender: chrome.runtime.MessageSender,
deps: RequestHelpDeps,
): Promise<void> {
const help = activeHelpRequests.get(message.requestId);
if (!help || help.settled) return;
const tabId = sender.tab?.id;
if (tabId !== undefined) {
const match = await findHelpForTab(tabId, deps);
if (match !== help) return;
}
await finishHelp(
help,
{
outcome: message.outcome,
...(message.note ? { note: message.note } : {}),
tab_id: help.primaryTabId,
resolved_targets: help.resolvedTargets,
},
true,
true,
);
}
function attachHelpTabListener(deps: RequestHelpDeps): () => void {
const onCreated = (tab: chrome.tabs.Tab) => {
if (typeof tab.id !== "number") return;
scheduleRearmForTab(tab.id, deps);
};
const onActivated = (activeInfo: chrome.tabs.TabActiveInfo) => {
scheduleRearmForTab(activeInfo.tabId, deps);
};
chrome.tabs.onCreated?.addListener(onCreated);
chrome.tabs.onActivated?.addListener(onActivated);
return () => {
chrome.tabs.onCreated?.removeListener(onCreated);
chrome.tabs.onActivated?.removeListener(onActivated);
};
}
function attachHelpNavigationListener(deps: RequestHelpDeps): () => void {
const onMainFrameComplete = (tabId: number) => scheduleRearmForTab(tabId, deps);
if (chrome.webNavigation?.onCompleted) {
const completedListener = (
details: chrome.webNavigation.WebNavigationFramedCallbackDetails,
) => {
if (details.frameId !== 0) return;
onMainFrameComplete(details.tabId);
};
chrome.webNavigation.onCompleted.addListener(completedListener);
return () => chrome.webNavigation.onCompleted.removeListener(completedListener);
}
const listener = (tabId: number, info: chrome.tabs.TabChangeInfo) => {
if (info.status !== "complete") return;
onMainFrameComplete(tabId);
};
chrome.tabs.onUpdated?.addListener(listener);
return () => chrome.tabs.onUpdated?.removeListener(listener);
}
async function evaluateCompletionCondition(
condition: HelpCompletionCondition,
help: ActiveHelpRequest,
): Promise<boolean> {
if (condition.url_contains || condition.url_matches) {
let url = "";
try {
const tab = await help.deps.tabsApi.get(help.primaryTabId);
url = tab.url ?? "";
} catch {
return false;
}
if (condition.url_contains && !url.includes(condition.url_contains)) return false;
if (condition.url_matches) {
try {
if (!new RegExp(condition.url_matches).test(url)) return false;
} catch {
return false;
}
}
}
if (
condition.selector_exists ||
condition.selector_missing ||
condition.text_exists ||
condition.text_missing
) {
if (!help.deps.cdp) return false;
const expression = `(() => {
const selectorExists = ${JSON.stringify(condition.selector_exists ?? null)};
const selectorMissing = ${JSON.stringify(condition.selector_missing ?? null)};
const textExists = ${JSON.stringify(condition.text_exists ?? null)};
const textMissing = ${JSON.stringify(condition.text_missing ?? null)};
if (selectorExists && !document.querySelector(selectorExists)) return false;
if (selectorMissing && document.querySelector(selectorMissing)) return false;
const text = document.body ? document.body.innerText || "" : "";
if (textExists && !text.includes(textExists)) return false;
if (textMissing && text.includes(textMissing)) return false;
return true;
})()`;
try {
const result = await help.deps.cdp.send<{ result?: { value?: boolean } }>(
help.primaryTabId,
"Runtime.evaluate",
{ expression, returnByValue: true },
);
if (result.result?.value !== true) return false;
} catch {
return false;
}
}
return true;
}
async function completionCriteriaMatches(help: ActiveHelpRequest): Promise<boolean> {
const criteria = help.completionCriteria;
if (!criteria) return false;
const any = criteria.any ?? [];
const all = criteria.all ?? [];
const hasAny = any.length > 0;
const hasAll = all.length > 0;
if (!hasAny && !hasAll) return false;
if (hasAll) {
for (const condition of all) {
if (!(await evaluateCompletionCondition(condition, help))) return false;
}
}
if (hasAny) {
for (const condition of any) {
if (await evaluateCompletionCondition(condition, help)) return true;
}
return false;
}
return true;
}
function startCompletionPolling(help: ActiveHelpRequest): void {
if (!help.completionCriteria) return;
const stableMs = help.completionCriteria.stable_for_ms ?? DEFAULT_COMPLETION_STABLE_MS;
const check = async () => {
if (help.settled) return;
const matched = await completionCriteriaMatches(help);
const now = Date.now();
if (!matched) {
help.completionMatchedSince = null;
return;
}
help.completionMatchedSince ??= now;
if (now - help.completionMatchedSince < stableMs) return;
void finishHelp(help, {
outcome: "completed",
completed_by: "system",
tab_id: help.primaryTabId,
resolved_targets: help.resolvedTargets,
});
};
help.completionTimer = setInterval(() => void check(), COMPLETION_POLL_MS);
void check();
}
export async function handleRequestHelp(
manager: SessionManager,
params: RequestHelpParams,
deps: RequestHelpDeps = getDefaultDeps(),
): Promise<RequestHelpResult | RpcError> {
const ctxOrErr = lookupSession(manager, params, "request_help");
if (isRpcError(ctxOrErr)) return ctxOrErr;
const ctx = ctxOrErr;
if (!params.prompt || typeof params.prompt !== "string") {
return { code: "invalid_params", message: "request_help requires a prompt" };
}
if (deps.signal?.aborted) return { code: "cancelled", message: "request_help aborted" };
const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "request_help");
if (denied) return denied;
if (deps.autoAttachLifecycle !== false) ensureHelpLifecycleListeners(deps);
const tabId = target.tabId;
const initialTargets = params.targets ?? [];
const { selectors, resolvedTargets } = await resolveHelpTargets(ctx, tabId, initialTargets, deps);
await deps.windows.update(target.windowId, { focused: true }).catch(() => {});
await deps.activateTab(tabId).catch(() => {});
@@ -253,7 +695,6 @@ export async function handleRequestHelp(
const notificationId = `bsk-help:${requestId}`;
const timeoutMs = params.timeout_ms ?? DEFAULT_HELP_TIMEOUT_MS;
// OS notification (best-effort).
if (deps.notifications) {
const copy = deps.notificationCopy ?? {
title: "BrowserSkill: Agent needs your help",
@@ -270,85 +711,65 @@ export async function handleRequestHelp(
.catch(() => {});
}
const cleanup = async () => {
await clearRefTags(deps.cdp, tabId);
if (deps.notifications) await deps.notifications.clear(notificationId).catch(() => {});
// Retract the overlay in case we resolved via timeout / abort.
await deps
.sendToTab(tabId, { type: HELP_CANCEL, requestId } satisfies HelpCancelMessage)
.catch(() => {});
};
return new Promise<RequestHelpResult | RpcError>((resolve) => {
const timeoutTimer = setTimeout(() => {
void finishHelp(help, {
outcome: "timed_out",
tab_id: help.primaryTabId,
resolved_targets: help.resolvedTargets,
});
}, timeoutMs);
const request: HelpRequestMessage = {
type: HELP_REQUEST,
requestId,
prompt: params.prompt,
...(params.title ? { title: params.title } : {}),
selectors,
timeoutMs,
};
const resolvedTargets = resolved.length > 0 ? resolved : undefined;
return new Promise<RequestHelpResult | RpcError>((resolveOuter) => {
let settled = false;
const unwatchNav = deps.watchTabNavigation(tabId, () => {
finish({ outcome: "navigated", tab_id: tabId, resolved_targets: resolvedTargets });
});
const finish = (value: RequestHelpResult | RpcError) => {
if (settled) return;
settled = true;
if (timer !== null) clearTimeout(timer);
unwatchNav();
deps.signal?.removeEventListener("abort", onAbort);
void cleanup();
resolveOuter(value);
const help: ActiveHelpRequest = {
ctx,
requestId,
primaryTabId: tabId,
overlayTabIds: new Set(),
prompt: params.prompt,
...(params.title ? { title: params.title } : {}),
targets: initialTargets,
selectors,
timeoutMs,
notificationId,
resolvedTargets,
completionCriteria: params.completion_criteria,
deps,
settled: false,
resolve,
timeoutTimer,
completionTimer: null,
completionMatchedSince: null,
abortHandler: null,
};
const onAbort = () => finish({ code: "cancelled", message: "request_help aborted" });
const timer = setTimeout(() => {
finish({ outcome: "timed_out", tab_id: tabId, resolved_targets: resolvedTargets });
}, timeoutMs) as unknown as number;
if (deps.signal) {
if (deps.signal.aborted) {
onAbort();
return;
}
deps.signal.addEventListener("abort", onAbort, { once: true });
const onAbort = () =>
void finishHelp(help, { code: "cancelled", message: "request_help aborted" });
help.abortHandler = onAbort;
activeHelpRequests.set(requestId, help);
if (deps.signal?.aborted) {
onAbort();
return;
}
deps.signal?.addEventListener("abort", onAbort, { once: true });
startCompletionPolling(help);
deps
.sendToTab(tabId, request)
.then((reply) => {
const res = reply as HelpResponseMessage | undefined;
if (
!res ||
res.type !== HELP_RESPONSE ||
(res.outcome !== "continued" && res.outcome !== "cancelled")
) {
finish({
code: "protocol_error",
message: `invalid help response from tab ${tabId}`,
});
return;
}
finish({
outcome: res.outcome,
...(res.note ? { note: res.note } : {}),
tab_id: tabId,
resolved_targets: resolvedTargets,
});
void sendCurrentHelpOverlay(help)
.then(async (legacyFinish) => {
if (!legacyFinish || help.settled) return;
await finishHelp(
help,
{
outcome: legacyFinish.outcome,
...(legacyFinish.note ? { note: legacyFinish.note } : {}),
tab_id: help.primaryTabId,
resolved_targets: help.resolvedTargets,
},
true,
true,
);
})
.catch((err) => {
finish({
code: "protocol_error",
message: `failed to show help overlay on tab ${tabId}: ${
err instanceof Error ? err.message : String(err)
}`,
});
.catch(() => {
scheduleRearmForTab(tabId, deps);
});
});
}
+6 -4
View File
@@ -21,10 +21,11 @@ import type {
MouseButton,
PressParams,
PressResult,
RpcError,
SelectParams,
SelectResult,
RpcError,
} from "@/transport/types";
import { attachDialogs, markDialogCursor } from "./dialogs";
import {
backendNodeToObject,
boxCentre,
@@ -33,7 +34,6 @@ import {
scrollNodeIntoView,
} from "./element-geometry";
import { rpcError } from "./errors";
import { resolveSnapshotRef } from "./snapshot-ref";
import {
type CdpRunner,
type ChromeTabsApi,
@@ -43,7 +43,7 @@ import {
lookupSession,
resolveTargetTab,
} from "./shared";
import { attachDialogs, markDialogCursor } from "./dialogs";
import { resolveSnapshotRef } from "./snapshot-ref";
export interface InteractionDeps {
cdp: CdpRunner;
@@ -858,7 +858,9 @@ export async function handleSelect(
);
}
const attrs = described.node.attributes ?? [];
const isMultiple = attrs.some((attr, idx) => idx % 2 === 0 && attr.toLowerCase() === "multiple");
const isMultiple = attrs.some(
(attr, idx) => idx % 2 === 0 && attr.toLowerCase() === "multiple",
);
if (!isMultiple && params.values.length !== 1) {
return rpcError(
"invalid_params",
+9 -5
View File
@@ -28,6 +28,7 @@ import type {
RpcError,
WaitUntil,
} from "@/transport/types";
import { attachDialogs, markDialogCursor } from "./dialogs";
import {
type CdpRunner,
type ChromeTabsApi,
@@ -37,7 +38,6 @@ import {
lookupSession,
resolveTargetTab,
} from "./shared";
import { attachDialogs, markDialogCursor } from "./dialogs";
export interface NavigationDeps {
cdp: CdpRunner;
@@ -120,9 +120,7 @@ export function lifecycleAlreadyReached(readyState: string, targetName: string):
case "DOMContentLoaded":
return readyState === "interactive" || readyState === "complete";
case "commit":
return (
readyState === "interactive" || readyState === "complete" || readyState === "loading"
);
return readyState === "interactive" || readyState === "complete" || readyState === "loading";
case "networkIdle":
// Must come from `Page.lifecycleEvent`; readyState does not imply network idle.
return false;
@@ -285,7 +283,13 @@ function startLifecycleWait(
const maybeFinishPending = () => {
if (settled || !pendingLifecycle) return;
if (!acceptLifecycleEvent(pendingLifecycle.name, pendingLifecycle.frameId, pendingLifecycle.loaderId)) {
if (
!acceptLifecycleEvent(
pendingLifecycle.name,
pendingLifecycle.frameId,
pendingLifecycle.loaderId,
)
) {
return;
}
if (lifecycleMeetsOrExceeds(pendingLifecycle.name, targetName)) {
+63
View File
@@ -0,0 +1,63 @@
import { ChromiumCdp } from "@/browser-driver/chromium-cdp";
import type { SessionManager } from "@/session-manager/manager";
import type { NetworkParams, NetworkResult, RpcError } from "@/transport/types";
import {
type ChromeTabsApi,
chromeTabsApi,
isRpcError,
lookupSession,
parseBufferedReadBounds,
resolveTargetTab,
} from "./shared";
export interface NetworkCdpRunner {
trackSessionTab?(sessionId: string, tabId: number): void;
ensureNetworkCapture(tabId: number): Promise<void>;
networkEntriesSince(
tabId: number,
since: number | undefined,
limit: number,
maxTextChars: number,
): NetworkResult;
}
export interface NetworkDeps {
cdp: NetworkCdpRunner;
tabsApi: ChromeTabsApi;
}
function defaultNetworkDeps(): NetworkDeps {
return {
cdp: new ChromiumCdp(),
tabsApi: chromeTabsApi,
};
}
export async function handleNetwork(
manager: SessionManager,
params: NetworkParams,
deps: NetworkDeps = defaultNetworkDeps(),
): Promise<NetworkResult | RpcError> {
const ctxOrErr = lookupSession(manager, params, "network");
if (isRpcError(ctxOrErr)) return ctxOrErr;
const bounds = parseBufferedReadBounds(params);
if (isRpcError(bounds)) return bounds;
const target = await resolveTargetTab(manager, ctxOrErr, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
try {
deps.cdp.trackSessionTab?.(ctxOrErr.sessionId, target.tabId);
await deps.cdp.ensureNetworkCapture(target.tabId);
return deps.cdp.networkEntriesSince(
target.tabId,
bounds.since,
bounds.limit,
bounds.maxTextChars,
);
} catch (err) {
return {
code: "cdp_failed",
message: err instanceof Error ? err.message : String(err),
};
}
}
File diff suppressed because it is too large Load Diff
+839
View File
@@ -0,0 +1,839 @@
// `tool.record_start` / `tool.record_stop` / `tool.record_await` — capture
// user actions in the Agent Window via the content script and return a
// semantic (LLM textbook) trace.
import {
isRecordFinishMessage,
isRecordQueryMessage,
isRecordStepMessage,
RECORD_CANCEL,
RECORD_START,
RECORD_STEP,
RECORD_STOP,
type RecordCancelMessage,
type RecordFinishMessage,
type RecordQueryResponse,
type RecordStartAck,
type RecordStartMessage,
type RecordStopMessage,
} from "@/lib/record-bridge";
import { appendRecordedPayload, observeRecordedNavigation } from "@/lib/recording-step-buffer";
import { reduceTraceSteps, resolveTraceStartUrl } from "@/lib/trace-reducer";
import type { SessionManager } from "@/session-manager/manager";
import type {
DraftTraceStep,
RecordAwaitParams,
RecordAwaitResult,
RecordStartParams,
RecordStartResult,
RecordStopParams,
RecordStopResult,
RpcError,
Trace,
} from "@/transport/types";
import { handleNavigate } from "./navigation";
import {
type CdpRunner,
type ChromeTabsApi,
chromeTabsApi,
isRpcError,
lookupSession,
resolveTargetTab,
} from "./shared";
interface ActiveRecording {
requestId: string;
tabId: number;
agentWindowId: number;
startUrl?: string;
purpose?: string;
steps: DraftTraceStep[];
startedAt: string;
startedAtMs: number;
finishPromise: Promise<Trace>;
resolveFinish: (trace: Trace) => void;
rejectFinish: (err: Error) => void;
settled: boolean;
finishing: boolean;
currentUrl?: string;
pendingNavigation: boolean;
pendingNavigationDeadline?: number;
}
const recordings = new Map<string, ActiveRecording>();
const RECORD_START_RETRIES = 3;
const RECORD_START_RETRY_DELAY_MS = 500;
const RECORD_REARM_DEBOUNCE_MS = 150;
const RECORD_REARM_MAX_ATTEMPTS = 12;
const RECORD_REARM_RETRY_DELAY_MS = 400;
const rearmTimers = new Map<number, ReturnType<typeof setTimeout>>();
function makeRequestId(tabId: number): string {
return `rec-${tabId}-${Date.now().toString(36)}`;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
/** Injectable http(s) landing page when `tool.record_start` omits `url`. */
export const RECORD_DEFAULT_START_URL = "https://example.com/";
/** Pages where MV3 content scripts cannot attach (Agent Window boots here). */
function isContentScriptRestrictedUrl(url: string | undefined): boolean {
if (!url) return true;
const lower = url.toLowerCase();
return (
lower === "about:blank" ||
lower.startsWith("about:") ||
lower.startsWith("chrome://") ||
lower.startsWith("chrome-extension://") ||
lower.startsWith("edge://") ||
lower.startsWith("devtools://") ||
lower.startsWith("devtools:") ||
lower.startsWith("https://chrome.google.com/webstore")
);
}
async function waitForTabReady(
tabId: number,
tabsApi: ChromeTabsApi,
timeoutMs = 10_000,
): Promise<void> {
try {
const tab = await tabsApi.get(tabId);
if (tab.status === "complete") return;
} catch {
return;
}
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => {
chrome.tabs.onUpdated.removeListener(listener);
reject(new Error("tab load timeout"));
}, timeoutMs);
const listener = (updatedTabId: number, info: chrome.tabs.TabChangeInfo) => {
if (updatedTabId !== tabId || info.status !== "complete") return;
clearTimeout(timer);
chrome.tabs.onUpdated.removeListener(listener);
resolve();
};
chrome.tabs.onUpdated.addListener(listener);
});
}
function isRecordStartAck(response: unknown): response is RecordStartAck {
return (
typeof response === "object" &&
response !== null &&
"ok" in response &&
(response as RecordStartAck).ok === true
);
}
async function sendRecordStartWithAck(
tabId: number,
msg: RecordStartMessage,
sendToTab: RecordDeps["sendToTab"],
): Promise<void> {
let lastError: unknown;
for (let attempt = 0; attempt < RECORD_START_RETRIES; attempt += 1) {
try {
const response = await sendToTab(tabId, msg);
if (isRecordStartAck(response)) return;
lastError = new Error("content script did not ack RECORD_START");
} catch (err) {
lastError = err;
}
if (attempt + 1 < RECORD_START_RETRIES) {
await sleep(RECORD_START_RETRY_DELAY_MS);
}
}
throw lastError ?? new Error("failed to start recording in content script");
}
function buildTrace(recording: ActiveRecording): Trace {
const { pages, steps } = reduceTraceSteps(recording.steps, recording.startUrl);
const startUrl = resolveTraceStartUrl(recording.steps, recording.startUrl, pages);
return {
...(recording.purpose ? { purpose: recording.purpose } : {}),
recorded_at: new Date().toISOString(),
started_at: recording.startedAt,
entry: { start_url: startUrl },
pages,
steps,
};
}
export interface RecordDeps {
tabsApi: ChromeTabsApi;
sendToTab(
tabId: number,
msg: RecordStartMessage | RecordStopMessage | RecordCancelMessage,
): Promise<unknown>;
bypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>;
cdp?: CdpRunner;
signal?: AbortSignal;
}
let defaultDeps: RecordDeps | null = null;
function getDefaultDeps(): RecordDeps {
if (!defaultDeps) {
defaultDeps = {
tabsApi: chromeTabsApi,
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
};
}
return defaultDeps;
}
/** Disposer for lazily attached tab / webNavigation observers. */
let detachBrowserObservation: (() => void) | null = null;
type AttachObservation = (deps: RecordDeps) => () => void;
// Deferred wrappers so we do not capture attach* before their declarations.
let attachTabObservation: AttachObservation = (deps) => attachRecordTabListener(deps);
let attachNavObservation: AttachObservation = (deps) => attachRecordNavigationListener(deps);
/** Test seam: swap real chrome listeners for fakes. */
export function setBrowserObservationAttachForTests(
tab: AttachObservation | null,
nav: AttachObservation | null,
): void {
attachTabObservation = tab ?? ((deps) => attachRecordTabListener(deps));
attachNavObservation = nav ?? ((deps) => attachRecordNavigationListener(deps));
}
export function isBrowserObservationAttachedForTests(): boolean {
return detachBrowserObservation !== null;
}
export function resetBrowserObservationForTests(): void {
detachBrowserObservation?.();
detachBrowserObservation = null;
recordings.clear();
attachTabObservation = (deps) => attachRecordTabListener(deps);
attachNavObservation = (deps) => attachRecordNavigationListener(deps);
}
/**
* Attach tab/webNavigation listeners while any recording is active.
* Must run before navigate-on-start so rearm observes the destination load.
*/
export function ensureBrowserObservationListeners(deps: RecordDeps = getDefaultDeps()): void {
if (detachBrowserObservation) return;
const detachTab = attachTabObservation(deps);
const detachNav = attachNavObservation(deps);
detachBrowserObservation = () => {
detachTab();
detachNav();
};
}
/** Detach when the recordings map is empty. */
export function releaseBrowserObservationListenersIfIdle(): void {
if (recordings.size > 0) return;
if (!detachBrowserObservation) return;
detachBrowserObservation();
detachBrowserObservation = null;
}
export function attachRecordStepListener(): () => void {
const listener = (
message: unknown,
_sender: chrome.runtime.MessageSender,
_sendResponse: () => void,
) => {
if (!isRecordStepMessage(message)) return false;
for (const recording of recordings.values()) {
if (recording.requestId !== message.requestId) continue;
appendRecordedPayload(recording, message.step);
return false;
}
return false;
};
chrome.runtime.onMessage.addListener(listener);
return () => chrome.runtime.onMessage.removeListener(listener);
}
export function attachRecordFinishListener(deps: RecordDeps = getDefaultDeps()): () => void {
const listener = (
message: unknown,
sender: chrome.runtime.MessageSender,
_sendResponse: () => void,
) => {
if (!isRecordFinishMessage(message)) return false;
const tabId = sender.tab?.id;
if (tabId === undefined) return false;
void finishRecordingByRequest(message.requestId, tabId, deps);
return false;
};
chrome.runtime.onMessage.addListener(listener);
return () => chrome.runtime.onMessage.removeListener(listener);
}
function findRecordingByTabId(tabId: number): ActiveRecording | null {
for (const recording of recordings.values()) {
if (recording.tabId === tabId && !recording.settled) return recording;
}
return null;
}
async function findRecordingForTab(
tabId: number,
deps: RecordDeps,
): Promise<ActiveRecording | null> {
const direct = findRecordingByTabId(tabId);
if (direct) return direct;
try {
const tab = await deps.tabsApi.get(tabId);
const windowId = tab.windowId;
if (typeof windowId !== "number") return null;
for (const recording of recordings.values()) {
if (!recording.settled && recording.agentWindowId === windowId) return recording;
}
} catch {
return null;
}
return null;
}
function clearRearmTimer(tabId: number): void {
const timer = rearmTimers.get(tabId);
if (timer) {
clearTimeout(timer);
rearmTimers.delete(tabId);
}
}
async function clearRearmTimersForRecording(
recording: ActiveRecording,
deps: RecordDeps,
): Promise<void> {
clearRearmTimer(recording.tabId);
try {
const tabs = await deps.tabsApi.query({ windowId: recording.agentWindowId });
for (const tab of tabs) {
if (typeof tab.id === "number") clearRearmTimer(tab.id);
}
} catch {
// Best-effort cleanup.
}
}
async function stopRecordingOnAllAgentTabs(
recording: ActiveRecording,
deps: RecordDeps,
): Promise<void> {
const stopMsg: RecordStopMessage = { type: RECORD_STOP, requestId: recording.requestId };
let tabIds = [recording.tabId];
try {
const tabs = await deps.tabsApi.query({ windowId: recording.agentWindowId });
tabIds = [
...new Set([
recording.tabId,
...tabs.flatMap((tab) => (typeof tab.id === "number" ? [tab.id] : [])),
]),
];
} catch {
// Fall back to the current recording tab.
}
for (const tabId of tabIds) {
try {
const response = await deps.sendToTab(tabId, stopMsg);
if (tabId === recording.tabId && !isRecordStartAck(response)) {
throw new Error("content script did not confirm recorded steps");
}
} catch {
if (tabId === recording.tabId) {
throw new Error("failed to flush recorded steps");
}
}
if (deps.bypassOverlay) {
try {
await deps.bypassOverlay(tabId, false);
} catch {
// Best-effort cleanup.
}
}
}
}
async function rearmRecording(
recording: ActiveRecording,
targetTabId: number,
deps: RecordDeps,
): Promise<boolean> {
// Do NOT toggle automation-bypass here: each retry used to increment the
// content-script counter, and a single stop decrement left the ControlOverlay
// stuck with pointer-events:none (page usable, Interrupt dead). RecordOverlay
// already hides the control chrome while activeRecord is set.
for (let attempt = 0; attempt < RECORD_REARM_MAX_ATTEMPTS; attempt += 1) {
const startMsg: RecordStartMessage = {
type: RECORD_START,
requestId: recording.requestId,
startedAtMs: recording.startedAtMs,
};
try {
await sendRecordStartWithAck(targetTabId, startMsg, deps.sendToTab);
recording.tabId = targetTabId;
return true;
} catch {
if (attempt + 1 < RECORD_REARM_MAX_ATTEMPTS) {
await sleep(RECORD_REARM_RETRY_DELAY_MS);
}
}
}
return false;
}
function scheduleRearmForTab(tabId: number, deps: RecordDeps): void {
const existing = rearmTimers.get(tabId);
if (existing) clearTimeout(existing);
rearmTimers.set(
tabId,
setTimeout(() => {
rearmTimers.delete(tabId);
void (async () => {
const current = await findRecordingForTab(tabId, deps);
if (current) await rearmRecording(current, tabId, deps);
})();
}, RECORD_REARM_DEBOUNCE_MS),
);
}
export function attachRecordTabListener(deps: RecordDeps = getDefaultDeps()): () => void {
const onCreated = (tab: chrome.tabs.Tab) => {
const tabId = tab.id;
const windowId = tab.windowId;
if (tabId === undefined || windowId === undefined) return;
for (const recording of recordings.values()) {
if (recording.settled || recording.agentWindowId !== windowId) continue;
scheduleRearmForTab(tabId, deps);
return;
}
};
const onActivated = (activeInfo: chrome.tabs.TabActiveInfo) => {
scheduleRearmForTab(activeInfo.tabId, deps);
};
chrome.tabs.onCreated.addListener(onCreated);
chrome.tabs.onActivated.addListener(onActivated);
return () => {
chrome.tabs.onCreated.removeListener(onCreated);
chrome.tabs.onActivated.removeListener(onActivated);
};
}
export function attachRecordNavigationListener(deps: RecordDeps = getDefaultDeps()): () => void {
const observeMainFrame = (tabId: number, url?: string, causedByAction?: boolean) => {
void (async () => {
const recording = await findRecordingForTab(tabId, deps);
if (recording && url) {
observeRecordedNavigation(recording, url, causedByAction);
}
})();
};
const onMainFrameComplete = (tabId: number, url?: string) => {
void (async () => {
observeMainFrame(tabId, url);
scheduleRearmForTab(tabId, deps);
})();
};
if (chrome.webNavigation?.onCompleted) {
const completedListener = (
details: chrome.webNavigation.WebNavigationFramedCallbackDetails,
) => {
if (details.frameId !== 0) return;
onMainFrameComplete(details.tabId, details.url);
};
const committedListener = (
details: chrome.webNavigation.WebNavigationTransitionCallbackDetails,
) => {
if (details.frameId !== 0) return;
const causedByAction =
details.transitionType === "link" || details.transitionType === "form_submit"
? true
: details.transitionType === "typed" ||
details.transitionType === "auto_bookmark" ||
details.transitionType === "generated" ||
details.transitionType === "keyword" ||
details.transitionType === "keyword_generated" ||
details.transitionType === "reload"
? false
: undefined;
observeMainFrame(details.tabId, details.url, causedByAction);
};
chrome.webNavigation.onCompleted.addListener(completedListener);
chrome.webNavigation.onCommitted?.addListener(committedListener);
return () => {
chrome.webNavigation.onCompleted.removeListener(completedListener);
chrome.webNavigation.onCommitted?.removeListener(committedListener);
};
}
const listener = (tabId: number, info: chrome.tabs.TabChangeInfo) => {
if (info.status !== "complete") return;
onMainFrameComplete(tabId, info.url);
};
chrome.tabs.onUpdated.addListener(listener);
return () => chrome.tabs.onUpdated.removeListener(listener);
}
export function attachRecordQueryListener(deps: RecordDeps = getDefaultDeps()): () => void {
const listener = (
message: unknown,
sender: chrome.runtime.MessageSender,
sendResponse: (response: RecordQueryResponse) => void,
) => {
if (!isRecordQueryMessage(message)) return false;
const tabId = sender.tab?.id;
if (tabId === undefined) {
sendResponse({ active: false });
return false;
}
void (async () => {
const recording = await findRecordingForTab(tabId, deps);
if (!recording) {
sendResponse({ active: false });
return;
}
await rearmRecording(recording, tabId, deps);
sendResponse({
active: true,
requestId: recording.requestId,
startedAtMs: recording.startedAtMs,
});
})();
return true;
};
chrome.runtime.onMessage.addListener(listener);
return () => chrome.runtime.onMessage.removeListener(listener);
}
async function finishRecordingByRequest(
requestId: string,
tabId: number,
deps: RecordDeps,
): Promise<void> {
for (const [sessionId, recording] of recordings) {
if (recording.requestId !== requestId || recording.settled) continue;
const match = await findRecordingForTab(tabId, deps);
if (match !== recording) continue;
await finishRecording(sessionId, deps);
return;
}
}
async function finishRecording(sessionId: string, deps: RecordDeps): Promise<Trace | null> {
const recording = recordings.get(sessionId);
if (!recording || recording.settled || recording.finishing) return null;
recording.finishing = true;
await clearRearmTimersForRecording(recording, deps);
try {
await stopRecordingOnAllAgentTabs(recording, deps);
} catch {
recording.finishing = false;
return null;
}
recording.settled = true;
recordings.delete(sessionId);
releaseBrowserObservationListenersIfIdle();
const trace = buildTrace(recording);
recording.resolveFinish(trace);
return trace;
}
export async function handleRecordStart(
manager: SessionManager,
params: RecordStartParams,
deps: RecordDeps = getDefaultDeps(),
): Promise<RecordStartResult | RpcError> {
const ctxOrErr = lookupSession(manager, params, "record_start");
if (isRpcError(ctxOrErr)) return ctxOrErr;
const ctx = ctxOrErr;
if (recordings.has(params.session_id)) {
return {
code: "protocol_error",
message: `session ${params.session_id} is already recording`,
};
}
const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
// Register the recording *before* navigate so content-script syncAgentOverlay
// on the destination page can RECORD_QUERY → rearm → show RecordOverlay
// instead of flashing ControlOverlay ("Agent 正在控制").
const requestId = makeRequestId(target.tabId);
let resolveFinish!: (trace: Trace) => void;
let rejectFinish!: (err: Error) => void;
const finishPromise = new Promise<Trace>((resolve, reject) => {
resolveFinish = resolve;
rejectFinish = reject;
});
const navigateUrl = params.url ?? RECORD_DEFAULT_START_URL;
const startedAtMs = Date.now();
recordings.set(params.session_id, {
requestId,
tabId: target.tabId,
agentWindowId: ctx.agentWindowId,
startUrl: navigateUrl,
...(params.purpose ? { purpose: params.purpose } : {}),
steps: [],
startedAt: new Date(startedAtMs).toISOString(),
startedAtMs,
finishPromise,
resolveFinish,
rejectFinish,
settled: false,
finishing: false,
currentUrl: navigateUrl,
pendingNavigation: false,
pendingNavigationDeadline: undefined,
});
// Observe navigations for the whole recording lifetime; attach before
// optional navigate so the destination load can rearm capture.
ensureBrowserObservationListeners(deps);
const abortPending = async (notifyContent: boolean) => {
recordings.delete(params.session_id);
releaseBrowserObservationListenersIfIdle();
if (notifyContent) {
try {
await deps.sendToTab(target.tabId, {
type: RECORD_CANCEL,
requestId,
});
} catch {
// Content may never have received RECORD_START.
}
}
if (deps.bypassOverlay) {
try {
await deps.bypassOverlay(target.tabId, false);
} catch {
// Ignore cleanup errors.
}
}
};
const cancelledError = (): RpcError => ({
code: "cancelled",
message: "record_start aborted",
});
/** Dispatcher may race-cancel while we still hold a provisional recording. */
const abortIfCancelled = async (notifyContent: boolean): Promise<RpcError | null> => {
if (!deps.signal?.aborted) return null;
await abortPending(notifyContent);
return cancelledError();
};
{
const cancelled = await abortIfCancelled(false);
if (cancelled) return cancelled;
}
if (deps.cdp) {
const nav = await handleNavigate(
manager,
{
session_id: params.session_id,
url: navigateUrl,
tab_id: target.tabId,
},
{ cdp: deps.cdp, tabsApi: deps.tabsApi, signal: deps.signal },
);
if (isRpcError(nav)) {
await abortPending(false);
return nav;
}
{
const cancelled = await abortIfCancelled(false);
if (cancelled) return cancelled;
}
try {
await waitForTabReady(target.tabId, deps.tabsApi);
} catch {
// Proceed with retries even if the tab never reports complete.
}
{
const cancelled = await abortIfCancelled(false);
if (cancelled) return cancelled;
}
}
let startUrl: string | undefined;
try {
const tab = await deps.tabsApi.get(target.tabId);
startUrl = tab.url;
} catch {
startUrl = navigateUrl;
}
const active = recordings.get(params.session_id);
if (!active) {
// Cleared by a concurrent abort / session teardown.
return cancelledError();
}
active.startUrl = startUrl;
active.currentUrl = startUrl;
if (isContentScriptRestrictedUrl(startUrl)) {
await abortPending(false);
return {
code: "invalid_params",
message: params.url
? `cannot record on restricted URL (${startUrl}); use an http(s) page`
: `cannot record on restricted URL (${startUrl}); default start page must be injectable http(s)`,
};
}
{
const cancelled = await abortIfCancelled(false);
if (cancelled) return cancelled;
}
if (deps.bypassOverlay) {
try {
// Single ref for the initial race before RecordOverlay mounts; rearm must
// not stack additional refs (see rearmRecording). Cleared on stop.
await deps.bypassOverlay(target.tabId, true);
} catch {
// Best-effort; activeRecord also hides the control overlay.
}
}
{
const cancelled = await abortIfCancelled(true);
if (cancelled) return cancelled;
}
const startMsg: RecordStartMessage = { type: RECORD_START, requestId, startedAtMs };
try {
await sendRecordStartWithAck(target.tabId, startMsg, deps.sendToTab);
} catch {
await abortPending(true);
return {
code: "protocol_error",
message:
"failed to start recording in content script — reload the BrowserSkill extension, then retry",
};
}
{
const cancelled = await abortIfCancelled(true);
if (cancelled) return cancelled;
}
return { tab_id: target.tabId, recording: true };
}
export async function handleRecordStop(
manager: SessionManager,
params: RecordStopParams,
deps: RecordDeps = getDefaultDeps(),
): Promise<RecordStopResult | RpcError> {
const ctxOrErr = lookupSession(manager, params, "record_stop");
if (isRpcError(ctxOrErr)) return ctxOrErr;
const recording = recordings.get(params.session_id);
if (!recording) {
return {
code: "not_found",
message: `no active recording for session ${params.session_id}`,
};
}
const trace = await finishRecording(params.session_id, deps);
if (!trace) {
return {
code: "protocol_error",
message: `failed to flush recorded steps for session ${params.session_id}; the recording is still active — retry \`bsk record stop\``,
};
}
return { trace };
}
export async function handleRecordAwait(
manager: SessionManager,
params: RecordAwaitParams,
deps: RecordDeps = getDefaultDeps(),
): Promise<RecordAwaitResult | RpcError> {
const ctxOrErr = lookupSession(manager, params, "record_await");
if (isRpcError(ctxOrErr)) return ctxOrErr;
const recording = recordings.get(params.session_id);
if (!recording) {
return {
code: "not_found",
message: `no active recording for session ${params.session_id}`,
};
}
if (deps.signal?.aborted) {
return { code: "cancelled", message: "record_await aborted" };
}
const outcome = await new Promise<{ trace: Trace } | { error: RpcError }>((resolve) => {
let settled = false;
const finish = (result: { trace: Trace } | { error: RpcError }) => {
if (settled) return;
settled = true;
if (timer) clearTimeout(timer);
deps.signal?.removeEventListener("abort", onAbort);
resolve(result);
};
const onAbort = () => finish({ error: { code: "cancelled", message: "record_await aborted" } });
const timer =
params.timeout_ms === undefined
? undefined
: setTimeout(
() =>
finish({
error: {
code: "timeout",
message: `record_await timed out after ${params.timeout_ms}ms`,
},
}),
params.timeout_ms,
);
deps.signal?.addEventListener("abort", onAbort, { once: true });
void recording.finishPromise.then(
(trace) => finish({ trace }),
() =>
finish({
error: { code: "cancelled", message: "recording was cleared" },
}),
);
});
return "trace" in outcome ? { trace: outcome.trace } : outcome.error;
}
export function clearRecordingForSession(sessionId: string): void {
const recording = recordings.get(sessionId);
if (!recording) {
recordings.delete(sessionId);
releaseBrowserObservationListenersIfIdle();
return;
}
void clearRearmTimersForRecording(recording, getDefaultDeps());
if (!recording.settled) {
recording.settled = true;
recording.rejectFinish(new Error("recording cleared"));
}
recordings.delete(sessionId);
releaseBrowserObservationListenersIfIdle();
}
export type { RecordFinishMessage };
+57 -2
View File
@@ -1,10 +1,55 @@
import type { SessionManager } from "@/session-manager/manager";
import type { RpcError } from "@/transport/types";
import { clearRecordingForSession } from "./record";
import { isRpcError } from "./shared";
import { returnBorrowedTab, type TabManagementDeps } from "./tabs";
/** Valid range for Agent Window dimensions in CSS pixels. */
export const WINDOW_SIZE_MIN = 100;
export const WINDOW_SIZE_MAX = 7680;
/**
* Validate an optional width/height pair. Both must be given (or
* neither), be integers, and fall within 100..=7680. Returns the size
* to pass down, or an `invalid_params` RpcError.
*/
export function validateWindowSize(
width: unknown,
height: unknown,
): { width: number; height: number } | undefined | RpcError {
if (width === undefined && height === undefined) return undefined;
if (width === undefined || height === undefined) {
return {
code: "invalid_params",
message: "width and height must be given together",
};
}
for (const [name, value] of [
["width", width],
["height", height],
] as const) {
if (
typeof value !== "number" ||
!Number.isInteger(value) ||
value < WINDOW_SIZE_MIN ||
value > WINDOW_SIZE_MAX
) {
return {
code: "invalid_params",
message: `${name} must be an integer in 100..=7680`,
};
}
}
return { width: width as number, height: height as number };
}
export interface SessionStartParams {
session_id: string;
browser_instance_id?: string;
/** Optional Agent Window outer width in CSS pixels (100..=7680). */
width?: number;
/** Optional Agent Window outer height in CSS pixels (100..=7680). */
height?: number;
}
export interface SessionStartResult {
@@ -51,8 +96,10 @@ export async function handleSessionStart(
message: "session.start requires session_id",
};
}
const sizeOrErr = validateWindowSize(params.width, params.height);
if (isRpcError(sizeOrErr)) return sizeOrErr;
try {
const ctx = await manager.start(params.session_id);
const ctx = await manager.start(params.session_id, sizeOrErr);
return { agent_window_id: ctx.agentWindowId };
} catch (err) {
// chrome.windows.create / SessionManager failures are not CDP
@@ -113,7 +160,13 @@ export async function handleSessionStop(
const borrowedIds = Array.from(ctx.borrowedTabs.keys());
for (const tabId of borrowedIds) {
try {
const outcome = await returnBorrowedTab(ctx, tabId, deps.tabManagement ?? {});
const tabManagement = {
...(deps.tabManagement ?? {}),
isAgentWindowId:
deps.tabManagement?.isAgentWindowId ??
((windowId: number) => manager.findByWindowId(windowId) !== null),
};
const outcome = await returnBorrowedTab(ctx, tabId, tabManagement);
if (typeof outcome === "object" && "code" in outcome) {
console.warn(`[bsk session_stop] auto-return failed for tab ${tabId}`, outcome);
returnFailures?.push({
@@ -150,6 +203,8 @@ export async function handleSessionStop(
// Step 2: clear the per-session RefStore (review M6/M7 parity).
ctx.refStore.clear();
clearRecordingForSession(params.session_id);
// Step 3: detach CDP sessions this session opened (no-op if none).
await deps.cdp?.detachSession(params.session_id);
+177 -7
View File
@@ -6,13 +6,17 @@
// exactly the same sandbox + visibility rules as the M6 observation
// handlers (review parity).
import { normaliseRef } from "@/session-manager/ref-store";
import type { SessionContext, SessionManager } from "@/session-manager/manager";
import type { RpcError } from "@/transport/types";
import type { JavaScriptDialogInfo } from "@/transport/types";
import type { DialogCursor } from "@/browser-driver/chromium-cdp";
import type { SessionContext, SessionManager } from "@/session-manager/manager";
import { normaliseRef } from "@/session-manager/ref-store";
import type { ConsoleResult, JavaScriptDialogInfo, RpcError } from "@/transport/types";
import { rpcError } from "./errors";
const DEFAULT_BUFFERED_READ_LIMIT = 50;
const MAX_BUFFERED_READ_LIMIT = 200;
const DEFAULT_MAX_TEXT_CHARS = 1000;
const MAX_TEXT_CHARS = 4096;
/**
* Subset of `chrome.tabs` we depend on across tool handlers. Kept on
* a thin interface so vitest can inject a fake without monkey-patching
@@ -28,6 +32,13 @@ export const chromeTabsApi: ChromeTabsApi = {
query: (q) => chrome.tabs.query(q),
};
export interface ResolvedTargetTab {
tabId: number;
windowId: number;
active: boolean;
url?: string;
}
export type { DialogCursor };
/**
@@ -40,12 +51,67 @@ export type { DialogCursor };
*/
export interface CdpRunner {
send<T = unknown>(tabId: number, method: string, params?: object): Promise<T>;
ensureAttachedToUrl?(tabId: number, expectedUrl: string | undefined): Promise<void>;
trackSessionTab?(sessionId: string, tabId: number): void;
onEvent?(handler: (source: chrome.debugger.Debuggee, method: string, params: unknown) => void): {
dispose(): void;
};
dialogCursor?(tabId: number): DialogCursor;
dialogsSince?(tabId: number, cursor: DialogCursor): JavaScriptDialogInfo[];
ensureConsoleCapture?(tabId: number): Promise<void>;
consoleEntriesSince?(
tabId: number,
since: number | undefined,
limit: number,
maxTextChars: number,
includeStack: boolean,
): ConsoleResult;
}
export interface BufferedReadBounds {
since: number | undefined;
limit: number;
maxTextChars: number;
}
/** Parse the common cursor and output bounds used by buffered read tools. */
export function parseBufferedReadBounds(params: {
since?: number;
limit?: number;
max_text_chars?: number;
}): BufferedReadBounds | RpcError {
const since = params.since;
if (since !== undefined && (!Number.isSafeInteger(since) || since < 0)) {
return { code: "invalid_params", message: "since must be a non-negative integer" };
}
const limit = boundedOptionalInteger(
params.limit,
DEFAULT_BUFFERED_READ_LIMIT,
MAX_BUFFERED_READ_LIMIT,
"limit",
);
if (isRpcError(limit)) return limit;
const maxTextChars = boundedOptionalInteger(
params.max_text_chars,
DEFAULT_MAX_TEXT_CHARS,
MAX_TEXT_CHARS,
"max_text_chars",
);
if (isRpcError(maxTextChars)) return maxTextChars;
return { since, limit, maxTextChars };
}
function boundedOptionalInteger(
value: number | undefined,
defaultValue: number,
maxValue: number,
field: string,
): number | RpcError {
if (value === undefined) return defaultValue;
if (!Number.isSafeInteger(value) || value <= 0) {
return { code: "invalid_params", message: `${field} must be a positive integer` };
}
return Math.min(value, maxValue);
}
/**
@@ -88,7 +154,7 @@ export async function resolveTargetTab(
ctx: SessionContext,
tabId: number | undefined,
api: ChromeTabsApi,
): Promise<{ tabId: number; windowId: number; active: boolean } | RpcError> {
): Promise<ResolvedTargetTab | RpcError> {
if (tabId !== undefined) {
if (!Number.isSafeInteger(tabId) || tabId <= 0) {
return {
@@ -118,7 +184,7 @@ export async function resolveTargetTab(
message: `tab ${tabId} not found in session scope`,
};
}
return { tabId: tab.id, windowId: tab.windowId, active: tab.active === true };
return { tabId: tab.id, windowId: tab.windowId, active: tab.active === true, url: tab.url };
}
const tabs = await api.query({ active: true, windowId: ctx.agentWindowId });
const first = tabs.find((t) => typeof t.id === "number");
@@ -128,7 +194,12 @@ export async function resolveTargetTab(
message: `no active tab in Agent Window ${ctx.agentWindowId}`,
};
}
return { tabId: first.id, windowId: ctx.agentWindowId, active: first.active === true };
return {
tabId: first.id,
windowId: ctx.agentWindowId,
active: first.active === true,
url: first.url,
};
}
export function isRpcError(v: unknown): v is RpcError {
@@ -144,6 +215,91 @@ export function isRpcError(v: unknown): v is RpcError {
/** Re-export so M6/M7 tools keep a stable import path. */
export { normaliseRef };
export type ToolEffect = "passive_read" | "transient_input" | "browser_mutation";
const CDP_BLOCKED_PROTOCOLS = new Set([
"chrome:",
"chrome-extension:",
"devtools:",
"edge:",
"brave:",
"vivaldi:",
"opera:",
]);
export function cdpBlockedUrlReason(url: string | undefined): string | null {
if (!url) return null;
let parsed: URL;
try {
parsed = new URL(url);
} catch {
return null;
}
if (CDP_BLOCKED_PROTOCOLS.has(parsed.protocol)) return parsed.protocol;
if (parsed.protocol === "about:" && parsed.pathname !== "blank") return "about:";
return null;
}
/**
* Page CDP tools cannot inspect browser/extension internal pages. Check this
* at the tool boundary so Chrome's low-level "Cannot access..." errors do not
* leak as ambiguous page-read failures.
*/
export function enforceCdpAccessibleTarget(
target: ResolvedTargetTab,
toolName: string,
): RpcError | null {
const reason = cdpBlockedUrlReason(target.url);
if (!reason) return null;
return rpcError(
"permission_denied",
"restricted_tab_url",
`${toolName} cannot access tab ${target.tabId} because its URL is ${target.url}; navigate the Agent Window to a web page first`,
);
}
function resolvedTargetFromChromeTab(
tab: chrome.tabs.Tab,
fallbackWindowId: number,
): ResolvedTargetTab | null {
if (typeof tab.id !== "number") return null;
return {
tabId: tab.id,
windowId: typeof tab.windowId === "number" ? tab.windowId : fallbackWindowId,
active: tab.active === true,
url: tab.url,
};
}
/**
* Resolve a target for page CDP reads. Explicit tab_ids stay exact and are
* rejected when browser security blocks the URL. For default targeting, skip a
* restricted Agent Window active tab if another CDP-accessible tab exists in
* the same Agent Window.
*/
export async function resolveCdpAccessibleTargetTab(
manager: SessionManager,
ctx: SessionContext,
tabId: number | undefined,
api: ChromeTabsApi,
toolName: string,
): Promise<ResolvedTargetTab | RpcError> {
const target = await resolveTargetTab(manager, ctx, tabId, api);
if (isRpcError(target)) return target;
const restricted = enforceCdpAccessibleTarget(target, toolName);
if (!restricted) return target;
if (tabId !== undefined) return restricted;
const tabs = await api.query({ windowId: ctx.agentWindowId });
for (const tab of tabs) {
const candidate = resolvedTargetFromChromeTab(tab, ctx.agentWindowId);
if (!candidate) continue;
if (!enforceCdpAccessibleTarget(candidate, toolName)) return candidate;
}
return restricted;
}
/**
* Sandbox guard: M7 write tools (click / fill / press / navigate*)
* MUST refuse to touch a tab outside the session's Agent Window
@@ -166,3 +322,17 @@ export function enforceAgentWindow(
}
return null;
}
/**
* Unified target-scope policy by tool effect. Passive reads may inspect user
* tabs; any tool that dispatches page input must stay inside the Agent Window.
*/
export function enforceToolTargetScope(
ctx: SessionContext,
target: { tabId: number; windowId: number },
effect: ToolEffect,
toolName: string,
): RpcError | null {
if (effect === "passive_read") return null;
return enforceAgentWindow(ctx, target, toolName);
}
+31 -4
View File
@@ -282,6 +282,16 @@ export interface TabManagementDeps {
approveBorrow?: BorrowConfirmationApprover;
/** Clears Agent-scoped overlays after a borrowed tab is returned. */
agentOverlayReset?: AgentOverlayResetApi;
/**
* Reports whether `windowId` is any live session's Agent Window.
* `tab_return`'s fallback window picker uses this to avoid moving a
* returned borrowed tab into *another* session's Agent Window (Agent
* Windows are created with `type: "normal"`, so they otherwise match
* the `getLastFocused({ windowTypes: ["normal"] })` query). Production
* callers derive this from the `SessionManager`; the `() => false`
* default only preserves legacy behaviour for direct/test callers.
*/
isAgentWindowId?: (windowId: number) => boolean;
}
function getTabsApi(deps: TabManagementDeps): TabMutationApi {
@@ -292,6 +302,10 @@ function getWindowsApi(deps: TabManagementDeps): ChromeWindowsApi {
return deps.windows ?? chromeWindowsApi;
}
function getIsAgentWindowId(deps: TabManagementDeps): (windowId: number) => boolean {
return deps.isAgentWindowId ?? (() => false);
}
function getAgentOverlayResetApi(deps: TabManagementDeps): AgentOverlayResetApi {
return deps.agentOverlayReset ?? chromeAgentOverlayResetApi;
}
@@ -818,12 +832,20 @@ function describeError(err: unknown): string {
async function chooseFallbackWindow(
ctx: SessionContext,
windowsApi: ChromeWindowsApi,
isAgentWindowId: (windowId: number) => boolean,
): Promise<{ windowId: number; index: number } | RpcError> {
let lastFocusedError: unknown;
try {
const last = await windowsApi.getLastFocused({ windowTypes: ["normal"] });
const lastId = typeof last?.id === "number" ? last.id : null;
if (lastId !== null && lastId !== ctx.agentWindowId) {
// Never relocate a user tab into *any* session's Agent Window: this
// session's is excluded explicitly, and other sessions' are excluded
// via `isAgentWindowId` (Agent Windows are `type: "normal"`, so they
// otherwise survive the `windowTypes: ["normal"]` filter). Parking a
// returned tab in another session's Agent Window would let that
// session write to it and, worse, see it destroyed when that session
// stops and closes its window.
if (lastId !== null && lastId !== ctx.agentWindowId && !isAgentWindowId(lastId)) {
return { windowId: lastId, index: -1 };
}
} catch (err) {
@@ -888,6 +910,7 @@ export async function returnBorrowedTab(
}
const tabsApi = getTabsApi(deps);
const windowsApi = getWindowsApi(deps);
const isAgentWindowId = getIsAgentWindowId(deps);
let targetWindowId = entry.originalWindowId;
let targetIndex = entry.originalIndex;
@@ -903,7 +926,7 @@ export async function returnBorrowedTab(
}
if (!originalAlive) {
fallback = true;
const target = await chooseFallbackWindow(ctx, windowsApi);
const target = await chooseFallbackWindow(ctx, windowsApi, isAgentWindowId);
if ("code" in target) return target;
targetWindowId = target.windowId;
targetIndex = target.index;
@@ -925,7 +948,7 @@ export async function returnBorrowedTab(
};
} catch (err) {
if (!fallback) {
const target = await chooseFallbackWindow(ctx, windowsApi);
const target = await chooseFallbackWindow(ctx, windowsApi, isAgentWindowId);
if ("code" in target) {
return {
code: "cdp_failed",
@@ -979,7 +1002,11 @@ export async function handleTabReturn(
message: `tab_return: tab ${params.tab_id} is not borrowed by this session`,
};
}
const outcome = await returnBorrowedTab(ctx, params.tab_id, deps);
const outcome = await returnBorrowedTab(ctx, params.tab_id, {
...deps,
isAgentWindowId:
deps.isAgentWindowId ?? ((windowId) => manager.findByWindowId(windowId) !== null),
});
if (isRpcError(outcome)) return outcome;
ctx.borrowedTabs.delete(params.tab_id);
const result: TabReturnResult = {
@@ -0,0 +1,912 @@
import { describe, expect, it, vi } from "vitest";
import { OVERLAY_HOST_MARKER_ATTR, OVERLAY_HOST_NAME } from "../../../lib/overlay-bridge";
import { captureViewModel, collectOverlayExcludedBackendIds } from "../capture";
// Minimal but format-accurate captureSnapshot reply: a body with one
// fixed full-screen overlay div carrying a password input.
function fakeSnapshotReply() {
// strings table
const S = [
"html",
"body",
"div",
"input",
"position",
"fixed",
"static",
"pointer-events",
"auto",
"type",
"password",
];
const i = (s: string) => S.indexOf(s);
return {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1, 2],
nodeType: [1, 1, 1, 1],
nodeName: [i("html"), i("body"), i("div"), i("input")],
backendNodeId: [10, 11, 12, 13],
attributes: [[], [], [], [i("type"), i("password")]],
},
layout: {
nodeIndex: [1, 2, 3],
// styles columns follow requested computedStyles order: [position, pointer-events]
styles: [
[i("static"), i("auto")],
[i("fixed"), i("auto")],
[i("static"), i("auto")],
],
bounds: [
[0, 0, 1000, 4000],
[0, 0, 1000, 800],
[400, 300, 200, 40],
],
paintOrders: [0, 50, 51],
},
},
],
};
}
function makeCdp(snapshot: unknown) {
return {
send: vi.fn(async (_tab: number, method: string) => {
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 } };
}
if (method === "Runtime.evaluate") {
return {
result: {
value: {
controls: [
{
state: "filled",
sensitive: true,
defaultValue: "",
placeholder: "secret",
},
],
childFrames: [],
},
},
};
}
throw new Error(`unexpected ${method}`);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
}
describe("captureViewModel", () => {
it("parses nodes, attrs, rects, paint order and styles", async () => {
const { nodes, viewport, iframeNodes } = await captureViewModel(
makeCdp(fakeSnapshotReply()),
4,
);
expect(iframeNodes.size).toBe(0);
expect(viewport).toEqual({ width: 1000, height: 800 });
const div = nodes.find((n) => n.backendNodeId === 12);
expect(div).toMatchObject({ tag: "div", position: "fixed", parentBackendNodeId: 11 });
expect(div?.rect).toEqual({ x: 0, y: 0, w: 1000, h: 800 });
expect(div?.paintOrder).toBe(50);
const input = nodes.find((n) => n.backendNodeId === 13);
expect(input?.attrs).toEqual({ type: "password" });
});
it("does not run conditional surface probes by default", async () => {
const cdp = makeCdp(fakeSnapshotReply());
const result = await captureViewModel(cdp, 4);
expect(result.surfaceProbes).toEqual([]);
expect(cdp.send).not.toHaveBeenCalledWith(4, "Input.dispatchMouseEvent", expect.anything());
});
it("batch-enriches form controls without per-node object resolution", async () => {
const cdp = makeCdp(fakeSnapshotReply());
const result = await captureViewModel(cdp, 4);
const input = result.nodes.find((node) => node.backendNodeId === 13);
expect(input).toMatchObject({
formState: "filled",
formDefaultValue: "",
formPlaceholder: "secret",
});
expect(input?.formValue).toBeUndefined();
expect(cdp.send).toHaveBeenCalledWith(
4,
"Runtime.evaluate",
expect.objectContaining({ returnByValue: true }),
);
expect(cdp.send).not.toHaveBeenCalledWith(4, "DOM.resolveNode", expect.anything());
expect(cdp.send).not.toHaveBeenCalledWith(4, "Runtime.callFunctionOn", expect.anything());
});
it("runs hover surface probes when explicitly enabled", async () => {
let runtimeCalls = 0;
const cdp = {
send: vi.fn(async (_tabId: number, method: string, params?: object) => {
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return fakeSnapshotReply();
if (method === "Page.getLayoutMetrics") {
return {
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
};
}
if (method === "Runtime.evaluate") {
const expression = (params as { expression?: string } | undefined)?.expression ?? "";
if (expression.includes("input,textarea,select")) {
return { result: { value: { controls: [], childFrames: [] } } };
}
runtimeCalls += 1;
return runtimeCalls === 1
? {
result: {
value: [
{
triggerSel: ".menu",
affectedSub: " .items",
label: "Products",
x: 50,
y: 20,
},
],
},
}
: { result: { value: ["Shoes", "Bags"] } };
}
if (method === "Input.dispatchMouseEvent") return {};
throw new Error(`unexpected ${method}`);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const result = await captureViewModel(cdp, 4, { conditionalSurfaceProbe: true });
expect(result.surfaceProbes).toEqual([
{ triggerLabel: "Products", triggerAction: "hover", subItems: ["Shoes", "Bags"] },
]);
expect(cdp.send).toHaveBeenCalledWith(
4,
"Input.dispatchMouseEvent",
expect.objectContaining({ type: "mouseMoved", x: 50, y: 20 }),
);
expect(cdp.send).toHaveBeenCalledWith(
4,
"Input.dispatchMouseEvent",
expect.objectContaining({ type: "mouseMoved", x: -10, y: -10 }),
);
});
it("excludes the agent's own overlay shadow host and its inlined shadow subtree", async () => {
// DOMSnapshot inlines an open shadow root's content as descendants of the
// host. The agent's WXT overlay host carries a fixed full-viewport
// click-blocker + the "中断" button. Capturing it makes the snapshot detect
// the agent's own mask as a blocking layer and occlude the real page, so the
// whole subtree must be dropped.
const S = [
"html",
"body",
"div",
OVERLAY_HOST_NAME,
"button",
OVERLAY_HOST_MARKER_ATTR,
"position",
"fixed",
"static",
"pointer-events",
"auto",
"cursor",
"pointer",
];
const i = (s: string) => S.indexOf(s);
const snapshot = {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1, 0, 3, 4],
nodeName: [i("html"), i("body"), i("div"), i(OVERLAY_HOST_NAME), i("div"), i("button")],
backendNodeId: [10, 11, 12, 13, 14, 15],
attributes: [[], [], [], [i(OVERLAY_HOST_MARKER_ATTR), -1], [], []],
},
layout: {
nodeIndex: [0, 1, 2, 3, 4, 5],
styles: [
[i("static"), i("auto"), i("auto")],
[i("static"), i("auto"), i("auto")],
[i("static"), i("auto"), i("auto")],
[i("static"), i("auto"), i("auto")],
[i("fixed"), i("auto"), i("auto")],
[i("fixed"), i("auto"), i("pointer")],
],
bounds: [
[0, 0, 1000, 4000],
[0, 0, 1000, 800],
[10, 10, 200, 40],
[0, 0, 0, 0],
[0, 0, 1000, 800],
[400, 750, 80, 40],
],
paintOrders: [0, 1, 2, 3, 9, 10],
},
},
],
};
const { nodes } = await captureViewModel(makeCdp(snapshot), 4);
const ids = nodes.map((n) => n.backendNodeId);
// Real page nodes survive.
expect(ids).toContain(12);
// Overlay host + its inlined shadow subtree are gone.
expect(ids).not.toContain(13);
expect(ids).not.toContain(14);
expect(ids).not.toContain(15);
});
it("returns excludedBackendNodeIds for the overlay host subtree", async () => {
const S = [
"html",
"body",
"div",
OVERLAY_HOST_NAME,
"button",
OVERLAY_HOST_MARKER_ATTR,
"position",
"fixed",
"static",
"pointer-events",
"auto",
"cursor",
"pointer",
];
const i = (s: string) => S.indexOf(s);
const snapshot = {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1, 0, 3, 4],
nodeName: [i("html"), i("body"), i("div"), i(OVERLAY_HOST_NAME), i("div"), i("button")],
backendNodeId: [10, 11, 12, 13, 14, 15],
attributes: [[], [], [], [i(OVERLAY_HOST_MARKER_ATTR), -1], [], []],
},
layout: {
nodeIndex: [0, 1, 2, 3, 4, 5],
styles: [
[i("static"), i("auto"), i("auto")],
[i("static"), i("auto"), i("auto")],
[i("static"), i("auto"), i("auto")],
[i("static"), i("auto"), i("auto")],
[i("fixed"), i("auto"), i("auto")],
[i("fixed"), i("auto"), i("pointer")],
],
bounds: [
[0, 0, 1000, 4000],
[0, 0, 1000, 800],
[10, 10, 200, 40],
[0, 0, 0, 0],
[0, 0, 1000, 800],
[400, 750, 80, 40],
],
paintOrders: [0, 1, 2, 3, 9, 10],
},
},
],
};
const { excludedBackendNodeIds } = await captureViewModel(makeCdp(snapshot), 4);
expect(excludedBackendNodeIds).toEqual(new Set([13, 14, 15]));
});
it("excludes overlay host matched by tag name only", async () => {
const S = [
"html",
"body",
OVERLAY_HOST_NAME,
"position",
"static",
"pointer-events",
"auto",
"cursor",
"auto",
];
const i = (s: string) => S.indexOf(s);
const snapshot = {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1],
nodeName: [i("html"), i("body"), i(OVERLAY_HOST_NAME)],
backendNodeId: [10, 11, 13],
attributes: [[], [], []],
},
layout: {
nodeIndex: [1, 2],
styles: [
[i("static"), i("auto"), i("auto")],
[i("static"), i("auto"), i("auto")],
],
bounds: [
[0, 0, 1000, 800],
[0, 0, 100, 40],
],
paintOrders: [0, 1],
},
},
],
};
const { nodes, excludedBackendNodeIds } = await captureViewModel(makeCdp(snapshot), 4);
expect(nodes.map((n) => n.backendNodeId)).not.toContain(13);
expect(excludedBackendNodeIds).toEqual(new Set([13]));
});
it("excludes overlay host matched by marker attribute only", async () => {
const S = [
"html",
"body",
"div",
OVERLAY_HOST_MARKER_ATTR,
"position",
"static",
"pointer-events",
"auto",
"cursor",
"auto",
];
const i = (s: string) => S.indexOf(s);
const snapshot = {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1],
nodeName: [i("html"), i("body"), i("div")],
backendNodeId: [10, 11, 13],
attributes: [[], [], [i(OVERLAY_HOST_MARKER_ATTR), -1]],
},
layout: {
nodeIndex: [1, 2],
styles: [
[i("static"), i("auto"), i("auto")],
[i("static"), i("auto"), i("auto")],
],
bounds: [
[0, 0, 1000, 800],
[0, 0, 100, 40],
],
paintOrders: [0, 1],
},
},
],
};
const { nodes, excludedBackendNodeIds } = await captureViewModel(makeCdp(snapshot), 4);
expect(nodes.map((n) => n.backendNodeId)).not.toContain(13);
expect(excludedBackendNodeIds).toEqual(new Set([13]));
});
it("parses the computed cursor from the third style column", async () => {
const S = [
"html",
"body",
"div",
"position",
"static",
"fixed",
"pointer-events",
"auto",
"cursor",
"pointer",
];
const i = (s: string) => S.indexOf(s);
const snapshot = {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1],
nodeName: [i("html"), i("body"), i("div")],
backendNodeId: [10, 11, 12],
attributes: [[], [], []],
},
layout: {
nodeIndex: [1, 2],
// styles columns: [position, pointer-events, cursor]
styles: [
[i("static"), i("auto"), i("auto")],
[i("fixed"), i("auto"), i("pointer")],
],
bounds: [
[0, 0, 1000, 800],
[10, 10, 40, 40],
],
paintOrders: [0, 1],
},
},
],
};
const { nodes } = await captureViewModel(makeCdp(snapshot), 4);
expect(nodes.find((n) => n.backendNodeId === 12)?.cursor).toBe("pointer");
expect(nodes.find((n) => n.backendNodeId === 11)?.cursor).toBe("auto");
});
it("subtracts scroll offset to make rects viewport-relative", async () => {
const cdp = {
send: vi.fn(async (_t: number, method: string) => {
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return fakeSnapshotReply();
if (method === "Page.getLayoutMetrics") {
return {
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 200 },
};
}
throw new Error(method);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const { nodes } = await captureViewModel(cdp, 4);
const div = nodes.find((n) => n.backendNodeId === 12);
// local rect preserves viewport-relative scroll math.
expect(div?.localRect?.y).toBe(-200);
// exported rect is clipped to the top-level viewport before VOM consumes it.
expect(div?.rect).toMatchObject({ y: 0, h: 600 });
});
it("normalizes device-pixel bounds by devicePixelRatio", async () => {
const S = ["html", "body", "div", "position", "fixed", "static", "pointer-events", "auto"];
const i = (s: string) => S.indexOf(s);
const snapshot = {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1],
nodeName: [i("html"), i("body"), i("div")],
backendNodeId: [10, 11, 12],
attributes: [[], [], []],
},
layout: {
nodeIndex: [1, 2],
styles: [
[i("static"), i("auto")],
[i("fixed"), i("auto")],
],
bounds: [
[0, 0, 2000, 3200],
[0, 0, 2000, 1600],
],
paintOrders: [0, 50],
},
},
],
};
const cdp = {
send: vi.fn(async (_t: number, method: string) => {
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") {
return {
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
layoutViewport: { clientWidth: 2000, clientHeight: 1600 },
};
}
throw new Error(method);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const { nodes } = await captureViewModel(cdp, 4);
expect(nodes.find((n) => n.backendNodeId === 12)?.rect).toEqual({
x: 0,
y: 0,
w: 1000,
h: 800,
});
});
it("extracts textContent from #text child nodes via nodeValue", async () => {
// Simulates a button with text "登录" and an <a> with text "忘记密码".
// nodeValue is a parallel array: element nodes = -1, text nodes = string index.
const S = [
"html",
"body",
"button",
"a",
"position",
"fixed",
"static",
"pointer-events",
"auto",
"登录",
"忘记密码",
];
const i = (s: string) => S.indexOf(s);
const snapshot = {
strings: S,
documents: [
{
scrollOffsetX: 0,
scrollOffsetY: 0,
nodes: {
parentIndex: [-1, 0, 1, 2, 1, 4],
nodeName: [i("html"), i("body"), i("button"), i("#text"), i("a"), i("#text")],
backendNodeId: [10, 11, 12, 13, 14, 15],
attributes: [[], [], [], [], [], []],
nodeValue: [-1, -1, -1, i("登录"), -1, i("忘记密码")],
},
layout: {
nodeIndex: [1, 2, 4],
styles: [
[i("static"), i("auto")],
[i("static"), i("auto")],
[i("static"), i("auto")],
],
bounds: [
[0, 0, 1000, 800],
[100, 300, 200, 60],
[100, 380, 120, 36],
],
paintOrders: [0, 10, 11],
},
},
],
};
const cdp = {
send: vi.fn(async (_t: number, method: string) => {
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") {
return {
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
};
}
throw new Error(method);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const { nodes } = await captureViewModel(cdp, 4);
const btn = nodes.find((n) => n.backendNodeId === 12);
expect(btn?.textContent).toBe("登录");
const link = nodes.find((n) => n.backendNodeId === 14);
expect(link?.textContent).toBe("忘记密码");
// Nodes without text children should have no textContent
const body = nodes.find((n) => n.backendNodeId === 11);
expect(body?.textContent).toBeUndefined();
});
it("parses iframe sub-documents and returns iframeNodes keyed by iframe backendNodeId", async () => {
const S = [
"html",
"body",
"div",
"iframe",
"input",
"position",
"fixed",
"static",
"pointer-events",
"auto",
"type",
"text",
"src",
"https://passport.jd.com/login",
];
const i = (s: string) => S.indexOf(s);
const snapshot = {
strings: S,
documents: [
{
scrollOffsetX: 0,
scrollOffsetY: 0,
nodes: {
parentIndex: [-1, 0, 1, 2],
nodeName: [i("html"), i("body"), i("div"), i("iframe")],
backendNodeId: [10, 11, 12, 13],
attributes: [[], [], [], [i("src"), i("https://passport.jd.com/login")]],
// Sparse: node at array-index 3 (the iframe, backendNodeId=13) → documents[1]
contentDocumentIndex: { index: [3], value: [1] },
},
layout: {
nodeIndex: [1, 2, 3],
styles: [
[i("static"), i("auto")],
[i("fixed"), i("auto")],
[i("static"), i("auto")],
],
bounds: [
[0, 0, 1000, 4000],
[0, 0, 1000, 800],
[100, 300, 800, 400],
],
paintOrders: [0, 50, 51],
},
},
// documents[1]: the iframe's sub-document with a login input
{
scrollOffsetX: 0,
scrollOffsetY: 0,
nodes: {
parentIndex: [-1, 0],
nodeName: [i("body"), i("input")],
backendNodeId: [100, 101],
attributes: [[], [i("type"), i("text")]],
},
layout: {
nodeIndex: [1],
styles: [[i("static"), i("auto")]],
bounds: [[0, 0, 200, 40]],
paintOrders: [0],
},
},
],
};
const cdp = {
send: vi.fn(async (_t: number, method: string) => {
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") {
return {
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
};
}
throw new Error(method);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const { nodes, iframeNodes } = await captureViewModel(cdp, 4);
// Main frame has 4 nodes; iframe is included
expect(nodes.find((n) => n.backendNodeId === 13)?.tag).toBe("iframe");
// iframeNodes keyed by the <iframe> element's backendNodeId=13
expect(iframeNodes.size).toBe(1);
expect(iframeNodes.has(13)).toBe(true);
const subNodes = iframeNodes.get(13)!;
const input = subNodes.find((n) => n.backendNodeId === 101);
expect(input?.tag).toBe("input");
expect(input?.attrs.type).toBe("text");
expect(input?.ownerFrameBackendNodeId).toBe(13);
expect(input?.localRect).toEqual({ x: 0, y: 0, w: 200, h: 40 });
expect(input?.rect).toEqual({ x: 100, y: 300, w: 200, h: 40 });
});
it("clips iframe sub-document rects to the iframe viewport in top coordinates", async () => {
const S = [
"html",
"body",
"iframe",
"input",
"position",
"static",
"pointer-events",
"auto",
"type",
"text",
];
const i = (s: string) => S.indexOf(s);
const snapshot = {
strings: S,
documents: [
{
scrollOffsetX: 0,
scrollOffsetY: 0,
nodes: {
parentIndex: [-1, 0, 1],
nodeName: [i("html"), i("body"), i("iframe")],
backendNodeId: [10, 11, 13],
attributes: [[], [], []],
contentDocumentIndex: { index: [2], value: [1] },
},
layout: {
nodeIndex: [1, 2],
styles: [
[i("static"), i("auto")],
[i("static"), i("auto")],
],
bounds: [
[0, 0, 1000, 800],
[950, 760, 200, 100],
],
paintOrders: [0, 1],
},
},
{
scrollOffsetX: 0,
scrollOffsetY: 0,
nodes: {
parentIndex: [-1, 0],
nodeName: [i("body"), i("input")],
backendNodeId: [100, 101],
attributes: [[], [i("type"), i("text")]],
},
layout: {
nodeIndex: [1],
styles: [[i("static"), i("auto")]],
bounds: [[20, 20, 120, 60]],
paintOrders: [0],
},
},
],
};
const cdp = {
send: vi.fn(async (_t: number, method: string) => {
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") {
return {
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
};
}
throw new Error(method);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const { iframeNodes } = await captureViewModel(cdp, 4);
const input = iframeNodes.get(13)?.find((n) => n.backendNodeId === 101);
expect(input?.localRect).toEqual({ x: 20, y: 20, w: 120, h: 60 });
expect(input?.rect).toEqual({ x: 970, y: 780, w: 30, h: 20 });
});
it("recursively normalizes nested iframe sub-documents", async () => {
const S = [
"html",
"body",
"iframe",
"input",
"position",
"static",
"pointer-events",
"auto",
"type",
"text",
];
const i = (s: string) => S.indexOf(s);
const snapshot = {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1],
nodeName: [i("html"), i("body"), i("iframe")],
backendNodeId: [10, 11, 13],
attributes: [[], [], []],
contentDocumentIndex: { index: [2], value: [1] },
},
layout: {
nodeIndex: [1, 2],
styles: [
[i("static"), i("auto")],
[i("static"), i("auto")],
],
bounds: [
[0, 0, 1000, 800],
[10, 20, 300, 200],
],
paintOrders: [0, 1],
},
},
{
nodes: {
parentIndex: [-1, 0, 1],
nodeName: [i("html"), i("body"), i("iframe")],
backendNodeId: [20, 21, 23],
attributes: [[], [], []],
contentDocumentIndex: { index: [2], value: [2] },
},
layout: {
nodeIndex: [1, 2],
styles: [
[i("static"), i("auto")],
[i("static"), i("auto")],
],
bounds: [
[0, 0, 300, 200],
[5, 6, 100, 80],
],
paintOrders: [0, 1],
},
},
{
nodes: {
parentIndex: [-1, 0],
nodeName: [i("body"), i("input")],
backendNodeId: [30, 31],
attributes: [[], [i("type"), i("text")]],
},
layout: {
nodeIndex: [1],
styles: [[i("static"), i("auto")]],
bounds: [[1, 2, 40, 20]],
paintOrders: [0],
},
},
],
};
const cdp = {
send: vi.fn(async (_t: number, method: string) => {
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") {
return {
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
};
}
throw new Error(method);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const { iframeNodes } = await captureViewModel(cdp, 4);
const nestedIframe = iframeNodes.get(13)?.find((n) => n.backendNodeId === 23);
const input = iframeNodes.get(23)?.find((n) => n.backendNodeId === 31);
expect(nestedIframe?.rect).toEqual({ x: 15, y: 26, w: 100, h: 80 });
expect(input?.ownerFrameBackendNodeId).toBe(23);
expect(input?.rect).toEqual({ x: 16, y: 28, w: 40, h: 20 });
});
it("normalizes bounds then subtracts CSS scroll at dpr>1", async () => {
const S = ["html", "body", "div", "position", "fixed", "static", "pointer-events", "auto"];
const i = (s: string) => S.indexOf(s);
const snapshot = {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1],
nodeName: [i("html"), i("body"), i("div")],
backendNodeId: [10, 11, 12],
attributes: [[], [], []],
},
layout: {
nodeIndex: [1, 2],
styles: [
[i("static"), i("auto")],
[i("fixed"), i("auto")],
],
bounds: [
[0, 0, 2000, 3200],
[0, 400, 2000, 1600],
],
paintOrders: [0, 50],
},
},
],
};
const cdp = {
send: vi.fn(async (_t: number, method: string) => {
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") {
return {
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 100 },
layoutViewport: { clientWidth: 2000, clientHeight: 1600 },
};
}
throw new Error(method);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const { nodes } = await captureViewModel(cdp, 4);
expect(nodes.find((n) => n.backendNodeId === 12)?.rect?.y).toBe(400 / 2 - 100);
});
it("collectOverlayExcludedBackendIds walks the pierced overlay host subtree", async () => {
const cdp = {
send: vi.fn(async (_t: number, method: string) => {
if (method === "DOM.getDocument") return { root: { nodeId: 1 } };
if (method === "DOM.querySelector") return { nodeId: 99 };
if (method === "DOM.describeNode") {
return {
node: {
backendNodeId: 200,
shadowRoots: [
{ backendNodeId: 201, children: [{ backendNodeId: 202, children: [] }] },
],
},
};
}
throw new Error(method);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const excluded = await collectOverlayExcludedBackendIds(cdp, 4);
expect(excluded).toEqual(new Set([200, 201, 202]));
});
});
+803
View File
@@ -0,0 +1,803 @@
// CDP capture adapter: DOMSnapshot.captureSnapshot + Page.getLayoutMetrics
// → CapturedNode[] + Viewport. This is the ONLY VOM module that touches
// raw CDP. The captureSnapshot reply is columnar (parallel arrays + a
// shared string table); we request exactly three computed styles so the
// `styles` columns are [position, pointer-events, cursor] in that order.
import type { Rect, Viewport } from "@browser-skill/vom";
import { isOverlayHostNode, OVERLAY_HOST_SELECTOR } from "../../lib/overlay-bridge";
import type { CdpRunner } from "../shared";
const REQUESTED_STYLES = ["position", "pointer-events", "cursor"] as const;
const STYLE_COL = Object.fromEntries(
REQUESTED_STYLES.map((name, index) => [name, index]),
) as Record<(typeof REQUESTED_STYLES)[number], number>;
export interface CapturedNode {
backendNodeId: number;
parentBackendNodeId: number | null;
/** Owning iframe backend node id; `null` for the top-level document. */
ownerFrameBackendNodeId?: number | null;
tag: string;
attrs: Record<string, string>;
/** Top-level viewport-relative CSS px, clipped to the owning frame viewport. */
rect: Rect | null;
/** Frame-local viewport-relative CSS px before top-level translation/clipping. */
localRect?: Rect | null;
paintOrder: number;
position: string;
pointerEvents: string;
/**
* computed `cursor`. `cursor: pointer` is the strongest CDP-free signal
* that a non-semantic element (a `<div>`/`<span>` with a click handler)
* is actually an interactive control — used by the adapter to surface
* custom buttons/checkboxes the AX tree drops as `generic`. Optional like
* `textContent`: the live parser always sets it, hand-built fixtures may not.
*/
cursor?: string;
textContent?: string;
formState?: "empty" | "filled" | "default";
formValue?: string;
formDefaultValue?: string;
formPlaceholder?: string;
}
export type CapturedIframeNodes = Map<number, CapturedNode[]>;
export interface CapturedSurfaceProbe {
triggerLabel: string;
triggerAction: "hover" | "focus" | string;
subItems: string[];
}
export interface CapturedViewModel {
nodes: CapturedNode[];
viewport: Viewport;
iframeNodes: CapturedIframeNodes;
surfaceProbes?: CapturedSurfaceProbe[];
/** Backend node ids belonging to the agent overlay host + its shadow subtree. */
excludedBackendNodeIds: Set<number>;
}
export interface CaptureViewModelOptions {
conditionalSurfaceProbe?: boolean;
}
/** Sparse array format Chrome uses for infrequently-set per-node fields. */
interface SparseArray {
index: number[];
value: number[];
}
interface SnapshotDocument {
scrollOffsetX?: number;
scrollOffsetY?: number;
nodes?: {
parentIndex?: number[];
nodeName?: number[];
backendNodeId?: number[];
attributes?: number[][];
/**
* Per-node text value (index into strings), set for `#text` / CDATA nodes.
* Element nodes carry -1. Same length as `backendNodeId`.
*/
nodeValue?: number[];
/** Maps node array index → index into `documents[]` for frame content. */
contentDocumentIndex?: SparseArray;
};
layout?: {
nodeIndex?: number[];
styles?: number[][];
bounds?: number[][];
paintOrders?: number[];
};
}
interface SnapshotReply {
strings?: string[];
documents?: SnapshotDocument[];
}
interface LayoutMetricsReply {
cssLayoutViewport?: {
clientWidth?: number;
clientHeight?: number;
pageX?: number;
pageY?: number;
};
layoutViewport?: { clientWidth?: number; clientHeight?: number; pageX?: number; pageY?: number };
}
function isFormControlTag(tag: string): boolean {
return tag === "input" || tag === "textarea" || tag === "select";
}
const MAX_FORM_ENRICH_CONTROLS = 250;
interface CapturedFormState {
value?: string;
defaultValue?: string;
placeholder?: string;
state?: "empty" | "filled" | "default";
sensitive?: boolean;
}
interface CapturedFrameFormState {
controls: CapturedFormState[];
childFrames: CapturedFrameFormState[];
}
function formStateBatchExpression(maxControls: number): string {
return `(() => {
const maxControls = ${JSON.stringify(maxControls)};
let remaining = maxControls;
const controlSelector = "input,textarea,select";
const controlState = (el) => {
const tag = el.tagName.toLowerCase();
const type = tag === "input" ? String(el.type || "text").toLowerCase() : tag;
const sensitive = type === "password" || type === "credit-card";
const rawValue = typeof el.value === "string" ? el.value : "";
const defaultValue = typeof el.defaultValue === "string" ? el.defaultValue : "";
const placeholder = typeof el.placeholder === "string" ? el.placeholder : "";
const state = rawValue === "" ? "empty" : rawValue === defaultValue ? "default" : "filled";
return {
state,
sensitive,
defaultValue,
placeholder,
...(sensitive ? {} : { value: rawValue }),
};
};
const collect = (doc) => {
const controls = [];
for (const el of Array.from(doc.querySelectorAll(controlSelector))) {
if (remaining <= 0) break;
controls.push(controlState(el));
remaining -= 1;
}
const childFrames = [];
for (const frame of Array.from(doc.querySelectorAll("iframe"))) {
let childDoc = null;
try { childDoc = frame.contentDocument; } catch { childDoc = null; }
childFrames.push(childDoc && remaining > 0 ? collect(childDoc) : { controls: [], childFrames: [] });
}
return { controls, childFrames };
};
return collect(document);
})()`;
}
function flattenFrameFormStates(
frame: CapturedFrameFormState | undefined,
out: CapturedFormState[][] = [],
): CapturedFormState[][] {
if (!frame) return out;
out.push(Array.isArray(frame.controls) ? frame.controls : []);
for (const child of Array.isArray(frame.childFrames) ? frame.childFrames : []) {
flattenFrameFormStates(child, out);
}
return out;
}
function applyFormStates(nodes: CapturedNode[], states: CapturedFormState[]): void {
let index = 0;
for (const node of nodes) {
if (!isFormControlTag(node.tag)) continue;
const state = states[index];
index += 1;
if (!state) continue;
node.formState = state.state;
node.formDefaultValue = state.defaultValue ?? "";
node.formPlaceholder = state.placeholder ?? "";
if (!state.sensitive && state.value !== undefined) {
node.formValue = state.value;
}
}
}
async function enrichFormControlStates(
cdp: CdpRunner,
tabId: number,
frameNodeGroups: CapturedNode[][],
): Promise<void> {
const hasControls = frameNodeGroups.some((nodes) =>
nodes.some((node) => isFormControlTag(node.tag)),
);
if (!hasControls) return;
try {
const result = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: formStateBatchExpression(MAX_FORM_ENRICH_CONTROLS),
returnByValue: true,
});
const frameStates = flattenFrameFormStates(runtimeValue<CapturedFrameFormState>(result));
for (let i = 0; i < frameNodeGroups.length; i += 1) {
applyFormStates(frameNodeGroups[i], frameStates[i] ?? []);
}
} catch {
// Best-effort enrichment. DOMSnapshot/AX data still carries the nodes.
}
}
interface RuntimeEvaluateReply {
result?: {
value?: unknown;
};
}
interface HoverCandidate {
triggerSel: string;
affectedSub: string;
label: string;
x: number;
y: number;
}
interface CdpDomNode {
backendNodeId?: number;
children?: CdpDomNode[];
shadowRoots?: CdpDomNode[];
}
interface ParseDocumentResult {
nodes: CapturedNode[];
excludedBackendNodeIds: Set<number>;
}
interface FrameContext {
ownerFrameBackendNodeId: number | null;
originInTop: { x: number; y: number };
clipRectInTop: Rect;
scrollX: number;
scrollY: number;
}
function str(strings: string[], idx: number | undefined): string {
if (idx === undefined || idx < 0) return "";
return strings[idx] ?? "";
}
function devicePixelRatio(metrics: LayoutMetricsReply): number {
const layoutW = metrics.layoutViewport?.clientWidth ?? 0;
const cssW = metrics.cssLayoutViewport?.clientWidth ?? 0;
if (!layoutW || !cssW) return 1;
const dpr = layoutW / cssW;
if (!Number.isFinite(dpr) || dpr <= 0) return 1;
return dpr >= 1 ? dpr : 1;
}
function viewportRect(viewport: Viewport): Rect {
return { x: 0, y: 0, w: Math.max(0, viewport.width), h: Math.max(0, viewport.height) };
}
function intersectRects(a: Rect, b: Rect): Rect | null {
const x1 = Math.max(a.x, b.x);
const y1 = Math.max(a.y, b.y);
const x2 = Math.min(a.x + a.w, b.x + b.w);
const y2 = Math.min(a.y + a.h, b.y + b.h);
if (x2 <= x1 || y2 <= y1) return null;
return { x: x1, y: y1, w: x2 - x1, h: y2 - y1 };
}
function rectInTop(localRect: Rect, context: FrameContext): Rect | null {
const translated = {
x: context.originInTop.x + localRect.x,
y: context.originInTop.y + localRect.y,
w: localRect.w,
h: localRect.h,
};
return intersectRects(translated, context.clipRectInTop);
}
function unclipRectInTop(localRect: Rect, context: FrameContext): Rect {
return {
x: context.originInTop.x + localRect.x,
y: context.originInTop.y + localRect.y,
w: localRect.w,
h: localRect.h,
};
}
function sparseIndexMap(sparse: SparseArray | undefined): Map<number, number> {
const out = new Map<number, number>();
if (!sparse?.index || !sparse.value) return out;
for (let i = 0; i < sparse.index.length; i++) {
const nodeIndex = sparse.index[i];
const docIndex = sparse.value[i];
if (nodeIndex !== undefined && docIndex !== undefined) out.set(nodeIndex, docIndex);
}
return out;
}
function collectBackendIdsFromDomNode(node: CdpDomNode | undefined, out: Set<number>): void {
if (!node) return;
if (typeof node.backendNodeId === "number") {
out.add(node.backendNodeId);
}
for (const child of node.children ?? []) {
collectBackendIdsFromDomNode(child, out);
}
for (const shadow of node.shadowRoots ?? []) {
collectBackendIdsFromDomNode(shadow, out);
}
}
const MAX_HOVER_PROBE_MS = 2_000;
const MAX_HOVER_TRIGGERS = 8;
const HOVER_SETTLE_MS = 200;
function runtimeValue<T>(reply: RuntimeEvaluateReply): T | undefined {
return reply.result?.value as T | undefined;
}
function hoverCssScanExpression(): string {
return `(() => {
const visibilityProps = ["display", "visibility", "opacity", "maxHeight", "height", "overflow"];
const pairs = [];
const seenRules = new Set();
for (const sheet of Array.from(document.styleSheets)) {
let rules;
try { rules = sheet.cssRules; } catch { continue; }
if (!rules) continue;
for (const rule of Array.from(rules)) {
if (rule.type !== 1 || !rule.selectorText || !rule.style) continue;
const selectorText = String(rule.selectorText);
if (!selectorText.includes(":hover")) continue;
if (!visibilityProps.some((prop) => rule.style[prop])) continue;
for (const rawPart of selectorText.split(",")) {
const part = rawPart.trim();
const hoverIndex = part.indexOf(":hover");
if (hoverIndex < 0) continue;
const triggerSel = part.slice(0, hoverIndex).trim();
const affectedSub = part.slice(hoverIndex + 6).trim();
if (!triggerSel) continue;
const key = triggerSel + "||" + affectedSub;
if (seenRules.has(key)) continue;
seenRules.add(key);
pairs.push({ triggerSel, affectedSub });
}
}
}
const candidates = [];
const seenLabels = new Set();
const isUnsafeTrigger = (el) => {
if (!(el instanceof HTMLElement)) return true;
const tag = el.tagName.toLowerCase();
if (["input", "textarea", "select", "option"].includes(tag)) return true;
if (el.isContentEditable) return true;
if (el.hasAttribute("disabled") || el.hasAttribute("inert")) return true;
if ((el.getAttribute("aria-disabled") || "").toLowerCase() === "true") return true;
return false;
};
for (const pair of pairs) {
let elements;
try { elements = Array.from(document.querySelectorAll(pair.triggerSel)); } catch { continue; }
for (const el of elements) {
if (!(el instanceof Element)) continue;
if (isUnsafeTrigger(el)) continue;
const rect = el.getBoundingClientRect();
const style = getComputedStyle(el);
if (rect.width <= 0 || rect.height <= 0 || style.visibility === "hidden" || style.display === "none" || style.pointerEvents === "none") continue;
const label = (el.textContent || "").replace(/\\s+/g, " ").trim();
if (!label || seenLabels.has(label)) continue;
seenLabels.add(label);
candidates.push({
triggerSel: pair.triggerSel,
affectedSub: pair.affectedSub,
label,
x: rect.left + rect.width / 2,
y: rect.top + rect.height / 2,
});
if (candidates.length >= ${MAX_HOVER_TRIGGERS}) return candidates;
}
}
return candidates;
})()`;
}
function hoverCollectExpression(candidate: HoverCandidate): string {
return `(() => {
const triggerSel = ${JSON.stringify(candidate.triggerSel)};
const rawAffectedSub = ${JSON.stringify(candidate.affectedSub)};
const x = ${JSON.stringify(candidate.x)};
const y = ${JSON.stringify(candidate.y)};
const hit = document.elementFromPoint(x, y);
const trigger = hit instanceof Element ? hit.closest(triggerSel) : null;
if (!trigger) return [];
const affectedSub = rawAffectedSub.replace(/^[>+~]\\s*/, "").trim();
let targets = [];
if (!affectedSub) {
targets = [trigger];
} else {
try { targets = Array.from(trigger.querySelectorAll(affectedSub)); } catch { targets = []; }
}
const items = [];
const seen = new Set();
const push = (value) => {
const text = String(value || "").replace(/\\s+/g, " ").trim();
if (!text || seen.has(text)) return;
seen.add(text);
items.push(text);
};
for (const target of targets) {
if (!(target instanceof Element)) continue;
const style = getComputedStyle(target);
if (style.display === "none" || style.visibility === "hidden" || style.opacity === "0") continue;
const clickable = target.querySelectorAll('a, button, [role="menuitem"], [role="option"]');
if (clickable.length > 0) {
for (const child of Array.from(clickable)) push(child.textContent);
} else {
push(target.textContent);
}
}
return items.slice(0, 12);
})()`;
}
async function wait(ms: number): Promise<void> {
await new Promise((resolve) => setTimeout(resolve, ms));
}
async function clearHover(cdp: CdpRunner, tabId: number): Promise<void> {
await cdp
.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: -10,
y: -10,
})
.catch(() =>
cdp.send(tabId, "Input.dispatchMouseEvent", { type: "mouseMoved", x: 0, y: 0 }).catch(() => {
// best effort
}),
);
}
async function probeHoverSurfaces(cdp: CdpRunner, tabId: number): Promise<CapturedSurfaceProbe[]> {
const started = Date.now();
try {
const scan = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverCssScanExpression(),
returnByValue: true,
});
const candidates = runtimeValue<HoverCandidate[]>(scan) ?? [];
const results: CapturedSurfaceProbe[] = [];
const seen = new Set<string>();
for (const candidate of candidates.slice(0, MAX_HOVER_TRIGGERS)) {
if (Date.now() - started > MAX_HOVER_PROBE_MS) break;
if (!candidate.label || seen.has(candidate.label)) continue;
try {
await cdp.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: candidate.x,
y: candidate.y,
});
await wait(HOVER_SETTLE_MS);
const collected = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverCollectExpression(candidate),
returnByValue: true,
});
const subItems = (runtimeValue<string[]>(collected) ?? [])
.map((item) => item.replace(/\s+/g, " ").trim())
.filter(Boolean);
if (subItems.length === 0) continue;
seen.add(candidate.label);
results.push({
triggerLabel: candidate.label,
triggerAction: "hover",
subItems,
});
} catch {
continue;
} finally {
await clearHover(cdp, tabId);
}
}
return results;
} catch (err) {
console.debug("[bsk capture] hover surface probe failed", err);
return [];
}
}
/**
* When DOMSnapshot is unavailable, locate the marked overlay host via CDP and
* collect every backendNodeId in its pierced subtree (open shadow included).
*/
export async function collectOverlayExcludedBackendIds(
cdp: CdpRunner,
tabId: number,
): Promise<Set<number>> {
const excluded = new Set<number>();
try {
const doc = await cdp.send<{ root?: { nodeId?: number } }>(tabId, "DOM.getDocument", {
depth: 0,
pierce: true,
});
const rootNodeId = doc.root?.nodeId;
if (typeof rootNodeId !== "number") return excluded;
const found = await cdp.send<{ nodeId?: number }>(tabId, "DOM.querySelector", {
nodeId: rootNodeId,
selector: OVERLAY_HOST_SELECTOR,
});
if (typeof found.nodeId !== "number" || found.nodeId === 0) return excluded;
const described = await cdp.send<{ node?: CdpDomNode }>(tabId, "DOM.describeNode", {
nodeId: found.nodeId,
depth: -1,
pierce: true,
});
collectBackendIdsFromDomNode(described.node, excluded);
} catch (err) {
console.debug("[bsk capture] overlay exclusion fallback failed", err);
}
return excluded;
}
/**
* Parse one DOMSnapshot document into CapturedNode[].
*
* @param doc - the raw DOMSnapshot document object
* @param strings - the shared string table for the whole snapshot
* @param dpr - device-pixel-ratio from Page.getLayoutMetrics
* @param context - frame coordinate context; output rects are top viewport-relative
*/
function parseDocumentNodes(
doc: SnapshotDocument,
strings: string[],
dpr: number,
context: FrameContext,
): ParseDocumentResult {
const dn = doc.nodes;
const dl = doc.layout;
if (!dn?.backendNodeId) {
return { nodes: [], excludedBackendNodeIds: new Set() };
}
const count = dn.backendNodeId.length;
const layoutByNode = new Map<number, number>();
(dl?.nodeIndex ?? []).forEach((nodeIdx, layoutIdx) => layoutByNode.set(nodeIdx, layoutIdx));
// The agent's own overlay (WXT shadow host, marked with OVERLAY_HOST_MARKER_ATTR)
// is injected into the page and its open shadow root
// is inlined by DOMSnapshot as descendants of the host. Its fixed
// full-viewport click-blocker would otherwise dominate occlusion detection
// and hide the real page, so mark the host + its whole subtree excluded.
// parentIndex always references an earlier node, so one forward pass suffices.
const excluded = new Array<boolean>(count).fill(false);
const excludedBackendNodeIds = new Set<number>();
for (let n = 0; n < count; n++) {
const parentIdx = dn.parentIndex?.[n] ?? -1;
const inherited = parentIdx >= 0 && excluded[parentIdx];
let isOverlayHost = inherited;
if (!isOverlayHost) {
const pairs = dn.attributes?.[n] ?? [];
const attrNames: string[] = [];
for (let a = 0; a + 1 < pairs.length; a += 2) {
attrNames.push(str(strings, pairs[a]));
}
isOverlayHost = isOverlayHostNode(str(strings, dn.nodeName?.[n]), attrNames);
}
if (!isOverlayHost) continue;
excluded[n] = true;
const backendNodeId = dn.backendNodeId[n];
if (backendNodeId !== undefined) {
excludedBackendNodeIds.add(backendNodeId);
}
}
const posCol = STYLE_COL.position;
const peCol = STYLE_COL["pointer-events"];
const cursorCol = STYLE_COL.cursor;
// Collect visible text from #text child nodes so element CapturedNodes
// carry a textContent value usable as a button/link label fallback.
// nodeValue is a parallel array: string index for text nodes, -1 otherwise.
const nodeTextContent = new Map<number, string>();
if (dn.nodeValue) {
for (let n = 0; n < count; n++) {
const nvIdx = dn.nodeValue[n] ?? -1;
if (nvIdx < 0) continue;
const text = str(strings, nvIdx).trim();
if (!text) continue;
const parentIdx = dn.parentIndex?.[n] ?? -1;
if (parentIdx >= 0) {
const existing = nodeTextContent.get(parentIdx);
nodeTextContent.set(parentIdx, existing ? `${existing} ${text}` : text);
}
}
}
// Build CapturedNodes.
const nodes: CapturedNode[] = [];
for (let n = 0; n < count; n++) {
if (excluded[n]) continue;
const backendNodeId = dn.backendNodeId[n];
const parentIdx = dn.parentIndex?.[n] ?? -1;
const parentBackendNodeId = parentIdx >= 0 ? (dn.backendNodeId[parentIdx] ?? null) : null;
const tag = str(strings, dn.nodeName?.[n]).toLowerCase();
const attrs: Record<string, string> = {};
const pairs = dn.attributes?.[n] ?? [];
for (let a = 0; a + 1 < pairs.length; a += 2) {
attrs[str(strings, pairs[a]).toLowerCase()] = str(strings, pairs[a + 1]);
}
let rect: Rect | null = null;
let localRect: Rect | null = null;
let paintOrder = 0;
let position = "static";
let pointerEvents = "auto";
let cursor = "auto";
const li = layoutByNode.get(n);
if (li !== undefined) {
const b = dl?.bounds?.[li];
if (b && b.length >= 4 && b[2] > 0 && b[3] > 0) {
localRect = {
x: b[0] / dpr - context.scrollX,
y: b[1] / dpr - context.scrollY,
w: b[2] / dpr,
h: b[3] / dpr,
};
rect = rectInTop(localRect, context);
}
paintOrder = dl?.paintOrders?.[li] ?? 0;
const styleRow = dl?.styles?.[li] ?? [];
position = str(strings, styleRow[posCol]) || "static";
pointerEvents = str(strings, styleRow[peCol]) || "auto";
cursor = str(strings, styleRow[cursorCol]) || "auto";
}
// Skip non-element nodes (#text, #cdata-section, etc.) — they carry no
// geometry and are never queried by callers.
if (tag.startsWith("#")) continue;
const textContent = nodeTextContent.get(n);
nodes.push({
backendNodeId,
parentBackendNodeId,
ownerFrameBackendNodeId: context.ownerFrameBackendNodeId,
tag,
attrs,
rect,
localRect,
paintOrder,
position,
pointerEvents,
cursor,
textContent,
});
}
return { nodes, excludedBackendNodeIds };
}
interface ParseFrameDocumentsResult {
iframeNodes: CapturedIframeNodes;
excludedBackendNodeIds: Set<number>;
}
function parseChildFrameDocuments(
documents: SnapshotDocument[],
strings: string[],
dpr: number,
parentDocIndex: number,
parentNodes: CapturedNode[],
parentContext: FrameContext,
visited = new Set<number>(),
): ParseFrameDocumentsResult {
const iframeNodes: CapturedIframeNodes = new Map();
const excludedBackendNodeIds = new Set<number>();
const parentDoc = documents[parentDocIndex];
const cdi = sparseIndexMap(parentDoc?.nodes?.contentDocumentIndex);
if (cdi.size === 0) return { iframeNodes, excludedBackendNodeIds };
const parentBackendIds = parentDoc?.nodes?.backendNodeId ?? [];
const parentNodeByBackendId = new Map(parentNodes.map((node) => [node.backendNodeId, node]));
for (const [nodeArrayIdx, childDocIndex] of cdi) {
if (visited.has(childDocIndex)) continue;
const childDoc = documents[childDocIndex];
if (!childDoc) continue;
const iframeBackendId = parentBackendIds[nodeArrayIdx];
if (iframeBackendId === undefined) continue;
const iframeNode = parentNodeByBackendId.get(iframeBackendId);
if (!iframeNode?.localRect) continue;
const iframeRectInTop = unclipRectInTop(iframeNode.localRect, parentContext);
const childClip = intersectRects(iframeRectInTop, parentContext.clipRectInTop);
if (!childClip) {
iframeNodes.set(iframeBackendId, []);
continue;
}
const childContext: FrameContext = {
ownerFrameBackendNodeId: iframeBackendId,
originInTop: { x: iframeRectInTop.x, y: iframeRectInTop.y },
clipRectInTop: childClip,
scrollX: childDoc.scrollOffsetX ?? 0,
scrollY: childDoc.scrollOffsetY ?? 0,
};
const nextVisited = new Set(visited);
nextVisited.add(childDocIndex);
const parsed = parseDocumentNodes(childDoc, strings, dpr, childContext);
iframeNodes.set(iframeBackendId, parsed.nodes);
for (const id of parsed.excludedBackendNodeIds) excludedBackendNodeIds.add(id);
const nested = parseChildFrameDocuments(
documents,
strings,
dpr,
childDocIndex,
parsed.nodes,
childContext,
nextVisited,
);
for (const [nestedFrameId, nestedNodes] of nested.iframeNodes) {
iframeNodes.set(nestedFrameId, nestedNodes);
}
for (const id of nested.excludedBackendNodeIds) excludedBackendNodeIds.add(id);
}
return { iframeNodes, excludedBackendNodeIds };
}
export async function captureViewModel(
cdp: CdpRunner,
tabId: number,
options: CaptureViewModelOptions = {},
): Promise<CapturedViewModel> {
const metrics = await cdp.send<LayoutMetricsReply>(tabId, "Page.getLayoutMetrics", {});
const dpr = devicePixelRatio(metrics);
const vpSrc = metrics.cssLayoutViewport ?? metrics.layoutViewport ?? {};
const viewport: Viewport = {
width: vpSrc.clientWidth ?? 0,
height: vpSrc.clientHeight ?? 0,
};
const scrollX = vpSrc.pageX ?? 0;
const scrollY = vpSrc.pageY ?? 0;
await cdp.send(tabId, "DOMSnapshot.enable", {});
const snap = await cdp.send<SnapshotReply>(tabId, "DOMSnapshot.captureSnapshot", {
computedStyles: REQUESTED_STYLES,
includePaintOrder: true,
includeDOMRects: true,
});
const strings = snap.strings ?? [];
const documents = snap.documents ?? [];
const doc0 = documents[0];
if (!doc0?.nodes?.backendNodeId) {
return {
nodes: [],
viewport,
iframeNodes: new Map(),
surfaceProbes: [],
excludedBackendNodeIds: new Set(),
};
}
const topContext: FrameContext = {
ownerFrameBackendNodeId: null,
originInTop: { x: 0, y: 0 },
clipRectInTop: viewportRect(viewport),
scrollX,
scrollY,
};
const mainParsed = parseDocumentNodes(doc0, strings, dpr, topContext);
const nodes = mainParsed.nodes;
const excludedBackendNodeIds = new Set(mainParsed.excludedBackendNodeIds);
const frameParsed = parseChildFrameDocuments(documents, strings, dpr, 0, nodes, topContext);
const iframeNodes = frameParsed.iframeNodes;
for (const id of frameParsed.excludedBackendNodeIds) {
excludedBackendNodeIds.add(id);
}
await enrichFormControlStates(cdp, tabId, [nodes, ...iframeNodes.values()]);
const surfaceProbes = options.conditionalSurfaceProbe ? await probeHoverSurfaces(cdp, tabId) : [];
return { nodes, viewport, iframeNodes, surfaceProbes, excludedBackendNodeIds };
}
+1 -1
View File
@@ -20,6 +20,7 @@ import type {
WaitForNavigationResult,
WaitUntil,
} from "@/transport/types";
import { attachDialogs, markDialogCursor } from "./dialogs";
import { cdpLifecycleName, ensureCdpReady, waitForLifecyclePassive } from "./navigation";
import {
type CdpRunner,
@@ -30,7 +31,6 @@ import {
lookupSession,
resolveTargetTab,
} from "./shared";
import { attachDialogs, markDialogCursor } from "./dialogs";
export interface WaitForNavigationDeps {
cdp: CdpRunner;
+82
View File
@@ -0,0 +1,82 @@
// Window-tool handlers: `tool.window_resize` resizes the session's
// Agent Window via `chrome.windows.update`.
import type { SessionManager } from "@/session-manager/manager";
import type { RpcError } from "@/transport/types";
import { validateWindowSize } from "./session";
import { isRpcError, lookupSession } from "./shared";
/**
* Mirror of bsk-protocol `WindowResizeParams` /
* `WindowResizeResult` (see crates/bsk-protocol/src/tools/window.rs).
*/
export interface WindowResizeParams {
session_id: string;
width: number;
height: number;
}
export interface WindowResizeResult {
window_id: number;
width: number;
height: number;
}
/**
* Subset of `chrome.windows` we depend on, injectable so unit tests
* can fake the browser without monkey-patching the global `chrome`.
*/
export interface WindowResizeApi {
update(windowId: number, updateInfo: { width: number; height: number }): Promise<unknown>;
}
export const chromeWindowResizeApi: WindowResizeApi = {
async update(windowId, updateInfo) {
return chrome.windows.update(windowId, updateInfo);
},
};
/**
* Handler for `tool.window_resize` (called by the daemon over WS).
*
* Resizes the session's Agent Window to the given outer dimensions in
* CSS pixels. chrome API failures are surfaced as `protocol_error`
* (§4.5 reserves `cdp_failed` for raw CDP errors).
*/
export async function handleWindowResize(
manager: SessionManager,
params: WindowResizeParams,
api: WindowResizeApi = chromeWindowResizeApi,
): Promise<WindowResizeResult | RpcError> {
const ctxOrErr = lookupSession(manager, params, "window_resize");
if (isRpcError(ctxOrErr)) return ctxOrErr;
const ctx = ctxOrErr;
const sizeOrErr = validateWindowSize(params.width, params.height);
if (isRpcError(sizeOrErr)) return sizeOrErr;
if (!sizeOrErr) {
// validateWindowSize only returns undefined when both are absent,
// which is valid for session_start but not for an explicit resize.
return {
code: "invalid_params",
message: "window_resize requires width and height",
};
}
try {
await api.update(ctx.agentWindowId, {
width: sizeOrErr.width,
height: sizeOrErr.height,
});
} catch (err) {
return {
code: "protocol_error",
message: err instanceof Error ? err.message : String(err),
};
}
return {
window_id: ctx.agentWindowId,
width: sizeOrErr.width,
height: sizeOrErr.height,
};
}
+7
View File
@@ -5,3 +5,10 @@
// at runtime (review M3 fix to round-1).
declare const __BSK_EXT_VERSION__: string;
/**
* WebSocket URL the extension uses to connect to the local bsk daemon.
* Defaults to {@code ws://127.0.0.1:52800}. Override at build time by
* setting the {@code BSK_DAEMON_WS_URL} environment variable.
*/
declare const __BSK_DAEMON_WS_URL__: string;
+286 -1
View File
@@ -30,6 +30,7 @@ export type RpcErrorReason =
| "option_not_found"
| "single_select_value_count"
| "tab_not_active"
| "restricted_tab_url"
| "borrow_conflict";
export interface RpcErrorData {
@@ -134,6 +135,75 @@ export interface JavaScriptDialogInfo {
sequence: number;
}
export type ConsoleEntryKind = "console" | "exception" | "log";
export interface ConsoleStackFrame {
function_name?: string;
url?: string;
line?: number;
column?: number;
}
export interface ConsoleEntry {
sequence: number;
kind: ConsoleEntryKind;
level: string;
text: string;
url?: string;
line?: number;
column?: number;
timestamp?: number;
stack_trace?: ConsoleStackFrame[];
truncated: boolean;
}
export interface ConsoleParams {
session_id: string;
tab_id?: number;
since?: number;
limit?: number;
max_text_chars?: number;
include_stack?: boolean;
}
export interface ConsoleResult {
tab_id: number;
entries: ConsoleEntry[];
next_since: number;
truncated: boolean;
}
export type NetworkEntryKind = "response" | "failure";
export interface NetworkEntry {
sequence: number;
kind: NetworkEntryKind;
method?: string;
url?: string;
status?: number;
status_text?: string;
mime_type?: string;
resource_type?: string;
error_text?: string;
timestamp?: number;
truncated: boolean;
}
export interface NetworkParams {
session_id: string;
tab_id?: number;
since?: number;
limit?: number;
max_text_chars?: number;
}
export interface NetworkResult {
tab_id: number;
entries: NetworkEntry[];
next_since: number;
truncated: boolean;
}
export type TabScopeFilter = "user" | "agent" | "all";
export interface TabInfo {
@@ -244,6 +314,9 @@ export interface SnapshotResult {
dialogs?: JavaScriptDialogInfo[];
}
export type ObserveParams = SnapshotParams;
export type ObserveResult = SnapshotResult;
export interface GetHtmlParams {
session_id: string;
tab_id?: number;
@@ -445,7 +518,28 @@ export interface HelpTarget {
selector?: string;
}
export type HelpOutcome = "continued" | "cancelled" | "timed_out" | "navigated";
export type HelpOutcome =
| "continued"
| "cancelled"
| "timed_out"
| "completed"
| "navigated"
| "disabled";
export interface HelpCompletionCondition {
url_contains?: string;
url_matches?: string;
selector_exists?: string;
selector_missing?: string;
text_exists?: string;
text_missing?: string;
}
export interface HelpCompletionCriteria {
any?: HelpCompletionCondition[];
all?: HelpCompletionCondition[];
stable_for_ms?: number;
}
export interface ResolvedTarget {
matched: boolean;
@@ -459,12 +553,203 @@ export interface RequestHelpParams {
prompt: string;
title?: string;
targets?: HelpTarget[];
completion_criteria?: HelpCompletionCriteria;
timeout_ms?: number;
}
export interface RequestHelpResult {
outcome: HelpOutcome;
completed_by?: "system";
note?: string;
tab_id: number;
resolved_targets?: ResolvedTarget[];
}
// --------------------------------------------------------------------------
// Device-emulation payloads — tool.emulate (mirror bsk-protocol emulate.rs)
// --------------------------------------------------------------------------
export interface UserAgentBrandVersion {
brand: string;
version: string;
}
/** Mirror of CDP `Emulation.UserAgentMetadata`; all fields optional. */
export interface UserAgentMetadata {
brands?: UserAgentBrandVersion[];
full_version?: string;
platform?: string;
platform_version?: string;
architecture?: string;
model?: string;
mobile?: boolean;
}
/**
* Concrete emulation overrides for one tab. The extension merges each
* request field by field onto the tab's remembered emulation state:
* fields present here overwrite the stored value, absent fields keep
* it, and the merged state is applied as a whole.
*/
export interface EmulateOverrides {
width?: number;
height?: number;
device_scale_factor?: number;
mobile?: boolean;
user_agent?: string;
accept_language?: string;
user_agent_metadata?: UserAgentMetadata;
touch?: boolean;
max_touch_points?: number;
}
export interface EmulateParams {
session_id: string;
tab_id?: number;
/** Clear every emulation override on the tab. Exclusive with `overrides`. */
off?: boolean;
/** Overrides to apply. Required unless `off` is set. */
overrides?: EmulateOverrides;
}
export interface EmulateResult {
tab_id: number;
/** True when overrides were cleared (`off`); false when applied. */
cleared: boolean;
/** Echo of the overrides that were applied. Absent when cleared. */
applied?: EmulateOverrides;
/** Scope note: overrides are per-tab (CDP target), not inherited by new tabs. */
note?: string;
}
// --------------------------------------------------------------------------
// Semantic record payloads — mirror bsk-protocol record.rs (Trace v2)
// --------------------------------------------------------------------------
export interface TargetDescriptor {
role?: string;
name?: string;
tag: string;
name_attr?: string;
placeholder?: string;
nearby_label?: string;
}
export interface TraceEntry {
start_url: string;
}
export interface PageRef {
id: string;
url: string;
title?: string;
}
export interface SelectedOption {
value: string;
label?: string;
}
export interface StepEffect {
navigated_to: string;
}
export interface StepCommon {
id: number;
page: string;
effect?: StepEffect;
}
/** Capture/buffer draft before v2 reduction. */
export type DraftTraceStep =
| {
op: "click";
target: TargetDescriptor;
navigated_to?: string;
page_url?: string;
}
| {
op: "fill";
target: TargetDescriptor;
value: string;
redacted?: boolean;
page_url?: string;
}
| {
op: "press";
key: string;
target?: TargetDescriptor;
modifiers?: KeyModifier[];
navigated_to?: string;
page_url?: string;
}
| {
op: "select";
target: TargetDescriptor;
values: string[];
labels?: string[];
navigated_to?: string;
page_url?: string;
}
| {
op: "navigate";
url: string;
page_url?: string;
};
/** Exported record-only step (trace v2). */
export type Step =
| ({ op: "navigate" } & StepCommon & { to: string })
| ({ op: "click" } & StepCommon & { target: TargetDescriptor })
| ({ op: "fill" } & StepCommon & {
target: TargetDescriptor;
value: string;
redacted?: boolean;
})
| ({ op: "select" } & StepCommon & {
target: TargetDescriptor;
selection: SelectedOption[];
})
| ({ op: "press" } & StepCommon & {
key: string;
modifiers?: KeyModifier[];
target?: TargetDescriptor;
});
export interface Trace {
recorded_at: string;
started_at?: string;
purpose?: string;
entry: TraceEntry;
pages: PageRef[];
steps: Step[];
}
export interface RecordStartParams {
session_id: string;
tab_id?: number;
url?: string;
purpose?: string;
}
export interface RecordStartResult {
tab_id: number;
recording: boolean;
}
export interface RecordStopParams {
session_id: string;
}
export interface RecordStopResult {
trace: Trace;
}
export interface RecordAwaitParams {
session_id: string;
timeout_ms?: number;
}
export interface RecordAwaitResult {
trace: Trace;
}
+1
View File
@@ -21,6 +21,7 @@ export default defineConfig({
},
define: {
__BSK_EXT_VERSION__: JSON.stringify(pkg.version),
__BSK_DAEMON_WS_URL__: JSON.stringify(process.env.BSK_DAEMON_WS_URL ?? "ws://127.0.0.1:52800"),
},
resolve: {
alias: {
+17 -1
View File
@@ -25,7 +25,20 @@ export default defineConfig({
name: "BrowserSkill",
description:
"Let AI agents use your logged-in browser in a separate Agent Window—without interrupting your work. Powered by the bsk CLI.",
permissions: ["alarms", "debugger", "notifications", "tabs", "storage", "windows"],
// Chrome 116 is the first version where WebSocket activity resets the
// service-worker idle timer, which the 20s heartbeat relies on to
// keep the daemon link alive.
minimum_chrome_version: "116",
permissions: [
"alarms",
"debugger",
"idle",
"notifications",
"tabs",
"storage",
"webNavigation",
"windows",
],
host_permissions: ["<all_urls>"],
icons: {
16: "icon/logo.png",
@@ -59,6 +72,9 @@ export default defineConfig({
],
define: {
__BSK_EXT_VERSION__: JSON.stringify(EXTENSION_VERSION),
__BSK_DAEMON_WS_URL__: JSON.stringify(
process.env.BSK_DAEMON_WS_URL ?? "ws://127.0.0.1:52800",
),
},
resolve: {
alias: {
+5
View File
@@ -48,6 +48,11 @@ fs2 = { workspace = true }
rand = { workspace = true }
uuid = { workspace = true }
bsk-protocol = { workspace = true }
reqwest = { workspace = true }
flate2 = { workspace = true }
tar = { workspace = true }
zip = { workspace = true }
sha2 = { workspace = true }
[target.'cfg(unix)'.dependencies]
nix = { workspace = true }
+86 -12
View File
@@ -46,6 +46,21 @@ Optional: `bsk session start --browser <instance-id-or-label>` when multiple bro
Emergency cleanup: `bsk session stop --all` or the Agent Window overlay **Stop all**.
## Stop when the goal is met
Every task is a **bounded goal**, not open-ended browsing. The goal may come from the user's request, a recorded `trace.json`, or both.
1. **Define success first** — one concrete, observable condition derived from the user's words, `purpose`, or the last meaningful step in a trace (e.g. "form submitted", "item added to cart", "playback started").
2. **Take the shortest path** — snapshot → act → at most one check. Do not wander, re-try unrelated actions, or stack exploratory steps.
3. **Stop as soon as success is reached** — run `bsk session stop <id>` immediately unless the user explicitly asked to keep the session open (e.g. "don't close yet", "keep browsing").
4. **No post-success work** — once the goal is met, do not click, refresh, navigate, re-search, switch tabs, or "double-check" that it worked. Further verification is a new task.
5. **When blocked, pause — do not brute-force** — if the page requires human input (login, captcha, OTP, payment confirmation) or an action fails twice with no progress, call `bsk request-help` instead of retrying blindly. See **Ask the human for help** below.
6. **When unsure** — at most one extra `bsk snapshot`. If success looks met, stop. If not, ask the user; do not keep clicking.
**With a trace:** replay steps in order using `target` role/name/tag and raw `value`/`selection` fields. After the last step (or when its `effect.navigated_to` / success hint is satisfied), apply rules 3–4 immediately. The trace guides execution; it does not extend control beyond the goal.
**Without a trace:** the user's request *is* the success condition. Satisfying it ends the task — same stop rules apply.
## Core interaction loop
Write operations only affect tabs in the **Agent Window** (or tabs you **borrowed** into it).
@@ -53,6 +68,7 @@ Write operations only affect tabs in the **Agent Window** (or tabs you **borrowe
```
bsk navigate <url> --session <id>
bsk snapshot --session <id> → aria tree with @e1, @e2, … refs
bsk observe --session <id> → semantic VOM view; may reveal hover/focus surfaces
bsk click @e3 --session <id> → or bsk fill, bsk select, bsk press
bsk snapshot --session <id> → again after navigation / DOM change
```
@@ -63,11 +79,12 @@ Prefer `@eN` refs from the latest snapshot over raw CSS selectors. Use `--ref` /
## Observation priority
Start with `bsk snapshot` to understand page structure, text, controls, and element refs. Only escalate when the latest snapshot cannot answer the question:
Start with `bsk snapshot` to understand page structure, text, controls, and element refs. Use `bsk observe` when semantic VOM output or conditional hover/focus surfaces would materially help. Only escalate to raw HTML or screenshots when the latest observation cannot answer the question:
1. `bsk snapshot` — default for page understanding and interaction planning
2. `bsk get-html` — when hidden DOM, metadata, or markup details are required
3. `bsk screenshot` — when visual layout, canvas/image content, or styling cannot be inferred from the snapshot. Use `--ref @eN` (from the latest snapshot) to crop to one element; omit `--ref` for the full visible tab.
1. `bsk snapshot` — strict static page understanding and interaction planning
2. `bsk observe` — semantic VOM observation; may run bounded perception probes such as hover-surface discovery
3. `bsk get-html` — when hidden DOM, metadata, or markup details are required
4. `bsk screenshot` — when visual layout, canvas/image content, or styling cannot be inferred from the observation. Use `--ref @eN` (from the latest snapshot/observe) to crop to one element; omit `--ref` for the full visible tab.
Do **not** call `bsk get-html` or `bsk screenshot` first just to inspect a page.
@@ -106,11 +123,33 @@ Details and flags: **`bsk <cmd> --help`**
| Command | Summary |
|---------|---------|
| `bsk session start` | Open Agent Window; prints **4-letter session id** |
| `bsk session start` | Open Agent Window (`--width`/`--height` for initial size); prints **4-letter session id** |
| `bsk session stop <id>` | End session, close Agent Window, auto-return borrowed tabs |
| `bsk session stop --all` | Stop every active session |
| `bsk session list` | List active sessions |
### Window (require `--session <id>`)
| Command | Summary |
|---------|---------|
| `bsk window resize` | Resize the Agent Window (`--width`, `--height`; 100..=7680 CSS px) |
### Device emulation — `bsk emulate` (requires `--session <id>`)
Emulate a mobile device environment on the agent tab via CDP — viewport, User-Agent, and touch — to debug a page's mobile layout and behaviour:
```bash
bsk emulate --session <id> --device iphone-14
bsk emulate --session <id> --width 390 --height 844 --dpr 3 --mobile --ua "Mozilla/5.0 (iPhone…" --touch
bsk emulate --session <id> --off
```
- Presets (`--device`): `iphone-14`, `iphone-14-pro-max`, `iphone-se`, `pixel-7`, `galaxy-s23`, `ipad-mini`, `galaxy-tab-s8`. Manual flags (`--width`/`--height`/`--dpr`/`--mobile`/`--ua`/`--accept-language`/`--touch`/`--max-touch-points`) also work without a preset, or override individual preset fields; `--no-mobile`/`--no-touch` turn a preset's mobile viewport / touch emulation back off.
- Repeated runs merge field by field onto the tab's current emulation state — only the flags you pass change. E.g. after `--device iphone-14`, `bsk emulate --session <id> --width 390 --height 844` keeps the preset's dpr (`3`) and mobile viewport. (The extension remembers the state per tab; after an extension reload the next run applies only the fields it carries.)
- `--off` clears every override (viewport, UA, touch) and the remembered state, restoring the tab's real environment.
- Scope: overrides apply to **one tab only** (CDP per-target) and are **not inherited by new tabs** — re-run `bsk emulate` after opening or switching to another tab (default target is the session's active tab; `--tab-id` overrides).
- Emulation covers viewport/UA/touch only; it does not throttle the network, fake geolocation, or synthesise real touch-event streams.
### Tabs (require `--session <id>`)
| Command | Summary |
@@ -126,7 +165,8 @@ Details and flags: **`bsk <cmd> --help`**
| Command | Summary |
|---------|---------|
| `bsk snapshot` | First-choice page understanding: accessibility tree with `@eN` element refs |
| `bsk snapshot` | First-choice static page understanding: accessibility tree with `@eN` element refs |
| `bsk observe` | Semantic VOM observation with bounded perception probes for conditional surfaces |
| `bsk get-html` | Raw HTML dump after snapshot is insufficient (high token cost) |
| `bsk screenshot` | PNG capture after snapshot is insufficient: full visible tab, or `--ref @eN` to crop to one element (`--out` path optional) |
@@ -163,7 +203,7 @@ Details and flags: **`bsk <cmd> --help`**
When a step needs a human (captcha, login, OTP) or you want the user to
confirm an important action, pause and ask:
bsk request-help --session <id> --prompt "Solve the captcha, then click Continue" \
bsk request-help --session <id> --prompt "Solve the captcha, then click Done only after the site accepts it" \
--title "Captcha required" --target @e7 --target "#submit" --timeout 5m
- `--prompt` (required): what the user should do.
@@ -178,19 +218,52 @@ confirm an important action, pause and ask:
prompt with no `--target` for cases where there is genuinely no specific
element to point at (e.g. "wait for the page to finish loading").
- `--timeout` (default `5m`): how long to wait.
- `--completion-criteria` (optional): JSON success detector. Use it only
when there is a concrete post-help success signal, e.g.
`{"any":[{"url_contains":"/dashboard"},{"selector_exists":"[data-testid='account-menu']"}],"stable_for_ms":1000}`.
The target tab is brought to the foreground; the page stays interactive
while the agent control mask is hidden. The call blocks until the user
acts. The result `outcome` is one of:
explicitly acts, the timeout expires, cancellation arrives, or explicit
completion criteria match. Page reloads, SPA route changes, and captcha
refreshes do not return control by themselves. The result `outcome` is one of:
- `continued` — the user finished and clicked Continue (treat as confirm).
- `continued` — the user finished and clicked Done / return control (treat as confirm).
- `cancelled` — the user clicked Cancel (treat as reject/abort).
- `timed_out` — nobody acted within the timeout.
- `navigated` — the page navigated while waiting (full reload or SPA URL change). Snapshot refs are stale; run `bsk snapshot` on the new page, then decide whether to call `bsk request-help` again.
- `completed` — the explicit `--completion-criteria` matched while the user had control.
- `navigated` — deprecated legacy outcome. Do not rely on navigation as a completion signal.
`note` carries any text the user typed back. `resolved_targets` reports
which refs/selectors matched a live element.
`request-help` does not refresh the page model after the user returns
control. After a `continued` or `completed` result, issue a separate
observation tool call (usually `bsk snapshot --session <id>`) before using
new refs or reasoning about the post-help page state.
#### Disabling request-help (unattended mode)
Set `BSK_REQUEST_HELP=off` on unattended servers: `bsk request-help` then
returns immediately with `outcome="disabled"` (no overlay, no waiting,
exit 0). Any other value keeps it enabled. If you get `disabled`, do not
retry — complete the task autonomously or stop gracefully.
### Recording — `bsk record`
Capture the user's own actions in the Agent Window to a `trace.json`, for later LLM-driven automation:
```bash
bsk record start --browser <instance-id-or-label> [--url https://…] [--purpose "publish a wiki doc"] [--output trace.json]
# `--url` is optional; default https://example.com/ when omitted (must be http(s)).
# Blocks until the user clicks Finish in the recording panel, then writes ./trace.json and closes the window.
bsk record stop [--output trace.json] # terminal fallback if the browser panel is unavailable
```
- The trace is a **record-only action log** (a `pages[]` dictionary + `navigate`/`click`/`fill`/`select`/`press` steps with `target` descriptors). It records *what the user did*; deciding which inputs are variable is left to the executing agent.
- `--purpose` is optional context metadata; it does **not** change what gets captured.
- There is **no** `bsk replay` — to redo a flow, read the trace and reuse the existing `session` / `snapshot` / `@eN` / `click` / `fill` tools. Follow **Stop when the goal is met**.
- Do **not** record on banking/SSO/password-manager pages; passwords are redacted but traces may still contain sensitive text.
## Error handling
### Exit codes (`echo $?` after `bsk …`)
@@ -221,8 +294,9 @@ Always **`bsk session stop <id>`** in a `finally`-style path so the Agent Window
1. **No token theft** — do not `bsk evaluate` on sensitive sites to read `localStorage`, cookies, or auth headers for exfiltration.
2. **No long borrow** — do not leave a user's personal tab in the Agent Window across unrelated tasks.
3. **No skip stop** — always `bsk session stop <id>`; never assume idle timeout will clean up.
4. **No observe escalation before snapshot** — use `bsk snapshot` first; only use `bsk get-html` or `bsk screenshot` when the snapshot is insufficient. Element screenshots (`--ref @eN`) still require a fresh snapshot ref — never skip snapshot just to grab a visual.
5. **`evaluate` is powerful and risky** — use only when snapshot + click/fill/select cannot suffice; never on credential surfaces.
4. **No post-success control** — once the user’s goal (or last trace step) is met, do not keep operating the page; stop the session unless they asked to keep it open.
5. **No raw observe escalation before snapshot/observe** — use `bsk snapshot` first; use `bsk observe` when VOM semantics or conditional surfaces help. Only use `bsk get-html` or `bsk screenshot` when snapshot/observe is insufficient. Element screenshots (`--ref @eN`) still require a fresh snapshot/observe ref — never skip observation just to grab a visual.
6. **`evaluate` is powerful and risky** — use only when snapshot + click/fill/select cannot suffice; never on credential surfaces.
---
+62
View File
@@ -0,0 +1,62 @@
//! Shared browser-connect wait durations for status/doctor/browsers commands.
use std::time::Duration;
/// Default wait passed to daemon status/list RPCs when the registry is empty.
///
/// Deliberately shorter than the daemon's [`EXTENSION_CONNECT_WAIT`]
/// (used by `session.start`): `bsk status` / `bsk browsers` are quick
/// informational snapshots, so blocking them for a full keepalive-alarm
/// period would hurt more than it helps. The long wait is reserved for
/// the command path that actually needs the extension online.
///
/// [`EXTENSION_CONNECT_WAIT`]: crate::daemon::browsers::EXTENSION_CONNECT_WAIT
pub const DEFAULT_BROWSER_CONNECT_WAIT: Duration = Duration::from_secs(5);
/// Keep CLI-side waits aligned with the daemon's IPC clamp so a bad env value
/// cannot inflate read timeouts after the daemon has already returned.
const MAX_BROWSER_CONNECT_WAIT: Duration = Duration::from_secs(60);
/// Browser wait for `bsk status` / `bsk browsers`. Override in tests via
/// `BSK_BROWSER_WAIT_MS`.
pub fn browser_connect_wait() -> Duration {
wait_from_env("BSK_BROWSER_WAIT_MS").unwrap_or(DEFAULT_BROWSER_CONNECT_WAIT)
}
/// Browser wait for `bsk doctor`. Falls back to [`browser_connect_wait`] when
/// `BSK_DOCTOR_BROWSER_WAIT_MS` is unset so tests can shorten only doctor.
pub fn doctor_browser_connect_wait() -> Duration {
wait_from_env("BSK_DOCTOR_BROWSER_WAIT_MS").unwrap_or_else(browser_connect_wait)
}
/// Encodes a wait duration for protocol params. A zero duration preserves the
/// legacy immediate-snapshot semantics.
pub fn wait_for_browser_ms(wait: Duration) -> Option<u64> {
let wait = clamp_browser_wait(wait);
if wait.is_zero() {
None
} else {
Some(wait.as_millis() as u64)
}
}
/// IPC call budget: the command's normal timeout, or browser wait plus
/// scheduling slack when a wait is requested.
pub fn browser_query_ipc_timeout(wait: Duration, default_timeout: Duration) -> Duration {
let wait = clamp_browser_wait(wait);
if wait.is_zero() {
default_timeout
} else {
wait.saturating_add(Duration::from_secs(2))
}
}
fn wait_from_env(key: &str) -> Option<Duration> {
std::env::var(key)
.ok()
.and_then(|v| v.parse::<u64>().ok())
.map(|ms| clamp_browser_wait(Duration::from_millis(ms)))
}
fn clamp_browser_wait(wait: Duration) -> Duration {
wait.min(MAX_BROWSER_CONNECT_WAIT)
}
+16 -3
View File
@@ -5,9 +5,13 @@ use std::time::Duration;
use anyhow::Context;
use bsk_protocol::Method;
use bsk_protocol::system::BrowserListParams;
use bsk_protocol::system::BrowserStatusEntry;
use serde::Deserialize;
use crate::cli::browser_wait::{
browser_connect_wait, browser_query_ipc_timeout, wait_for_browser_ms,
};
use crate::cli::ensure_daemon::ensure_daemon;
use crate::cli::error::{CliError, Format};
@@ -22,7 +26,12 @@ pub fn dispatch(format: Format) -> Result<(), CliError> {
}
fn run_list(sock: PathBuf, format: Format) -> Result<(), CliError> {
let reply: ListReply = call(sock, Duration::from_secs(5))?;
let wait = browser_connect_wait();
let params = BrowserListParams {
wait_for_browser_ms: wait_for_browser_ms(wait),
};
let timeout = browser_query_ipc_timeout(wait, Duration::from_secs(5));
let reply: ListReply = call(sock, params, timeout)?;
match format {
Format::Json => {
println!(
@@ -92,7 +101,11 @@ fn run_list(sock: PathBuf, format: Format) -> Result<(), CliError> {
Ok(())
}
fn call(sock: PathBuf, timeout: Duration) -> Result<ListReply, CliError> {
fn call(
sock: PathBuf,
params: BrowserListParams,
timeout: Duration,
) -> Result<ListReply, CliError> {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
@@ -101,7 +114,7 @@ fn call(sock: PathBuf, timeout: Duration) -> Result<ListReply, CliError> {
rt.block_on(async move {
let mut client = crate::ipc_client::IpcClient::connect(sock).await?;
let outcome = client
.call::<(), ListReply>("browser-list-1", Method::BrowserList, None, timeout)
.call("browser-list-1", Method::BrowserList, Some(params), timeout)
.await?;
outcome.map_err(CliError::from_rpc)
})
+131
View File
@@ -0,0 +1,131 @@
//! `bsk console` — read buffered page console/log/exception messages.
use std::path::PathBuf;
use anyhow::Context;
use bsk_protocol::Method;
use bsk_protocol::tools::{ConsoleEntry, ConsoleParams, ConsoleResult};
use clap::Args;
use crate::cli::TOOL_IPC_TIMEOUT;
use crate::cli::ensure_daemon::ensure_daemon;
use crate::cli::error::{CliError, Format};
#[derive(Debug, Clone, Args)]
pub struct ConsoleArgs {
/// Session id (must be active).
#[arg(long)]
pub session: String,
/// Target tab. Defaults to the Agent Window's active tab.
#[arg(long = "tab-id")]
pub tab_id: Option<i64>,
/// Return entries with sequence greater than this cursor.
#[arg(long)]
pub since: Option<u64>,
/// Maximum number of entries to return. Defaults to 50; extension caps at 200.
#[arg(long, value_parser = clap::value_parser!(u32).range(1..))]
pub limit: Option<u32>,
/// Maximum characters per entry text. Defaults to 1000; extension caps at 4096.
#[arg(long = "max-text-chars", value_parser = clap::value_parser!(u32).range(1..))]
pub max_text_chars: Option<u32>,
/// Include structured stack frames in JSON/human output.
#[arg(long = "include-stack", default_value_t = false)]
pub include_stack: bool,
}
pub fn dispatch(args: ConsoleArgs, format: Format) -> Result<(), CliError> {
let info = ensure_daemon().context("ensure daemon is running")?;
run(info.sock_path, args, format)
}
fn run(sock: PathBuf, args: ConsoleArgs, format: Format) -> Result<(), CliError> {
let params = ConsoleParams {
session_id: args.session,
tab_id: args.tab_id,
since: args.since,
limit: args.limit,
max_text_chars: args.max_text_chars,
include_stack: args.include_stack.then_some(true),
};
let reply: ConsoleResult = call(sock, params)?;
render(&reply, format)
}
fn call(sock: PathBuf, params: ConsoleParams) -> Result<ConsoleResult, CliError> {
crate::cli::business_rpc::call::<ConsoleParams, ConsoleResult>(
sock,
"console",
Method::ToolConsole,
Some(params),
TOOL_IPC_TIMEOUT,
)
}
fn render(reply: &ConsoleResult, format: Format) -> Result<(), CliError> {
match format {
Format::Json => {
let json = serde_json::to_string_pretty(reply)
.map_err(|e| CliError::Local(anyhow::anyhow!(e)))?;
println!("{json}");
}
Format::Human => {
if reply.entries.is_empty() {
println!("(no console messages captured)");
} else {
for entry in &reply.entries {
println!("{}", render_entry(entry));
for frame in &entry.stack_trace {
let loc = render_location(frame.url.as_deref(), frame.line, frame.column);
let name = frame.function_name.as_deref().unwrap_or("<anonymous>");
println!(" at {name} {loc}");
}
}
}
if reply.truncated {
eprintln!(
"warning: console output truncated (next_since={}). Use --since / --limit / --max-text-chars to request a different slice.",
reply.next_since
);
}
}
}
Ok(())
}
fn render_entry(entry: &ConsoleEntry) -> String {
let loc = render_location(entry.url.as_deref(), entry.line, entry.column);
if loc.is_empty() {
format!(
"#{} {} {} {}",
entry.sequence,
entry.level,
entry.kind.as_str(),
entry.text
)
} else {
format!(
"#{} {} {} {} {}",
entry.sequence,
entry.level,
entry.kind.as_str(),
loc,
entry.text
)
}
}
fn render_location(url: Option<&str>, line: Option<i64>, column: Option<i64>) -> String {
let Some(url) = url else {
return String::new();
};
match (line, column) {
(Some(line), Some(column)) => format!("{url}:{line}:{column}"),
(Some(line), None) => format!("{url}:{line}"),
_ => url.to_string(),
}
}
+1 -1
View File
@@ -25,7 +25,7 @@ pub enum DaemonCmd {
Restart(StartArgs),
}
#[derive(Debug, Clone, clap::Args)]
#[derive(Debug, Clone, Default, clap::Args)]
pub struct StartArgs {
/// Override the WebSocket port (default 52800).
#[arg(long, value_name = "PORT")]
+56 -67
View File
@@ -1,23 +1,19 @@
//! `bsk doctor` — guided diagnostics + repair hints.
use std::time::{Duration, Instant};
use std::time::Duration;
use anyhow::Result;
use bsk_protocol::StatusResult;
use console::style;
use serde::Serialize;
use crate::cli::browser_wait::doctor_browser_connect_wait;
use crate::cli::ensure_daemon::ensure_daemon;
use crate::cli::status::{self, Output};
use crate::daemon::browsers::EXTENSION_CONNECT_WAIT;
use crate::daemon::info::{self, DaemonInfo};
use crate::daemon::paths;
use crate::daemon::state::PROTOCOL_VERSION;
/// How often `warm_up` polls `system.status` while waiting for the
/// browser extension to complete its WS handshake.
const BROWSER_CONNECT_POLL: Duration = Duration::from_millis(200);
/// Chrome Web Store listing for the browser-skill extension.
const EXTENSION_STORE_URL: &str =
"https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi";
@@ -99,8 +95,8 @@ impl CheckResult {
}
pub fn run(output: Output) -> Result<Vec<CheckResult>> {
warm_up(output);
let checks = collect_checks();
let state = resolve_daemon_state(output);
let checks = collect_checks(state);
match output {
Output::Human => render_human(&checks),
Output::Json => render_json(&checks)?,
@@ -108,74 +104,53 @@ pub fn run(output: Output) -> Result<Vec<CheckResult>> {
Ok(checks)
}
/// Ensure the daemon is up and give the browser extension time to connect.
///
/// All errors are silently swallowed — failures are reported by the
/// individual checks in `collect_checks()` with actionable hints.
///
/// Flow:
/// 1. Read the current daemon state. **Only** proceed for `Missing` and
/// `StaleDead`: broken states (`ReadError`, `IpcUnreachable`,
/// `PidMismatch`) are real diagnostic findings that `collect_checks()`
/// should surface — fixing them here would mask the real problem.
/// 2. Call `ensure_daemon()` to auto-start the daemon (spawns a detached
/// child and waits up to 3 s for it to write a valid `daemon.json`).
/// 3. Poll `system.status` for up to [`EXTENSION_CONNECT_WAIT`] so the MV3
/// service worker has time to wake up, discover the WS port, and finish
/// its handshake before the checks are evaluated.
/// The wait duration can be shortened for tests by setting the env var
/// `BSK_DOCTOR_BROWSER_WAIT_MS` to a millisecond count.
fn warm_up(output: Output) {
// Only spawn for states where the daemon is genuinely absent or dead.
match current_state() {
DaemonState::Missing | DaemonState::StaleDead(_) => {}
_ => return,
/// Whether the rendered doctor report contains an active failure.
/// `NotApplicable` remains informational and must not change the exit code.
pub fn has_failures(checks: &[CheckResult]) -> bool {
checks.iter().any(|check| check.status == CheckStatus::Fail)
}
/// Ensure the daemon is reachable and give the browser extension time to
/// connect before checks run. Returns a single [`DaemonState`] snapshot
/// for check evaluation.
fn resolve_daemon_state(output: Output) -> DaemonState {
let mut state = current_state(Duration::ZERO);
if matches!(state, DaemonState::Missing | DaemonState::StaleDead(_)) && ensure_daemon().is_err()
{
return current_state(Duration::ZERO);
}
let info = match ensure_daemon() {
Ok(info) => info,
Err(_) => return,
if matches!(state, DaemonState::Missing | DaemonState::StaleDead(_)) {
state = current_state(Duration::ZERO);
}
let wait = if needs_browser_wait(&state) {
doctor_browser_connect_wait()
} else {
Duration::ZERO
};
// Daemon was freshly spawned. Give the MV3 service worker time to
// wake up, discover the WS port, and complete its handshake.
if output == Output::Human {
if wait > Duration::ZERO && output == Output::Human {
eprintln!("waiting for browser extension to connect…");
}
let wait = browser_connect_wait();
let deadline = Instant::now() + wait;
loop {
// Check deadline before sleeping so a zero-duration wait exits
// immediately and we never start a query_sock call past the deadline.
if Instant::now() >= deadline {
break;
}
std::thread::sleep(BROWSER_CONNECT_POLL);
if Instant::now() >= deadline {
break;
}
if let Ok(s) = status::query_sock(info.sock_path.clone()) {
if !s.browsers.is_empty() {
break;
}
}
if wait.is_zero() {
state
} else {
current_state(wait)
}
}
/// The browser connect wait duration for `warm_up`. Defaults to
/// [`EXTENSION_CONNECT_WAIT`]; can be overridden in tests via the
/// `BSK_DOCTOR_BROWSER_WAIT_MS` environment variable.
fn browser_connect_wait() -> Duration {
std::env::var("BSK_DOCTOR_BROWSER_WAIT_MS")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.map(Duration::from_millis)
.unwrap_or(EXTENSION_CONNECT_WAIT)
fn needs_browser_wait(state: &DaemonState) -> bool {
match state {
DaemonState::Verified { status } => status.browsers.is_empty(),
_ => false,
}
}
/// What the disk + IPC says about a possibly-running daemon. Threaded
/// through every check so they share one IPC round-trip.
/// through every check so they share one snapshot.
enum DaemonState {
/// No `daemon.json` at all.
Missing,
@@ -208,8 +183,7 @@ impl DaemonState {
}
}
fn collect_checks() -> Vec<CheckResult> {
let state = current_state();
fn collect_checks(state: DaemonState) -> Vec<CheckResult> {
vec![
check_home_writable(),
check_skill_up_to_date(),
@@ -220,7 +194,7 @@ fn collect_checks() -> Vec<CheckResult> {
]
}
fn current_state() -> DaemonState {
fn current_state(browser_wait: Duration) -> DaemonState {
let info = match info::read() {
Ok(Some(info)) => info,
Ok(None) => return DaemonState::Missing,
@@ -229,7 +203,7 @@ fn current_state() -> DaemonState {
if !crate::daemon::lockfile::pid_alive(info.pid) {
return DaemonState::StaleDead(info);
}
match status::query_sock(info.sock_path.clone()) {
match status::query_sock_with_wait(info.sock_path.clone(), browser_wait) {
Ok(status) => {
if status.pid == info.pid {
DaemonState::Verified { status }
@@ -532,6 +506,21 @@ mod m2_tests {
);
}
#[test]
fn only_active_failures_make_doctor_unsuccessful() {
let healthy = vec![
CheckResult::ok("ok", "ready"),
CheckResult::na("optional", "not connected"),
];
assert!(!has_failures(&healthy));
let unhealthy = vec![
CheckResult::ok("ok", "ready"),
CheckResult::fail("broken", "not ready", "repair it"),
];
assert!(has_failures(&unhealthy));
}
#[test]
fn browsers_check_reports_na_when_no_browser_connected() {
// Review M2: 0 browsers means there is nothing to compare

Some files were not shown because too many files have changed in this diff Show More