From fd0e9de61bddee5696dd0c1b076214e39fe06dda Mon Sep 17 00:00:00 2001 From: drakezhang Date: Thu, 10 Sep 2026 10:15:57 +0800 Subject: [PATCH] docs: document scroll-to primitive contract --- CHANGELOG.md | 7 ++ README.md | 3 + README.zh-CN.md | 2 + crates/bsk-cli/skill/SKILL.md | 4 +- docs/architecture.md | 5 + docs/scroll-to.md | 131 +++++++++++++++++++++ packages/dsh-plugin-browserskill/README.md | 3 + skill/SKILL.md | 4 +- 8 files changed, 155 insertions(+), 4 deletions(-) create mode 100644 docs/scroll-to.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 4669d4f..b0900e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/). Starting from 0.2.0, CLI / Extension / DSH Plugin share the same version number. +## [Unreleased] + +### Added + +- [Scroll-to element primitive](docs/scroll-to.md) across CLI, Extension and DSH Plugin, + with ancestor-clipped visible bounds, iframe support and cooperative cancellation + ## [0.2.1] - 2026-09-09 ### Changed diff --git a/README.md b/README.md index 5cf4e81..919a220 100644 --- a/README.md +++ b/README.md @@ -207,6 +207,9 @@ through the [plugin](#deepseek-harness-plugin): the agent calls injected ## For Developers +The [scroll-to primitive reference](docs/scroll-to.md) covers its CLI, protocol +and plugin entry points, visible bounds and interruption behavior. + The repository is a Cargo + pnpm workspace: - `crates/bsk-cli` — `bsk` CLI and local daemon diff --git a/README.zh-CN.md b/README.zh-CN.md index b0c8582..2d2bd3c 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -177,6 +177,8 @@ Agent 不直接与浏览器通信。它通过 `bsk` CLI 下发浏览器任务； ## 面向开发者 +[scroll-to 原语说明](docs/scroll-to.md)介绍 CLI、协议和插件入口，以及可见区域、错误和中断语义。 + 本仓库是 Cargo + pnpm workspace： - `crates/bsk-cli` — `bsk` CLI 与本地 daemon diff --git a/crates/bsk-cli/skill/SKILL.md b/crates/bsk-cli/skill/SKILL.md index c48b13e..765b91c 100644 --- a/crates/bsk-cli/skill/SKILL.md +++ b/crates/bsk-cli/skill/SKILL.md @@ -55,7 +55,7 @@ Use this default loop: ```text bsk navigate --session bsk observe --session -bsk click|hover|focus|blur|fill|select|press ... --session +bsk click|hover|scroll-to|focus|blur|fill|select|press ... --session bsk observe --session # after navigation or a meaningful DOM change ``` @@ -63,7 +63,7 @@ bsk observe --session # after navigation or a meaningful DOM ch Use a fresh element ref for iframe/shadow-root targets; CSS selectors search the main document. The result is the visible border-box portion's bounds in top-level viewport CSS pixels after ancestor clipping. Partial visibility is enough; hidden or fully clipped targets fail with -`element_not_visible`. The result does not guarantee that other elements do not cover the target. +`permission_denied` and `data.reason=element_not_visible`. This does not test occlusion by other elements. For a specific tab or deadline: `bsk scroll-to @e3 --session --tab-id 42 --timeout 5s`. `bsk focus ` explicitly focuses a target; `bsk blur ` removes focus and reports whether diff --git a/docs/architecture.md b/docs/architecture.md index e55f592..d7887db 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -84,6 +84,11 @@ Shared Rust types + JSON Schema generation. TypeScript mirrors frame shapes in 4. Extension dispatcher validates sandbox rules, invokes CDP via `BrowserDriver`. 5. Response travels CLI ← daemon ← extension; CLI prints result and exits. +The [scroll-to primitive reference](scroll-to.md) documents `tool.scroll_to`, +including its CLI/plugin mappings, visible-bounds contract and cancellation +behavior. It follows the same routing path and is classified as a browser +mutation for session queueing and user-interruption gating. + ## Session and sandbox model - **Session** = opaque ID (4 lowercase letters in v0.1) + dedicated **Agent Window** diff --git a/docs/scroll-to.md b/docs/scroll-to.md new file mode 100644 index 0000000..f1b3838 --- /dev/null +++ b/docs/scroll-to.md @@ -0,0 +1,131 @@ +# Scroll-to primitive + +`scroll-to` brings an existing element and its containing frames into view, then +returns the visible portion's bounds. Use it to reveal a target before inspecting +the page or taking a screenshot. It accepts an element target rather than a pixel +distance or scroll direction. + +| Entry point | Name | +| --- | --- | +| CLI | `bsk scroll-to` | +| Wire protocol | `tool.scroll_to` | +| DeepSeek Harness | `browser_interact` with `action: "scroll-to"` | + +## CLI + +Use a session you started and a fresh ref from an observation. The ids below are +examples; substitute the session, tab and ref returned by your own calls. + +```sh +bsk scroll-to @e3 --session abcd --tab-id 42 --timeout 5s --json +bsk scroll-to --selector '#details' --session abcd +``` + +Supply exactly one target: a positional ref or selector, `--ref`, or `--selector`. +Refs accept `@e3` and `e3`. CSS selectors search only the main document; use refs +for elements in iframes (including out-of-process iframes) or shadow roots. +Refs belong to one session and tab and may become stale after page changes. + +`--session` is required. `--tab-id` defaults to the Agent Window's active tab. +`--timeout` defaults to `30s` and must be positive; durations such as `5000ms` and +`5s` are accepted. The command operates on Agent Window tabs, including user tabs +explicitly borrowed into that window. + +Example JSON output: + +```json +{ + "tab_id": 42, + "used_ref": "e3", + "x": 10, + "y": 20, + "width": 100, + "height": 80 +} +``` + +`used_ref` or `used_selector` identifies the resolved target. The optional +`dialogs` array reports JavaScript dialogs handled during the call. + +## Result contract + +`x` and `y` are the upper-left corner, and `width` and `height` are the size of +the visible border-box portion's bounding rectangle. All four values use +**top-level viewport CSS pixels**, after scrolling and clipping by ancestor +containers and frame viewports. They are not document or screenshot pixel +coordinates. + +Partial visibility is sufficient: a 400px-high element clipped by an 80px-high +scroll container can succeed with `height: 80`. Hidden or fully clipped targets +fail. The rectangle is not an occlusion or hit test: another element can cover +the target, and the rectangle's center is not a guaranteed click point. Layout +can change after the result; use fresh refs for subsequent interactions. + +## Protocol and plugin + +The CLI sends a normal request through the daemon to the extension: + +```json +{ + "id": "scroll-1", + "method": "tool.scroll_to", + "params": { + "session_id": "abcd", + "ref": "@e3", + "tab_id": 42, + "timeout_ms": 5000 + } +} +``` + +Protocol callers must supply exactly one nonempty `ref` or `selector`. +`tab_id` is optional; `timeout_ms` is a positive integer and defaults to 30000. +These target constraints are enforced by the handler. See the generated +[parameter schema](../crates/bsk-protocol/schema/tool_scroll_to_params.json) and +[result schema](../crates/bsk-protocol/schema/tool_scroll_to_result.json) for field +types. The wire response wraps the result above in `{ "id": "scroll-1", "result": ... }`. + +The plugin exposes the same action with its usual camelCase parameters: + +```json +{ + "action": "scroll-to", + "session": "abcd", + "target": "@e3", + "tabId": 42, + "timeoutMs": 5000 +} +``` + +Call this through `browser_interact` using a plugin-owned session. Its result +contains `session`, `tabId`, `x`, `y`, `width` and `height` with the same bounds +semantics. Omitted `session` uses the plugin's current session. + +## Errors and interruption + +The wire response uses the normal `error.code`, `error.message` and optional +`error.data.reason` fields. Common cases are: + +| Code | `data.reason` | Meaning | +| --- | --- | --- | +| `invalid_params` | — | Missing/conflicting target or invalid timeout/tab id | +| `not_found` | `ref_not_found` | Ref is unknown, expired or belongs to another tab | +| `not_found` | `selector_not_found` | Main-document selector matched no element | +| `permission_denied` | `agent_window_scope` | Target tab has not been borrowed into the Agent Window | +| `permission_denied` | `element_not_visible` | No visible area remains after scrolling | +| `cancelled` | — | The call was cancelled | +| `timeout` | — | The action deadline expired | +| `cdp_failed` | varies | Browser command, frame geometry or visibility measurement failed | + +The protocol classifies scroll-to as a browser mutation, so it uses the existing +per-session queue and user-interruption gate. The extension releases any retained +hover state before dispatch. Cancellation and deadline checks surround each CDP +step, including frame lookup, scrolling and measurement. Once detected, they +prevent further scrolling and a successful response; remote-object cleanup still +runs. Already performed scrolling is not rolled back, so inspect the current +page before retrying an interrupted call. + +The implementation is scoped to scroll-to: it reuses frame scrolling and +coordinate projection, with a separate visibility measurement that accounts for +ancestor clipping and hidden frame owners. Shared click, hover and screenshot +geometry is unchanged. diff --git a/packages/dsh-plugin-browserskill/README.md b/packages/dsh-plugin-browserskill/README.md index 42e141b..b2d13bf 100644 --- a/packages/dsh-plugin-browserskill/README.md +++ b/packages/dsh-plugin-browserskill/README.md @@ -58,6 +58,9 @@ the `bsk` CLI and browser extension separately when a release requires it. | `browser_tabs` | `list`, `create`, `select`, `close`, `borrow`, `return` | Manage Agent Window tabs and temporarily borrow user tabs. | | `browser_assist` | `resize`, `emulate`, `request-help` | Resize or emulate the browser and pause for human-only steps. | +For `browser_interact` with `action: "scroll-to"`, see the +[scroll-to reference](../../docs/scroll-to.md) for parameters, visible bounds and errors. + Arbitrary page-script evaluation and interaction recording are not supported. ## Multi-session model diff --git a/skill/SKILL.md b/skill/SKILL.md index c48b13e..765b91c 100644 --- a/skill/SKILL.md +++ b/skill/SKILL.md @@ -55,7 +55,7 @@ Use this default loop: ```text bsk navigate --session bsk observe --session -bsk click|hover|focus|blur|fill|select|press ... --session +bsk click|hover|scroll-to|focus|blur|fill|select|press ... --session bsk observe --session # after navigation or a meaningful DOM change ``` @@ -63,7 +63,7 @@ bsk observe --session # after navigation or a meaningful DOM ch Use a fresh element ref for iframe/shadow-root targets; CSS selectors search the main document. The result is the visible border-box portion's bounds in top-level viewport CSS pixels after ancestor clipping. Partial visibility is enough; hidden or fully clipped targets fail with -`element_not_visible`. The result does not guarantee that other elements do not cover the target. +`permission_denied` and `data.reason=element_not_visible`. This does not test occlusion by other elements. For a specific tab or deadline: `bsk scroll-to @e3 --session --tab-id 42 --timeout 5s`. `bsk focus ` explicitly focuses a target; `bsk blur ` removes focus and reports whether