mirror of
https://github.com/Tencent/BrowserSkill.git
synced 2026-09-10 13:01:23 +08:00
perf(vom): make active hover probing opt-in on observe
Every tool.observe moved the real cursor across the page to look for hover-only menus and tooltips. Measured against real Chrome, that cost a median of 2.5s and up to 4.7s per observation — 73% to 99% of total observe wall clock — and returned 3 hits across 12 pages. A positive control confirms the probes work; the yield on real pages simply does not justify putting them on the default path. observe now probes only when asked, via `probe_hover` (`--probe-hover`). Agents that know which element hides a menu should keep using `bsk hover <ref>`, which is both cheaper and precise. A held hover latch still suppresses probing, since probing would move the cursor off the element the caller is deliberately holding. Correctness fixes that apply whether or not probing is requested: - Hover probing ran between the DOM snapshot and the accessibility tree fetch, so a hover that opened a menu left one half of an observation describing the page before the change and the other half after it. Both chains now run after all capture is complete. - The overlay bypass toggled once per chain and swallowed restore failures, which could leave the agent overlay hidden for the rest of the tab's life. It is now reference counted across both chains, and a failed restore is reported instead of discarded. - Observations report whether they hovered the page and whether that revealed content the static tree lacks, so callers can judge how far the live page may have drifted from the returned snapshot. Probe budgets were checked at the top of the loop, so a candidate could start with 1ms remaining and still run a full settle window, overshooting by up to 634ms. Candidates are now admitted only if they fit. Raising MAX_HOVER_PROBE_MS to 2600 is part of that fix rather than a relaxation: 2000 was never the real ceiling, and enforcing it literally would have silently dropped a candidate the positive control depends on. Tooltip misses are cached too, with a shorter TTL than hits, so repeat observations of the same page no longer re-probe every unlabelled control from scratch.
This commit is contained in:
@@ -67,7 +67,7 @@ Write operations only affect tabs in the **Agent Window** (or tabs you **borrowe
|
||||
|
||||
```
|
||||
bsk navigate <url> --session <id>
|
||||
bsk observe --session <id> → primary semantic VOM view; reveals hover/focus surfaces
|
||||
bsk observe --session <id> → primary semantic VOM view
|
||||
bsk snapshot --session <id> → static aria tree fallback when VOM is insufficient
|
||||
bsk hover @e3 --session <id> → reveal hover-triggered menus before re-observing/clicking
|
||||
bsk click @e4 --session <id> → or bsk fill, bsk select, bsk press
|
||||
@@ -80,11 +80,13 @@ Prefer `@eN` refs from the latest snapshot over raw CSS selectors. Use `--ref` /
|
||||
|
||||
When VOM renders `[hover first: …]` on an element, the listed items are not currently clickable refs. Run `bsk hover <that-ref> --session <id>`, then immediately run `bsk snapshot` or `bsk observe` again and click the newly visible menu item ref. Do not click the trigger itself unless the user explicitly wants the trigger action.
|
||||
|
||||
`bsk observe` does not hover the page on its own. When a menu bar, toolbar, or icon row looks like it must hide content behind hover and you cannot tell which element to hover, add `--probe-hover` for one observation: it moves the real cursor over a bounded set of likely triggers to discover hover-only menus and tooltips. It costs a few seconds and touches the live page, so prefer `bsk hover <ref>` once you know the trigger.
|
||||
|
||||
## Observation priority
|
||||
|
||||
Start with `bsk observe` to understand page structure, text, controls, element refs, and conditional hover/focus surfaces. Use `bsk snapshot` only when you need the stricter static accessibility tree or VOM is insufficient. Only escalate to raw HTML or screenshots when the latest observation cannot answer the question:
|
||||
Start with `bsk observe` to understand page structure, text, controls, and element refs. Use `bsk snapshot` only when you need the stricter static accessibility tree or VOM is insufficient. Only escalate to raw HTML or screenshots when the latest observation cannot answer the question:
|
||||
|
||||
1. `bsk observe` — primary semantic VOM observation; may run bounded perception probes such as hover-surface discovery
|
||||
1. `bsk observe` — primary semantic VOM observation; add `--probe-hover` only when hover-only content is suspected
|
||||
2. `bsk snapshot` — strict static accessibility tree fallback
|
||||
3. `bsk get-html` — when hidden DOM, metadata, or markup details are required
|
||||
4. `bsk screenshot` — when visual layout, canvas/image content, or styling cannot be inferred from the observation. Use `--ref @eN` (from the latest snapshot/observe) to crop to one element; omit `--ref` for the full visible tab.
|
||||
@@ -172,7 +174,7 @@ bsk emulate --session <id> --off
|
||||
| Command | Summary |
|
||||
|---------|---------|
|
||||
| `bsk snapshot` | First-choice static page understanding: accessibility tree with `@eN` element refs |
|
||||
| `bsk observe` | Semantic VOM observation with bounded perception probes for conditional surfaces |
|
||||
| `bsk observe` | Semantic VOM observation; `--probe-hover` adds bounded active hover probing for conditional surfaces |
|
||||
| `bsk get-html` | Raw HTML dump after snapshot is insufficient (high token cost) |
|
||||
| `bsk screenshot` | PNG capture after snapshot is insufficient: full visible tab, or `--ref @eN` to crop to one element (`--out` path optional) |
|
||||
|
||||
@@ -319,7 +321,7 @@ Always **`bsk session stop <id>`** in a `finally`-style path so the Agent Window
|
||||
2. **No long borrow** — do not leave a user's personal tab in the Agent Window across unrelated tasks.
|
||||
3. **No skip stop** — always `bsk session stop <id>`; never assume idle timeout will clean up.
|
||||
4. **No post-success control** — once the user’s goal (or last trace step) is met, do not keep operating the page; stop the session unless they asked to keep it open.
|
||||
5. **No raw observe escalation before snapshot/observe** — use `bsk snapshot` first; use `bsk observe` when VOM semantics or conditional surfaces help. Only use `bsk get-html` or `bsk screenshot` when snapshot/observe is insufficient. Element screenshots (`--ref @eN`) still require a fresh snapshot/observe ref — never skip observation just to grab a visual.
|
||||
5. **No raw observe escalation before snapshot/observe** — use `bsk snapshot` first; use `bsk observe` when VOM semantics help. Only use `bsk get-html` or `bsk screenshot` when snapshot/observe is insufficient. Element screenshots (`--ref @eN`) still require a fresh snapshot/observe ref — never skip observation just to grab a visual.
|
||||
6. **`evaluate` is powerful and risky** — use only when snapshot + click/fill/select cannot suffice; never on credential surfaces.
|
||||
|
||||
---
|
||||
|
||||
@@ -33,6 +33,12 @@ pub struct ObserveArgs {
|
||||
/// Include conditional surface probe diagnostics in JSON output.
|
||||
#[arg(long = "debug-surfaces")]
|
||||
pub debug_surfaces: bool,
|
||||
|
||||
/// Actively hover page controls to reveal hover-only menus and tooltips.
|
||||
/// Costs seconds of wall clock and touches the live page, so it is off
|
||||
/// unless a static observation looks like it is missing hover content.
|
||||
#[arg(long = "probe-hover")]
|
||||
pub probe_hover: bool,
|
||||
}
|
||||
|
||||
pub fn dispatch(args: ObserveArgs, format: Format) -> Result<(), CliError> {
|
||||
@@ -47,6 +53,7 @@ fn run(sock: PathBuf, args: ObserveArgs, format: Format) -> Result<(), CliError>
|
||||
max_depth: args.max_depth,
|
||||
max_tokens: args.max_tokens,
|
||||
debug_surfaces: args.debug_surfaces,
|
||||
probe_hover: args.probe_hover,
|
||||
};
|
||||
let reply: ObserveResult = call(sock, params)?;
|
||||
match format {
|
||||
|
||||
@@ -478,6 +478,7 @@ async fn observe_returns_semantic_text_and_ref_count() {
|
||||
tab_id: 13,
|
||||
truncated: false,
|
||||
dialogs: vec![],
|
||||
hover_probe: None,
|
||||
debug: None,
|
||||
})
|
||||
.unwrap(),
|
||||
@@ -494,6 +495,7 @@ async fn observe_returns_semantic_text_and_ref_count() {
|
||||
max_depth: None,
|
||||
max_tokens: None,
|
||||
debug_surfaces: false,
|
||||
probe_hover: false,
|
||||
},
|
||||
)
|
||||
.await
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "ObserveParams",
|
||||
"description": "Parameters for `tool.observe` — produce a semantic VOM observation for one tab plus a fresh `@e<N>` ref-store on the extension side.\n\nUnlike `tool.snapshot`, this path may run bounded perception probes such as hover-surface discovery. It must not submit input, click, navigate, or otherwise commit page state.",
|
||||
"description": "Parameters for `tool.observe` — produce a semantic VOM observation for one tab plus a fresh `@e<N>` ref-store on the extension side.\n\nUnlike `tool.snapshot`, this path may run bounded perception probes such as hover-surface discovery, but only when `probe_hover` asks for them. It must not submit input, click, navigate, or otherwise commit page state.",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"session_id"
|
||||
@@ -30,6 +30,11 @@
|
||||
"format": "uint32",
|
||||
"minimum": 0.0
|
||||
},
|
||||
"probe_hover": {
|
||||
"description": "Opt in to active hover probing: the extension moves the real cursor over a bounded set of controls to discover hover-only menus and tooltips.\n\nOff by default. Probing costs seconds of wall clock, mutates the live page, and pays off only on pages whose content is genuinely hidden behind hover. Request it when a static observation looks like it is missing hover-revealed structure.",
|
||||
"default": false,
|
||||
"type": "boolean"
|
||||
},
|
||||
"session_id": {
|
||||
"type": "string"
|
||||
},
|
||||
|
||||
@@ -24,6 +24,17 @@
|
||||
"$ref": "#/definitions/JavaScriptDialogInfo"
|
||||
}
|
||||
},
|
||||
"hover_probe": {
|
||||
"description": "Present only when `probe_hover` was requested and probing actually ran.",
|
||||
"anyOf": [
|
||||
{
|
||||
"$ref": "#/definitions/HoverProbeReport"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
]
|
||||
},
|
||||
"ref_count": {
|
||||
"description": "Number of `@e<N>` refs registered for this session by this observation. Equivalent to the size of the new ref-store map.",
|
||||
"type": "integer",
|
||||
@@ -46,6 +57,24 @@
|
||||
}
|
||||
},
|
||||
"definitions": {
|
||||
"HoverProbeReport": {
|
||||
"description": "Reports what active hover probing did during an observation, so the agent can judge how far the live page may have drifted from the returned text.",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"performed",
|
||||
"revealed_content"
|
||||
],
|
||||
"properties": {
|
||||
"performed": {
|
||||
"description": "The real cursor was moved over page content during this observation.",
|
||||
"type": "boolean"
|
||||
},
|
||||
"revealed_content": {
|
||||
"description": "Hovering surfaced content the static tree does not contain, so the page reacts to the cursor and may not have fully reverted.",
|
||||
"type": "boolean"
|
||||
}
|
||||
}
|
||||
},
|
||||
"JavaScriptDialogHandledAction": {
|
||||
"description": "How the extension resolved the dialog so CDP could continue.",
|
||||
"type": "string",
|
||||
|
||||
@@ -58,8 +58,9 @@ pub struct SnapshotResult {
|
||||
/// for one tab plus a fresh `@e<N>` ref-store on the extension side.
|
||||
///
|
||||
/// Unlike `tool.snapshot`, this path may run bounded perception probes
|
||||
/// such as hover-surface discovery. It must not submit input, click,
|
||||
/// navigate, or otherwise commit page state.
|
||||
/// such as hover-surface discovery, but only when `probe_hover` asks for
|
||||
/// them. It must not submit input, click, navigate, or otherwise commit
|
||||
/// page state.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
|
||||
pub struct ObserveParams {
|
||||
pub session_id: String,
|
||||
@@ -77,6 +78,16 @@ pub struct ObserveParams {
|
||||
/// Include implementation diagnostics for conditional surface probes.
|
||||
#[serde(default)]
|
||||
pub debug_surfaces: bool,
|
||||
/// Opt in to active hover probing: the extension moves the real cursor
|
||||
/// over a bounded set of controls to discover hover-only menus and
|
||||
/// tooltips.
|
||||
///
|
||||
/// Off by default. Probing costs seconds of wall clock, mutates the live
|
||||
/// page, and pays off only on pages whose content is genuinely hidden
|
||||
/// behind hover. Request it when a static observation looks like it is
|
||||
/// missing hover-revealed structure.
|
||||
#[serde(default)]
|
||||
pub probe_hover: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
|
||||
@@ -102,6 +113,17 @@ pub struct ObserveDebug {
|
||||
pub surface_probes: Vec<SurfaceProbeDebug>,
|
||||
}
|
||||
|
||||
/// Reports what active hover probing did during an observation, so the agent
|
||||
/// can judge how far the live page may have drifted from the returned text.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
|
||||
pub struct HoverProbeReport {
|
||||
/// The real cursor was moved over page content during this observation.
|
||||
pub performed: bool,
|
||||
/// Hovering surfaced content the static tree does not contain, so the page
|
||||
/// reacts to the cursor and may not have fully reverted.
|
||||
pub revealed_content: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
|
||||
pub struct ObserveResult {
|
||||
/// Semantic VOM observation text. Refs are rendered as `@e<N>` so
|
||||
@@ -118,6 +140,9 @@ pub struct ObserveResult {
|
||||
pub truncated: bool,
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub dialogs: Vec<JavaScriptDialogInfo>,
|
||||
/// Present only when `probe_hover` was requested and probing actually ran.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub hover_probe: Option<HoverProbeReport>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub debug: Option<ObserveDebug>,
|
||||
}
|
||||
@@ -308,6 +333,7 @@ mod tests {
|
||||
tab_id: 42,
|
||||
truncated: false,
|
||||
dialogs: Vec::new(),
|
||||
hover_probe: None,
|
||||
debug: None,
|
||||
};
|
||||
let v = serde_json::to_value(&r).unwrap();
|
||||
@@ -315,4 +341,30 @@ mod tests {
|
||||
let round: ObserveResult = serde_json::from_value(v).unwrap();
|
||||
assert_eq!(round, r);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn observe_params_default_to_no_hover_probing() {
|
||||
let params: ObserveParams =
|
||||
serde_json::from_value(serde_json::json!({ "session_id": "s1" })).unwrap();
|
||||
assert!(!params.probe_hover);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn observe_result_round_trips_with_hover_probe_report() {
|
||||
let r = ObserveResult {
|
||||
text: "@vom 1\n".into(),
|
||||
ref_count: 0,
|
||||
tab_id: 42,
|
||||
truncated: false,
|
||||
dialogs: Vec::new(),
|
||||
hover_probe: Some(HoverProbeReport {
|
||||
performed: true,
|
||||
revealed_content: true,
|
||||
}),
|
||||
debug: None,
|
||||
};
|
||||
let v = serde_json::to_value(&r).unwrap();
|
||||
let round: ObserveResult = serde_json::from_value(v).unwrap();
|
||||
assert_eq!(round, r);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user