Compare commits

..
Author SHA1 Message Date
BB-fat 9504072914 chore(release): extension 0.1.6 (#74) 2026-08-09 00:34:47 +08:00
BB-fat b4450fbb8c chore(release): bsk 0.1.10 (#73) 2026-08-08 23:22:03 +08:00
BB-fat 1268761611 fix(screenshot): hide overlays during capture so shots only contain page content (#64) 2026-08-08 22:48:50 +08:00
Zhang GH 37d3e2b22f Merge pull request #72 from Tencent/issue-69-cws-publishing
CI: add automate Chrome Web Store publishing with service account
2026-08-07 19:24:30 +08:00
drakezhang ce8bc6c870 add automate Chrome Web Store publishing with service account 2026-08-07 14:30:59 +08:00
Zhang GH 93df62a356 Merge pull request #42 from shnpd/feat/popup-function-icon
feat(extension): use function launcher icon
2026-08-06 16:47:52 +08:00
Zhang GH 20002e1ff2 Merge pull request #63 from Tencent/feat/record-overlay-timer
feat(extension): compact recording overlay with live timer
2026-08-06 16:47:11 +08:00
“haonan” 674c53339f fix(extension): keep record overlay timer across navigations 2026-08-06 16:28:21 +08:00
BB-fat d0bee1a09b feat(install-skill): add Kimi Code harness support (#65)
* feat(install-skill): add Kimi Code harness support

Add kimi-code to the install-skill harness list: installs the bundled SKILL.md into ~/.kimi/skills (Kimi Code CLI's brand-group user skills directory, highest priority per https://github.com/MoonshotAI/kimi-cli/blob/main/docs/en/customization/skills.md). Detection signals: ~/.kimi directory and kimi/kimi-cli on PATH. Aliases: kimi, kimi-code, kimi-cli, kimicode, kimi_cli.

* fix(install-skill): point kimi-code harness at ~/.kimi-code and honor KIMI_CODE_HOME

Kimi Code CLI's canonical repo is MoonshotAI/kimi-code: user-level skills live in $KIMI_CODE_HOME/skills (default ~/.kimi-code/skills), not kimi-cli's ~/.kimi/skills. Detection checks the kimi-code home dir (env-aware label) and the kimi binary on PATH. Serialize env-mutating kimi tests with a shared mutex to avoid a parallel-test race on KIMI_CODE_HOME.
2026-08-06 13:25:18 +08:00
“haonan” 66684c4ce3 feat(extension): compact recording overlay with live timer
Show the recording message for 10s, then switch to a fixed-width MM:SS
timer with the icon-only finish button.
2026-08-05 20:26:43 +08:00
“haonan”andCursor aab3b7a8e5 feat(extension): use function launcher icon
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-23 15:44:11 +08:00
63 changed files with 1472 additions and 4982 deletions
+199
View File
@@ -126,3 +126,202 @@ jobs:
make_latest: false
generate_release_notes: true
files: dist/*.zip
publish-cws:
needs: [resolve, build-extension]
runs-on: ubuntu-latest
environment: chrome-web-store
permissions:
contents: read
steps:
- name: Download build artifact
uses: actions/download-artifact@v8
with:
name: extension-zip
path: dist
- name: Validate Chrome Web Store configuration
shell: bash
env:
PUBLISHER_ID: ${{ vars.CHROME_WEBSTORE_PUBLISHER_ID }}
EXTENSION_ID: ${{ vars.CHROME_WEBSTORE_EXTENSION_ID }}
run: |
set -euo pipefail
if [ -z "$PUBLISHER_ID" ]; then
echo "::error::Repository variable CHROME_WEBSTORE_PUBLISHER_ID is not configured"
exit 1
fi
if [ -z "$EXTENSION_ID" ]; then
echo "::error::Repository variable CHROME_WEBSTORE_EXTENSION_ID is not configured"
exit 1
fi
ARCHIVE="$(find dist -maxdepth 1 -name '*.zip' -type f -print -quit)"
if [ -z "$ARCHIVE" ]; then
echo "::error::No extension zip found in the downloaded artifact"
exit 1
fi
echo "ARCHIVE=$ARCHIVE" >> "$GITHUB_ENV"
- name: Authenticate with service account
id: cws-auth
env:
SERVICE_ACCOUNT_JSON: ${{ secrets.CHROME_WEBSTORE_SERVICE_ACCOUNT_JSON }}
shell: bash
run: |
set -euo pipefail
node --input-type=module <<'NODE'
import { sign } from "node:crypto";
import { appendFileSync } from "node:fs";
const rawCredentials = process.env.SERVICE_ACCOUNT_JSON;
if (!rawCredentials) {
throw new Error(
"Repository secret CHROME_WEBSTORE_SERVICE_ACCOUNT_JSON is not configured",
);
}
let credentials;
try {
credentials = JSON.parse(rawCredentials);
} catch {
throw new Error(
"Repository secret CHROME_WEBSTORE_SERVICE_ACCOUNT_JSON is not valid JSON",
);
}
if (!credentials.client_email || !credentials.private_key) {
throw new Error(
"Service account JSON must contain client_email and private_key",
);
}
const encode = (value) =>
Buffer.from(JSON.stringify(value)).toString("base64url");
const issuedAt = Math.floor(Date.now() / 1000);
const header = {
alg: "RS256",
typ: "JWT",
kid: credentials.private_key_id,
};
const claim = {
iss: credentials.client_email,
scope: "https://www.googleapis.com/auth/chromewebstore",
aud: "https://oauth2.googleapis.com/token",
iat: issuedAt,
exp: issuedAt + 3600,
};
const unsignedJwt = `${encode(header)}.${encode(claim)}`;
const signature = sign(
"RSA-SHA256",
Buffer.from(unsignedJwt),
credentials.private_key,
).toString("base64url");
const assertion = `${unsignedJwt}.${signature}`;
const response = await fetch("https://oauth2.googleapis.com/token", {
method: "POST",
headers: { "Content-Type": "application/x-www-form-urlencoded" },
body: new URLSearchParams({
grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer",
assertion,
}),
});
const responseText = await response.text();
if (!response.ok) {
throw new Error(
`Service account token exchange failed (${response.status}): ${responseText}`,
);
}
const tokenResponse = JSON.parse(responseText);
if (!tokenResponse.access_token) {
throw new Error("Service account token response has no access_token");
}
console.log(`::add-mask::${tokenResponse.access_token}`);
appendFileSync(
process.env.GITHUB_OUTPUT,
`access_token=${tokenResponse.access_token}\n`,
);
NODE
- name: Upload and publish Chrome Web Store extension
shell: bash
env:
ACCESS_TOKEN: ${{ steps.cws-auth.outputs.access_token }}
PUBLISHER_ID: ${{ vars.CHROME_WEBSTORE_PUBLISHER_ID }}
EXTENSION_ID: ${{ vars.CHROME_WEBSTORE_EXTENSION_ID }}
run: |
set -euo pipefail
ITEM_NAME="publishers/${PUBLISHER_ID}/items/${EXTENSION_ID}"
API_ROOT="https://chromewebstore.googleapis.com"
UPLOAD_RESPONSE="$RUNNER_TEMP/cws-upload-response.json"
STATUS_RESPONSE="$RUNNER_TEMP/cws-status-response.json"
PUBLISH_RESPONSE="$RUNNER_TEMP/cws-publish-response.json"
HTTP_STATUS="$(curl --silent --show-error \
--output "$UPLOAD_RESPONSE" \
--write-out '%{http_code}' \
--request POST \
--header "Authorization: Bearer $ACCESS_TOKEN" \
--header "Content-Type: application/zip" \
--data-binary "@$ARCHIVE" \
"$API_ROOT/upload/v2/$ITEM_NAME:upload")"
if [ "$HTTP_STATUS" -lt 200 ] || [ "$HTTP_STATUS" -ge 300 ]; then
echo "::error::Chrome Web Store upload failed with HTTP $HTTP_STATUS"
jq . "$UPLOAD_RESPONSE" || cat "$UPLOAD_RESPONSE"
exit 1
fi
UPLOAD_STATE="$(jq -r '.uploadState // empty' "$UPLOAD_RESPONSE")"
echo "Chrome Web Store upload state: $UPLOAD_STATE"
if [ "$UPLOAD_STATE" = "IN_PROGRESS" ] || [ "$UPLOAD_STATE" = "UPLOAD_IN_PROGRESS" ]; then
for attempt in $(seq 1 30); do
sleep 5
HTTP_STATUS="$(curl --silent --show-error \
--output "$STATUS_RESPONSE" \
--write-out '%{http_code}' \
--header "Authorization: Bearer $ACCESS_TOKEN" \
"$API_ROOT/v2/$ITEM_NAME:fetchStatus")"
if [ "$HTTP_STATUS" -lt 200 ] || [ "$HTTP_STATUS" -ge 300 ]; then
echo "::error::Chrome Web Store status check failed with HTTP $HTTP_STATUS"
jq . "$STATUS_RESPONSE" || cat "$STATUS_RESPONSE"
exit 1
fi
UPLOAD_STATE="$(jq -r '.lastAsyncUploadState // empty' "$STATUS_RESPONSE")"
echo "Upload status check $attempt/30: $UPLOAD_STATE"
if [ "$UPLOAD_STATE" = "SUCCEEDED" ]; then
break
fi
if [ "$UPLOAD_STATE" != "IN_PROGRESS" ] && [ "$UPLOAD_STATE" != "UPLOAD_IN_PROGRESS" ]; then
echo "::error::Chrome Web Store upload did not succeed: $UPLOAD_STATE"
jq . "$STATUS_RESPONSE"
exit 1
fi
done
fi
if [ "$UPLOAD_STATE" != "SUCCEEDED" ]; then
echo "::error::Chrome Web Store upload did not complete: $UPLOAD_STATE"
jq . "$UPLOAD_RESPONSE"
exit 1
fi
HTTP_STATUS="$(curl --silent --show-error \
--output "$PUBLISH_RESPONSE" \
--write-out '%{http_code}' \
--request POST \
--header "Authorization: Bearer $ACCESS_TOKEN" \
--header "Content-Type: application/json" \
--data '{"publishType":"DEFAULT_PUBLISH"}' \
"$API_ROOT/v2/$ITEM_NAME:publish")"
if [ "$HTTP_STATUS" -lt 200 ] || [ "$HTTP_STATUS" -ge 300 ]; then
echo "::error::Chrome Web Store publish failed with HTTP $HTTP_STATUS"
jq . "$PUBLISH_RESPONSE" || cat "$PUBLISH_RESPONSE"
exit 1
fi
echo "Chrome Web Store submission created successfully:"
jq '{itemId, state, warningInfo}' "$PUBLISH_RESPONSE"
Generated
+2 -2
View File
@@ -134,7 +134,7 @@ dependencies = [
[[package]]
name = "bsk"
version = "0.1.9"
version = "0.1.10"
dependencies = [
"anyhow",
"bsk-protocol",
@@ -168,7 +168,7 @@ dependencies = [
[[package]]
name = "bsk-protocol"
version = "0.1.9"
version = "0.1.10"
dependencies = [
"schemars",
"semver",
+1 -1
View File
@@ -7,7 +7,7 @@ members = [
default-members = ["crates/bsk-cli"]
[workspace.package]
version = "0.1.9"
version = "0.1.10"
edition = "2024"
rust-version = "1.85"
license = "MIT"
+4
View File
@@ -0,0 +1,4 @@
<svg width="24" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M6.30589 17.9712C6.21652 18.4603 6.63298 18.8722 7.12749 18.7838C7.88158 18.6491 8.86635 18.4075 9.25454 18.0236C9.49544 17.7853 9.61164 17.4432 9.57757 17.0726C9.5435 16.702 9.36195 16.3331 9.07287 16.0472C8.78378 15.7613 8.41084 15.5818 8.03608 15.5481C7.66133 15.5144 7.31545 15.6293 7.07455 15.8676C6.6865 16.2513 6.4422 17.2253 6.30589 17.9712Z" fill="#E94500"/>
<path d="M7.77425 8.64281C10.7374 4.12636 15.4876 3.7066 17.9364 4.13499C18.3335 4.20461 18.6474 4.48261 18.7821 4.84105C20.6715 9.87191 18.4167 13.7056 16.5008 15.5334C16.1262 15.8909 15.9395 16.3745 16.0282 16.8215L16.12 17.2834C16.24 17.887 15.871 18.4819 15.2723 18.6496L12.7811 19.3469C12.1057 19.536 11.4124 19.1112 11.285 18.4289L11.036 17.0959C10.9844 16.82 10.8353 16.5712 10.6151 16.3948L8.12777 14.4035C7.90665 14.2266 7.62818 14.1345 7.34359 14.1428L5.72835 14.1897C4.9877 14.2112 4.40832 13.5657 4.51644 12.8401L4.89632 10.2912C4.98194 9.71734 5.47627 9.2895 6.06234 9.28148L6.69124 9.27171C7.11716 9.26594 7.52243 9.02649 7.77425 8.64281ZM14.4959 7.91429C13.8284 7.91436 13.2869 8.45568 13.2869 9.12327C13.2871 9.79078 13.8284 10.3322 14.4959 10.3323C15.1635 10.3323 15.7048 9.79082 15.7049 9.12327C15.7049 8.45564 15.1636 7.91429 14.4959 7.91429Z" fill="#E94500"/>
</svg>

After

Width:  |  Height:  |  Size: 1.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.9 KiB

+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@browser-skill/extension",
"version": "0.1.5",
"version": "0.1.6",
"private": true,
"type": "module",
"scripts": {
+216 -43
View File
@@ -1,9 +1,21 @@
import { useTranslation } from "@browser-skill/i18n/react";
import { RiStopCircleLine } from "@remixicon/react";
import { useEffect, useState } from "react";
import {
type PointerEvent as ReactPointerEvent,
useCallback,
useEffect,
useRef,
useState,
} from "react";
import recordIconUrl from "../../assets/record-icon.png";
export interface RecordRequestData {
id: string;
/**
* Epoch ms when the recording began. Supplied by the background so the timer
* keeps counting the whole session across navigations; falls back to mount
* time when the background could not report it.
*/
startedAtMs?: number;
onFinish: () => void;
}
@@ -11,9 +23,51 @@ type Props = {
request: RecordRequestData | null;
};
const VIEWPORT_MARGIN = 16;
const DRAG_THRESHOLD_PX = 8;
const INTRO_DURATION_MS = 10_000;
const TIMER_REFRESH_MS = 250;
function formatElapsed(seconds: number): string {
const minutes = Math.floor(seconds / 60);
const remainingSeconds = seconds % 60;
return `${String(minutes).padStart(2, "0")}:${String(remainingSeconds).padStart(2, "0")}`;
}
function clamp(value: number, min: number, max: number): number {
return Math.max(min, Math.min(max, value));
}
function clampDragPos(
top: number,
left: number,
panelW: number,
panelH: number,
): { top: number; left: number } {
const maxLeft = window.innerWidth - panelW - VIEWPORT_MARGIN;
const maxTop = window.innerHeight - panelH - VIEWPORT_MARGIN;
return {
top: clamp(top, VIEWPORT_MARGIN, Math.max(VIEWPORT_MARGIN, maxTop)),
left: clamp(left, VIEWPORT_MARGIN, Math.max(VIEWPORT_MARGIN, maxLeft)),
};
}
export function RecordOverlay({ request }: Props) {
const { t } = useTranslation("extension");
const [show, setShow] = useState(false);
const [elapsedSeconds, setElapsedSeconds] = useState(0);
const [dragPos, setDragPos] = useState<{ top: number; left: number } | null>(null);
const [dragging, setDragging] = useState(false);
const pillRef = useRef<HTMLDivElement>(null);
const dragStartRef = useRef<{
pointerX: number;
pointerY: number;
originLeft: number;
originTop: number;
panelW: number;
panelH: number;
armed: boolean;
} | null>(null);
useEffect(() => {
if (request) {
@@ -23,8 +77,87 @@ export function RecordOverlay({ request }: Props) {
setShow(false);
}, [request]);
const requestStartedAtMs = request?.startedAtMs;
useEffect(() => {
setDragPos(null);
setDragging(false);
dragStartRef.current = null;
if (!request) {
setElapsedSeconds(0);
return;
}
const startedAt = requestStartedAtMs ?? Date.now();
const updateElapsed = () => {
setElapsedSeconds(Math.max(0, Math.floor((Date.now() - startedAt) / 1_000)));
};
updateElapsed();
const timer = window.setInterval(updateElapsed, TIMER_REFRESH_MS);
return () => window.clearInterval(timer);
}, [request?.id, requestStartedAtMs]);
const onPillPointerDown = useCallback((e: ReactPointerEvent<HTMLDivElement>) => {
if (e.button !== 0) return;
if (e.target instanceof Element && e.target.closest("button, a, input, textarea, select")) {
return;
}
const pill = pillRef.current;
if (!pill) return;
const rect = pill.getBoundingClientRect();
dragStartRef.current = {
pointerX: e.clientX,
pointerY: e.clientY,
originLeft: rect.left,
originTop: rect.top,
panelW: pill.offsetWidth || rect.width,
panelH: pill.offsetHeight || rect.height,
armed: false,
};
}, []);
useEffect(() => {
const onMove = (e: PointerEvent) => {
const start = dragStartRef.current;
if (!start) return;
const dx = e.clientX - start.pointerX;
const dy = e.clientY - start.pointerY;
if (!start.armed) {
if (Math.hypot(dx, dy) < DRAG_THRESHOLD_PX) return;
start.armed = true;
setDragging(true);
setDragPos({ top: start.originTop, left: start.originLeft });
}
setDragPos(
clampDragPos(start.originTop + dy, start.originLeft + dx, start.panelW, start.panelH),
);
};
const onUp = () => {
const start = dragStartRef.current;
if (!start) return;
if (start.armed) {
setDragging(false);
}
dragStartRef.current = null;
};
window.addEventListener("pointermove", onMove);
window.addEventListener("pointerup", onUp);
window.addEventListener("pointercancel", onUp);
return () => {
window.removeEventListener("pointermove", onMove);
window.removeEventListener("pointerup", onUp);
window.removeEventListener("pointercancel", onUp);
};
}, []);
if (!request) return null;
const introVisible = elapsedSeconds * 1_000 < INTRO_DURATION_MS;
const positioned = dragPos !== null;
const defaultTransform = show
? "translateX(-50%) translateY(0)"
: "translateX(-50%) translateY(8px)";
return (
<>
<style>{`
@@ -35,78 +168,118 @@ export function RecordOverlay({ request }: Props) {
`}</style>
<div
ref={pillRef}
data-slot="record-overlay-pill"
data-phase={introVisible ? "intro" : "timer"}
data-dragging={dragging ? "true" : "false"}
onPointerDown={onPillPointerDown}
style={{
position: "fixed",
bottom: 32,
left: "50%",
...(positioned
? { top: dragPos.top, left: dragPos.left, bottom: "auto" }
: { bottom: 32, left: "50%" }),
zIndex: 2147483647,
pointerEvents: "auto",
display: "flex",
alignItems: "center",
gap: 10,
gap: 5,
backgroundColor: "#fff",
borderRadius: 9999,
padding: "10px 10px 10px 20px",
boxShadow: "0 8px 32px rgba(15,23,42,0.16), 0 2px 8px rgba(0,0,0,0.1)",
height: 50,
boxSizing: "border-box",
padding: "6px 8px 6px 16px",
boxShadow: dragging
? "0 12px 40px rgba(15,23,42,0.22), 0 2px 8px rgba(0,0,0,0.12)"
: "0 8px 32px rgba(15,23,42,0.16), 0 2px 8px rgba(0,0,0,0.1)",
opacity: show ? 1 : 0,
transform: show ? "translateX(-50%) translateY(0)" : "translateX(-50%) translateY(8px)",
transition: "opacity 300ms ease-out, transform 300ms ease-out",
transform: positioned ? "none" : defaultTransform,
transition: dragging
? "box-shadow 150ms ease-out"
: "opacity 300ms ease-out, transform 300ms ease-out, box-shadow 150ms ease-out",
fontFamily:
'-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif',
maxWidth: "min(420px, calc(100vw - 32px))",
cursor: dragging ? "grabbing" : "grab",
userSelect: "none",
touchAction: "none",
}}
>
<span
data-slot="record-overlay-indicator"
style={{
width: 9,
height: 9,
borderRadius: "50%",
backgroundColor: "#ef4444",
flexShrink: 0,
animation: "bsk-rec-pulse 1.4s ease-in-out infinite",
}}
/>
<span
<div
data-slot="record-overlay-status"
style={{
display: "flex",
alignItems: "center",
gap: 8,
minWidth: 0,
flex: 1,
fontSize: 16,
fontWeight: 500,
color: "#333",
whiteSpace: "nowrap",
userSelect: "none",
overflow: "hidden",
textOverflow: "ellipsis",
borderRadius: 9999,
outline: "none",
minHeight: 32,
paddingRight: 4,
}}
>
{t("recordOverlay.recording")}
</span>
<span
data-slot="record-overlay-indicator"
style={{
width: 9,
height: 9,
borderRadius: "50%",
backgroundColor: "#ef4444",
flexShrink: 0,
animation: "bsk-rec-pulse 1.4s ease-in-out infinite",
}}
/>
<span
data-slot="record-overlay-label"
style={{
display: "block",
flex: "0 1 auto",
fontSize: 16,
fontWeight: 500,
color: "#333",
whiteSpace: "nowrap",
userSelect: "none",
overflow: "hidden",
textOverflow: "ellipsis",
width: introVisible ? "auto" : 48,
maxWidth: introVisible ? 280 : 48,
fontVariantNumeric: introVisible ? undefined : "tabular-nums",
textAlign: introVisible ? "left" : "center",
margin: 0,
padding: 0,
minWidth: 0,
transition: "max-width 180ms ease-out",
}}
>
{introVisible ? t("recordOverlay.recording") : formatElapsed(elapsedSeconds)}
</span>
</div>
<button
type="button"
data-slot="record-overlay-finish"
aria-label={t("recordOverlay.finish")}
onClick={request.onFinish}
style={{
pointerEvents: "auto",
display: "flex",
alignItems: "center",
gap: 6,
display: "block",
width: 32,
height: 32,
border: "none",
borderRadius: 9999,
padding: "8px 20px 8px 16px",
fontSize: 15,
fontWeight: 600,
color: "#fff",
backgroundColor: "#f97316",
padding: 0,
backgroundColor: "transparent",
cursor: "pointer",
transition: "background-color 150ms ease-out, opacity 150ms ease-out",
whiteSpace: "nowrap",
lineHeight: 1,
flexShrink: 0,
}}
>
<RiStopCircleLine size={18} color="#fff" />
{t("recordOverlay.finish")}
<img
src={recordIconUrl}
alt=""
width={24}
height={24}
draggable={false}
style={{ display: "block", margin: "auto" }}
/>
</button>
</div>
</>
@@ -1,37 +1,181 @@
import { cleanup, fireEvent, render } from "@testing-library/react";
import { i18n } from "@browser-skill/i18n";
import { I18nextProvider } from "@browser-skill/i18n/react";
import { act, cleanup, fireEvent, render, screen } from "@testing-library/react";
import { createElement } from "react";
import { afterEach, describe, expect, it, vi } from "vitest";
import { RecordOverlay } from "../RecordOverlay";
function renderOverlay(request: { id: string; startedAtMs?: number; onFinish: () => void } | null) {
return render(
createElement(I18nextProvider, { i18n }, createElement(RecordOverlay, { request })),
);
}
function advanceRecording(ms: number) {
act(() => {
vi.advanceTimersByTime(ms);
});
}
describe("RecordOverlay", () => {
afterEach(() => {
cleanup();
vi.useRealTimers();
});
it("renders the bottom pill with orange finish CTA and no logo", () => {
it("renders the bottom pill with the icon-only finish button", () => {
const onFinish = vi.fn();
const { container } = render(<RecordOverlay request={{ id: "rec-1", onFinish }} />);
const { container } = renderOverlay({ id: "rec-1", onFinish });
const pill = container.querySelector("[data-slot='record-overlay-pill']");
expect(pill).toBeTruthy();
expect(pill?.querySelector("img")).toBeNull();
expect((pill as HTMLElement).style.borderRadius).toBe("9999px");
expect((pill as HTMLElement).style.backgroundColor).toBe("#fff");
expect((pill as HTMLElement).style.gap).toBe("5px");
expect((pill as HTMLElement).style.padding).toBe("6px 8px 6px 16px");
expect((pill as HTMLElement).style.height).toBe("50px");
expect((pill as HTMLElement).style.boxSizing).toBe("border-box");
expect(container.querySelector("[data-slot='record-overlay-collapse']")).toBeNull();
const finish = container.querySelector("[data-slot='record-overlay-finish']");
const status = container.querySelector("[data-slot='record-overlay-status']") as HTMLElement;
expect(status.style.minHeight).toBe("32px");
const finish = screen.getByRole("button", {
name: i18n.t("recordOverlay.finish", { ns: "extension" }),
});
expect(finish).toBeTruthy();
expect((finish as HTMLElement).style.backgroundColor).toBe("#f97316");
expect(finish.textContent).toBe("");
const icon = finish.querySelector("img");
expect(icon).toBeTruthy();
expect(icon?.getAttribute("src")).toContain("record-icon.png");
expect((finish as HTMLElement).style.width).toBe("32px");
expect((finish as HTMLElement).style.height).toBe("32px");
expect(icon?.getAttribute("width")).toBe("24");
expect(icon?.getAttribute("height")).toBe("24");
expect((finish as HTMLElement).style.borderRadius).toBe("9999px");
fireEvent.click(finish!);
fireEvent.click(finish);
expect(onFinish).toHaveBeenCalledTimes(1);
});
it("shows a pulsing recording indicator and no full-screen glow layer", () => {
const { container } = render(<RecordOverlay request={{ id: "rec-1", onFinish: vi.fn() }} />);
const { container } = renderOverlay({ id: "rec-1", onFinish: vi.fn() });
expect(container.querySelector("[data-slot='record-overlay']")).toBeNull();
const indicator = container.querySelector("[data-slot='record-overlay-indicator']");
expect(indicator).toBeTruthy();
expect((indicator as HTMLElement).style.animation).toContain("bsk-rec-pulse");
});
it("shows the recording message for ten seconds, then displays elapsed time", () => {
vi.useFakeTimers();
const { container } = renderOverlay({ id: "rec-1", onFinish: vi.fn() });
const pill = container.querySelector("[data-slot='record-overlay-pill']") as HTMLElement;
const label = container.querySelector("[data-slot='record-overlay-label']") as HTMLElement;
expect(pill.dataset.phase).toBe("intro");
expect(label.textContent).toBe(i18n.t("recordOverlay.recording", { ns: "extension" }));
advanceRecording(9999);
expect(pill.dataset.phase).toBe("intro");
advanceRecording(1);
expect(pill.dataset.phase).toBe("timer");
expect(label.textContent).toBe("00:10");
});
it("formats elapsed time as zero-padded minutes and seconds", () => {
vi.useFakeTimers();
const { container } = renderOverlay({ id: "rec-1", onFinish: vi.fn() });
const label = container.querySelector("[data-slot='record-overlay-label']") as HTMLElement;
advanceRecording(125_000);
expect(label.textContent).toBe("02:05");
});
it("keeps a fixed timer label width with tabular numerals", () => {
vi.useFakeTimers();
const { container } = renderOverlay({ id: "rec-1", onFinish: vi.fn() });
const label = container.querySelector("[data-slot='record-overlay-label']") as HTMLElement;
advanceRecording(10_000);
expect(label.style.width).toBe("48px");
expect(label.style.maxWidth).toBe("48px");
expect(label.style.fontVariantNumeric).toBe("tabular-nums");
expect(label.style.textAlign).toBe("center");
advanceRecording(1_000);
expect(label.textContent).toBe("00:11");
expect(label.style.width).toBe("48px");
});
it("resets the intro and elapsed time for a new request", () => {
vi.useFakeTimers();
const first = { id: "rec-1", onFinish: vi.fn() };
const second = { id: "rec-2", onFinish: vi.fn() };
const { container, rerender } = renderOverlay(first);
advanceRecording(65_000);
rerender(
createElement(I18nextProvider, { i18n }, createElement(RecordOverlay, { request: second })),
);
const pill = container.querySelector("[data-slot='record-overlay-pill']") as HTMLElement;
const label = container.querySelector("[data-slot='record-overlay-label']") as HTMLElement;
expect(pill.dataset.phase).toBe("intro");
expect(label.textContent).toBe(i18n.t("recordOverlay.recording", { ns: "extension" }));
advanceRecording(10_000);
expect(pill.dataset.phase).toBe("timer");
expect(label.textContent).toBe("00:10");
});
it("continues the session timer when the overlay remounts on a new page", () => {
vi.useFakeTimers();
const startedAtMs = Date.now() - 65_000;
const { container } = renderOverlay({ id: "rec-1", startedAtMs, onFinish: vi.fn() });
const pill = container.querySelector("[data-slot='record-overlay-pill']") as HTMLElement;
const label = container.querySelector("[data-slot='record-overlay-label']") as HTMLElement;
expect(pill.dataset.phase).toBe("timer");
expect(label.textContent).toBe("01:05");
advanceRecording(5_000);
expect(label.textContent).toBe("01:10");
});
it("does not start drag when pointer down on finish", () => {
const { container } = renderOverlay({ id: "rec-1", onFinish: vi.fn() });
const pill = container.querySelector("[data-slot='record-overlay-pill']") as HTMLElement;
const finish = container.querySelector("[data-slot='record-overlay-finish']") as HTMLElement;
fireEvent.pointerDown(finish, { button: 0, clientX: 10, clientY: 10, pointerId: 1 });
expect(pill.getAttribute("data-dragging")).toBe("false");
});
it("moves the pill when dragging the non-button area without toggling collapse", () => {
const { container } = renderOverlay({ id: "rec-1", onFinish: vi.fn() });
const pill = container.querySelector("[data-slot='record-overlay-pill']") as HTMLElement;
pill.getBoundingClientRect = () =>
({
top: 600,
left: 200,
width: 320,
height: 48,
right: 520,
bottom: 648,
}) as DOMRect;
Object.defineProperty(pill, "offsetWidth", { value: 320, configurable: true });
Object.defineProperty(pill, "offsetHeight", { value: 48, configurable: true });
fireEvent.pointerDown(pill, { button: 0, clientX: 300, clientY: 620, pointerId: 1 });
fireEvent(window, new PointerEvent("pointermove", { clientX: 360, clientY: 580 }));
fireEvent(window, new PointerEvent("pointerup", { pointerId: 1 }));
expect(pill.getAttribute("data-phase")).toBe("intro");
expect(pill.style.left).toBe("260px");
expect(pill.style.top).toBe("560px");
expect(pill.style.bottom).toBe("auto");
expect(pill.style.transform).toBe("none");
});
});
@@ -0,0 +1,125 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { CAPTURE_SUPPRESS, type CaptureSuppressAck } from "@/lib/capture-suppress-bridge";
import { CAPTURE_HIDDEN_ATTR, createCaptureSuppressController } from "../capture-suppress";
/**
* rAF is stubbed with a manual queue so tests can drive the compositor
* frame-by-frame and assert the ack only lands after the second frame.
*/
describe("capture-suppress controller", () => {
let rafQueue: FrameRequestCallback[];
let host: HTMLElement;
beforeEach(() => {
rafQueue = [];
vi.stubGlobal("requestAnimationFrame", (cb: FrameRequestCallback) => {
rafQueue.push(cb);
return rafQueue.length;
});
vi.stubGlobal("cancelAnimationFrame", vi.fn());
document.body.innerHTML = "";
host = document.createElement("div");
document.body.append(host);
});
/** Run every queued rAF callback, draining microtasks between rounds. */
async function flushFrames(rounds = 4): Promise<void> {
for (let round = 0; round < rounds && rafQueue.length > 0; round += 1) {
const callbacks = rafQueue.splice(0);
for (const cb of callbacks) cb(0);
await Promise.resolve();
await Promise.resolve();
}
}
it("begin hides the host immediately but only acks after two frames", async () => {
const controller = createCaptureSuppressController(() => host);
const sendResponse = vi.fn();
const needsAsync = controller.handleMessage(
{ type: CAPTURE_SUPPRESS, phase: "begin" },
sendResponse,
);
expect(needsAsync).toBe(true);
expect(controller.pendingCount).toBe(1);
expect(host.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(true);
expect(sendResponse).not.toHaveBeenCalled();
// First frame commits the style change; the ack must still be pending.
await flushFrames(1);
expect(sendResponse).not.toHaveBeenCalled();
// Second frame: the compositor has produced an overlay-free frame.
await flushFrames();
const expected: CaptureSuppressAck = { type: CAPTURE_SUPPRESS, ok: true };
expect(sendResponse).toHaveBeenCalledWith(expected);
});
it("end removes the attribute once the count returns to zero", async () => {
const controller = createCaptureSuppressController(() => host);
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "begin" }, vi.fn());
await flushFrames();
expect(host.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(true);
const sendResponse = vi.fn();
const needsAsync = controller.handleMessage(
{ type: CAPTURE_SUPPRESS, phase: "end" },
sendResponse,
);
expect(needsAsync).toBe(false);
expect(controller.pendingCount).toBe(0);
expect(host.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(false);
expect(sendResponse).toHaveBeenCalledWith({ type: CAPTURE_SUPPRESS, ok: true });
});
it("reference-counts concurrent begins so one end does not unhide early", async () => {
const controller = createCaptureSuppressController(() => host);
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "begin" }, vi.fn());
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "begin" }, vi.fn());
await flushFrames();
expect(controller.pendingCount).toBe(2);
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "end" }, vi.fn());
expect(controller.pendingCount).toBe(1);
expect(host.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(true);
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "end" }, vi.fn());
expect(controller.pendingCount).toBe(0);
expect(host.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(false);
});
it("treats a stray end as a no-op", () => {
const controller = createCaptureSuppressController(() => host);
const sendResponse = vi.fn();
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "end" }, sendResponse);
expect(controller.pendingCount).toBe(0);
expect(host.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(false);
expect(sendResponse).toHaveBeenCalledWith({ type: CAPTURE_SUPPRESS, ok: true });
});
it("re-applies the attribute when the host is remounted mid-capture", () => {
let currentHost: HTMLElement | null = host;
const controller = createCaptureSuppressController(() => currentHost);
controller.handleMessage({ type: CAPTURE_SUPPRESS, phase: "begin" }, vi.fn());
// Host lost and rebuilt before `end` arrives.
currentHost = null;
const rebuilt = document.createElement("div");
controller.onHostMounted(rebuilt);
expect(rebuilt.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(true);
});
it("leaves a remounted host visible once suppression has ended", () => {
const controller = createCaptureSuppressController(() => host);
const rebuilt = document.createElement("div");
controller.onHostMounted(rebuilt);
expect(rebuilt.hasAttribute(CAPTURE_HIDDEN_ATTR)).toBe(false);
});
});
File diff suppressed because it is too large Load Diff
@@ -1,109 +0,0 @@
import { describe, expect, it, vi } from "vitest";
import { isHoverSurfaceCandidateElement, isLikelyHoverSurfaceOwner } from "../record-hover-surface";
function mockRect(
el: Element,
rect: { left: number; top: number; width: number; height: number },
): void {
vi.spyOn(el, "getBoundingClientRect").mockReturnValue({
x: rect.left,
y: rect.top,
top: rect.top,
left: rect.left,
right: rect.left + rect.width,
bottom: rect.top + rect.height,
width: rect.width,
height: rect.height,
toJSON: () => ({}),
});
}
describe("record-hover-surface", () => {
it("recognizes popover containers without treating topbar wrappers or menu items as surfaces", () => {
document.body.innerHTML = `
<ul class="header-menu tw-flex navbar-app__authentication-menu">
<li class="tg-popover">
<div class="tg-avatar tg-avatar--img"></div>
</li>
</ul>
<div class="tg-popover__popper">
<div class="tg-popover__content">
<div class="tg-popover__content-body">
<li class="header-menu__item">
<a class="header-dropdown-menu__link" href="/u/me">My profile</a>
</li>
<li class="header-menu__item">
<a class="header-dropdown-menu__link" href="/dashboard/groups">My groups</a>
</li>
</div>
</div>
</div>
`;
expect(isHoverSurfaceCandidateElement(document.querySelector(".header-menu")!)).toBe(false);
expect(isHoverSurfaceCandidateElement(document.querySelector(".tg-popover")!)).toBe(false);
expect(isHoverSurfaceCandidateElement(document.querySelector(".header-menu__item")!)).toBe(
false,
);
expect(
isHoverSurfaceCandidateElement(document.querySelector(".header-dropdown-menu__link")!),
).toBe(false);
expect(isHoverSurfaceCandidateElement(document.querySelector(".tg-popover__popper")!)).toBe(
true,
);
expect(isHoverSurfaceCandidateElement(document.querySelector(".tg-popover__content")!)).toBe(
true,
);
expect(
isHoverSurfaceCandidateElement(document.querySelector(".tg-popover__content-body")!),
).toBe(true);
});
it("does not treat elements covered by an open surface as the surface owner", () => {
document.body.innerHTML = `
<button class="avatar">image</button>
<a class="fork" href="/fork">Fork</a>
<button class="clone">Clone & Download</button>
<div class="tg-popover__content-body"></div>
`;
const avatar = document.querySelector(".avatar")!;
const fork = document.querySelector(".fork")!;
const clone = document.querySelector(".clone")!;
const surface = document.querySelector(".tg-popover__content-body")!;
mockRect(avatar, { left: 1340, top: 11, width: 26, height: 26 });
mockRect(fork, { left: 1275, top: 56, width: 69, height: 28 });
mockRect(clone, { left: 1159, top: 111, width: 180, height: 30 });
mockRect(surface, { left: 1111, top: 44, width: 240, height: 496 });
expect(isLikelyHoverSurfaceOwner(avatar, surface)).toBe(true);
expect(isLikelyHoverSurfaceOwner(fork, surface)).toBe(false);
expect(isLikelyHoverSurfaceOwner(clone, surface)).toBe(false);
});
it("does not treat fixed navigation containers as plain floating surfaces", () => {
document.body.innerHTML = `
<header class="navbar-app">
<a href="/fork">Fork</a>
<button>image</button>
</header>
<div class="plain-floating"></div>
`;
const navbar = document.querySelector(".navbar-app")!;
const floating = document.querySelector(".plain-floating")!;
mockRect(navbar, { left: 0, top: 0, width: 1386, height: 51 });
mockRect(floating, { left: 1111, top: 44, width: 240, height: 496 });
vi.spyOn(window, "getComputedStyle").mockImplementation((el) => {
const style = {
cursor: "",
display: "block",
pointerEvents: "auto",
position: el === navbar ? "fixed" : "absolute",
visibility: "visible",
} as CSSStyleDeclaration;
return style;
});
expect(isHoverSurfaceCandidateElement(navbar)).toBe(false);
expect(isHoverSurfaceCandidateElement(floating)).toBe(true);
});
});
@@ -0,0 +1,99 @@
/**
* Content-script side of the capture-suppress bridge: hides the overlay
* shadow host while the background captures a screenshot.
*
* `begin` sets `data-bsk-capture-hidden` on the host (overlay.css maps it
* to `display: none !important`, which removes the whole shadow subtree
* from compositing) and only acks after two animation frames — the first
* frame commits the style change, the second guarantees the compositor
* has produced a frame without the overlay, so the capture API cannot
* pick up a stale frame that still contains it. `end` is
* reference-counted so overlapping captures do not unhide the host early.
*/
import {
CAPTURE_SUPPRESS,
type CaptureSuppressAck,
type CaptureSuppressMessage,
} from "@/lib/capture-suppress-bridge";
export const CAPTURE_HIDDEN_ATTR = "data-bsk-capture-hidden";
/**
* Safety net for hidden/background tabs where rAF callbacks never fire:
* the attribute is already set synchronously, so any frame the compositor
* produces from now on excludes the overlay. Waiting longer would just
* deadlock the capture.
*/
const FRAME_FALLBACK_MS = 500;
function nextFrame(): Promise<void> {
return new Promise((resolve) => {
const raf = requestAnimationFrame(() => {
clearTimeout(timer);
resolve();
});
const timer = setTimeout(() => {
cancelAnimationFrame(raf);
resolve();
}, FRAME_FALLBACK_MS);
});
}
export interface CaptureSuppressController {
/**
* Handle one bridge message. Returns true when the ack is sent
* asynchronously and the caller must keep the message channel open.
*/
handleMessage(
message: CaptureSuppressMessage,
sendResponse: (ack: CaptureSuppressAck) => void,
): boolean;
/** Re-apply the hidden attribute when the overlay host is (re)mounted. */
onHostMounted(host: HTMLElement): void;
/** Number of in-flight `begin` phases (exposed for tests). */
readonly pendingCount: number;
}
export function createCaptureSuppressController(
getHost: () => HTMLElement | null,
): CaptureSuppressController {
let pending = 0;
const ack: CaptureSuppressAck = { type: CAPTURE_SUPPRESS, ok: true };
function setHidden(host: HTMLElement | null, hidden: boolean): void {
if (!host) return;
if (hidden) host.setAttribute(CAPTURE_HIDDEN_ATTR, "");
else host.removeAttribute(CAPTURE_HIDDEN_ATTR);
}
return {
get pendingCount() {
return pending;
},
handleMessage(message, sendResponse) {
if (message.phase === "begin") {
pending += 1;
setHidden(getHost(), true);
void (async () => {
await nextFrame();
await nextFrame();
sendResponse(ack);
})();
return true;
}
// `end` without a matching `begin` (e.g. a retried message) must not
// drive the counter negative or clear a concurrent capture's flag.
pending = Math.max(0, pending - 1);
if (pending === 0) setHidden(getHost(), false);
sendResponse(ack);
return false;
},
onHostMounted(host) {
// Host lost + rebuilt mid-capture: the fresh host must stay hidden
// until the last `end` arrives.
setHidden(host, pending > 0);
},
};
}
+7
View File
@@ -14,3 +14,10 @@
:host([data-bsk-overlay-surface][data-bsk-overlay-blocking]) {
pointer-events: auto !important;
}
/* Screenshot suppression: the host lives in the main DOM, so hiding it
removes the whole shadow subtree from compositing and captured frames
only contain page content. */
:host([data-bsk-capture-hidden]) {
display: none !important;
}
+3 -491
View File
@@ -1,17 +1,4 @@
import {
describeEventTarget,
describeTarget,
resolveClickableElement,
resolveHoverElement,
type TargetDescriptor,
} from "@/lib/describe-target";
import {
evaluateHoverTrigger,
type HoverTriggerDecision,
type HoverTriggerRect,
hasDirectHoverInteractiveSignal,
hasStrongHoverExpansionSignal,
} from "@/lib/hover-trigger-policy";
import { describeEventTarget, describeTarget, type TargetDescriptor } from "@/lib/describe-target";
import {
isRecordCancelMessage,
isRecordStartMessage,
@@ -28,14 +15,6 @@ import {
type RecordStopMessage,
} from "@/lib/record-bridge";
import { shouldRecordPress } from "@/lib/trace-reducer";
import {
closestHoverSurfaceCandidate,
collectHoverSurfaceStates,
decideHoverSurfaceRelation,
type HoverSurfaceState,
isHoverSurfaceCandidateElement,
isLikelyHoverSurfaceOwner,
} from "./record-hover-surface";
const pendingStepSends = new Map<string, Set<Promise<boolean>>>();
const failedStepDeliveries = new Set<string>();
@@ -70,29 +49,8 @@ interface FillSession {
lastValue: string;
}
interface HoverCandidate {
element: Element;
target: TargetDescriptor;
recordedAt: number;
score: number;
eligible: boolean;
}
interface HoverSurfaceNode {
element: Element;
signature: string;
owner: HoverCandidate;
parent?: HoverSurfaceNode;
}
type FillableElement = HTMLInputElement | HTMLTextAreaElement | HTMLElement;
const HOVER_BEFORE_CLICK_MAX_MS = 10_000;
const HOVER_SURFACE_CONTEXT_MAX_MS = 30_000;
const HOVER_REPLACE_SCORE_MARGIN = 50;
const HOVER_CANDIDATE_LIMIT = 24;
const HOVER_TRIGGER_LABEL_MAX = 48;
function eventTarget(event: Event): EventTarget | null {
return event.composedPath()[0] ?? event.target;
}
@@ -164,173 +122,6 @@ function fillableValue(el: FillableElement): string {
return el.textContent ?? "";
}
function hoverTriggerAttrs(el: Element): Record<string, string | undefined> {
return {
id: el.id || undefined,
class: typeof el.className === "string" ? el.className || undefined : undefined,
"data-testid": el.getAttribute("data-testid") ?? undefined,
"data-test": el.getAttribute("data-test") ?? undefined,
"data-cy": el.getAttribute("data-cy") ?? undefined,
"aria-label": el.getAttribute("aria-label") ?? undefined,
"aria-haspopup": el.getAttribute("aria-haspopup") ?? undefined,
"aria-controls": el.getAttribute("aria-controls") ?? undefined,
"aria-expanded": el.getAttribute("aria-expanded") ?? undefined,
"aria-hidden": el.getAttribute("aria-hidden") ?? undefined,
"aria-disabled": el.getAttribute("aria-disabled") ?? undefined,
contenteditable: el.getAttribute("contenteditable") ?? undefined,
disabled: el.hasAttribute("disabled") ? "" : undefined,
hidden: el.hasAttribute("hidden") ? "" : undefined,
inert: el.hasAttribute("inert") ? "" : undefined,
onclick: el.getAttribute("onclick") ?? undefined,
onmouseenter: el.getAttribute("onmouseenter") ?? undefined,
onmouseover: el.getAttribute("onmouseover") ?? undefined,
role: el.getAttribute("role") ?? undefined,
tabindex: el.getAttribute("tabindex") ?? undefined,
title: el.getAttribute("title") ?? undefined,
};
}
function hoverTriggerRect(el: Element): HoverTriggerRect {
const rect = el.getBoundingClientRect();
return { x: rect.left, y: rect.top, w: rect.width, h: rect.height };
}
function hoverTriggerStyle(el: Element): { cursor?: string; pointerEvents?: string } {
if (!(el instanceof HTMLElement)) return {};
const style = getComputedStyle(el);
return { cursor: style.cursor, pointerEvents: style.pointerEvents };
}
function hasHoverGraphicDescendant(el: Element): boolean {
return el.querySelector("img,svg,use,path,i") !== null;
}
function normalizeLabelText(value: string): string {
return value.replace(/\s+/g, " ").trim();
}
function truncateLabel(value: string, max = HOVER_TRIGGER_LABEL_MAX): string {
const normalized = normalizeLabelText(value);
if (normalized.length <= max) return normalized;
return `${normalized.slice(0, max - 1)}…`;
}
function collectHoverTriggerLabelText(root: Element): string {
let text = "";
const visit = (node: Node) => {
if (node.nodeType === Node.TEXT_NODE) {
text += ` ${node.textContent ?? ""}`;
return;
}
if (!(node instanceof Element)) return;
if (node !== root && isHoverSurfaceCandidateElement(node)) {
return;
}
for (const child of node.childNodes) visit(child);
};
for (const child of root.childNodes) visit(child);
return normalizeLabelText(text);
}
function compactHoverTargetName(el: Element, desc: TargetDescriptor): TargetDescriptor {
if (!desc.name) return desc;
const fullText = normalizeLabelText(el.textContent ?? "");
const compactName = collectHoverTriggerLabelText(el);
if (!compactName || compactName === desc.name) return desc;
const compactComparable = compactName.replace(/\s+/g, "");
const descComparable = desc.name.replace(/…$/, "").replace(/\s+/g, "");
const fullTextComparable = fullText.replace(/\s+/g, "");
if (
descComparable.startsWith(compactComparable) &&
fullTextComparable.startsWith(descComparable) &&
compactName.length < desc.name.length
) {
return { ...desc, name: truncateLabel(compactName) };
}
return desc;
}
function isWeakHoverTarget(target: TargetDescriptor): boolean {
return !target.role && !target.name && target.tag === "div";
}
function looksLikeAvatarElement(el: Element): boolean {
const className = typeof el.className === "string" ? el.className : "";
return /\b(avatar|user-avatar)\b/i.test(className);
}
function normalizeHoverTarget(
el: Element,
desc: TargetDescriptor,
decision: HoverTriggerDecision,
): TargetDescriptor {
if (desc.role === "img" && !desc.name && looksLikeAvatarElement(el)) {
return { ...desc, name: "image" };
}
if (
isWeakHoverTarget(desc) &&
(looksLikeAvatarElement(el) ||
(hasHoverGraphicDescendant(el) && decision.reasons.includes("icon-only")))
) {
return { tag: "img", role: "img", name: "image" };
}
return desc;
}
function hoverTriggerSignals(el: Element, desc: TargetDescriptor) {
if (!(el instanceof HTMLElement)) return null;
const style = hoverTriggerStyle(el);
return {
tag: el.tagName.toLowerCase(),
role: desc.role,
label: desc.name,
attrs: hoverTriggerAttrs(el),
rect: hoverTriggerRect(el),
cursor: style.cursor,
pointerEvents: style.pointerEvents,
hasGraphicDescendant: hasHoverGraphicDescendant(el),
};
}
function hoverCandidateFromEvent(target: EventTarget | null): HoverCandidate | null {
if (!(target instanceof Element)) return null;
const element = resolveHoverElement(target);
if (!element) return null;
const desc = describeTarget(element);
const signals = hoverTriggerSignals(element, desc);
if (!signals) return null;
const decision = evaluateHoverTrigger(signals);
const hasExpansionSignal = hasStrongHoverExpansionSignal(signals);
const hasDirectSignal = hasDirectHoverInteractiveSignal(signals);
if (!decision.eligible && !hasExpansionSignal && !hasDirectSignal) return null;
if (!decision.eligible && !desc.name && !desc.role) return null;
const normalizedTarget = normalizeHoverTarget(
element,
compactHoverTargetName(element, desc),
decision,
);
const now = Date.now();
return {
element,
target: normalizedTarget,
recordedAt: now,
score: decision.score,
eligible: decision.eligible,
};
}
function shouldReplaceHoverCandidate(
current: HoverCandidate,
next: HoverCandidate,
now: number,
): boolean {
if (next.element === current.element) return false;
if (now - current.recordedAt > HOVER_BEFORE_CLICK_MAX_MS) return true;
if (current.element.contains(next.element)) return false;
if (next.element.contains(current.element)) return true;
return next.score >= current.score + HOVER_REPLACE_SCORE_MARGIN;
}
/** Clicks that only pick an autocomplete/suggestion value — not a semantic submit action. */
function isInputCompletionClick(target: EventTarget | null, session: FillSession | null): boolean {
if (!session || !(target instanceof Element)) return false;
@@ -377,17 +168,10 @@ export function startRecordCapture(
const emitStep = (step: RecordStepPayload) => {
sendStep({ page_url: location.href, ...step });
};
const hoverSurfaceStateMap = (states: HoverSurfaceState[]): Map<Element, string> =>
new Map(states.map((state) => [state.element, state.signature]));
let fillSession: FillSession | null = null;
let composing = false;
let lastUrl = location.href;
let keyboardActivation: { target: EventTarget | null; recordedAt: number } | undefined;
let pendingHover: HoverCandidate | null = null;
const recentHoverCandidates: HoverCandidate[] = [];
const emittedHoverElements = new WeakSet<Element>();
let hoverSurfaceStates = hoverSurfaceStateMap(collectHoverSurfaceStates());
const hoverSurfaceNodes = new Map<Element, HoverSurfaceNode>();
let generatedControlClick: Element | null = null;
let navigationActionPending = false;
let navigationActionVersion = 0;
@@ -453,223 +237,6 @@ export function startRecordCapture(
});
};
const rememberHoverCandidate = (hover: HoverCandidate) => {
const duplicate = recentHoverCandidates.findIndex(
(candidate) => candidate.element === hover.element,
);
if (duplicate >= 0) recentHoverCandidates.splice(duplicate, 1);
recentHoverCandidates.push(hover);
if (recentHoverCandidates.length > HOVER_CANDIDATE_LIMIT) {
recentHoverCandidates.splice(0, recentHoverCandidates.length - HOVER_CANDIDATE_LIMIT);
}
};
const surfaceArea = (el: Element): number => {
const rect = el.getBoundingClientRect();
return rect.width * rect.height;
};
const smallestContaining = <T>(
target: Element,
items: Iterable<T>,
elementFor: (item: T) => Element,
): T | undefined => {
let best: T | undefined;
for (const item of items) {
const element = elementFor(item);
if (!element.contains(target)) continue;
if (!best || surfaceArea(element) < surfaceArea(elementFor(best))) {
best = item;
}
}
return best;
};
const ownedSurfaceContaining = (target: Element): HoverSurfaceNode | undefined =>
smallestContaining(target, hoverSurfaceNodes.values(), (node) => node.element);
const surfaceStateContaining = (
target: Element,
states: HoverSurfaceState[],
): HoverSurfaceState | undefined => smallestContaining(target, states, (state) => state.element);
const inferOwnerForSurface = (
surface: Element,
now: number,
before?: HoverCandidate,
): HoverCandidate | undefined => {
for (let index = recentHoverCandidates.length - 1; index >= 0; index -= 1) {
const candidate = recentHoverCandidates[index];
if (!candidate) continue;
if (before && candidate.element === before.element) continue;
if (before && candidate.recordedAt > before.recordedAt) continue;
if (surface.contains(candidate.element)) continue;
if (now - candidate.recordedAt > HOVER_SURFACE_CONTEXT_MAX_MS) continue;
if (isLikelyHoverSurfaceOwner(candidate.element, surface)) return candidate;
}
return undefined;
};
const nodeForUnownedSurface = (
surfaceState: HoverSurfaceState,
currentStates: HoverSurfaceState[],
now: number,
): HoverSurfaceNode | undefined => {
const owner = inferOwnerForSurface(surfaceState.element, now);
if (!owner) return undefined;
const parent = parentSurfaceNodeForOwner(owner, currentStates, now);
const node: HoverSurfaceNode = {
element: surfaceState.element,
signature: surfaceState.signature,
owner,
...(parent ? { parent } : {}),
};
hoverSurfaceNodes.set(surfaceState.element, node);
return node;
};
const ownedSurfaceForAction = (target: Element): HoverSurfaceNode | undefined => {
const closestSurface = closestHoverSurfaceCandidate(target);
if (closestSurface) {
const owned = hoverSurfaceNodes.get(closestSurface);
if (owned) return owned;
}
const ownedContaining = ownedSurfaceContaining(target);
if (ownedContaining) return ownedContaining;
const now = Date.now();
const currentStates = collectHoverSurfaceStates();
pruneGoneHoverSurfaces(currentStates);
const surfaceState = closestSurface
? currentStates.find((state) => state.element === closestSurface)
: surfaceStateContaining(target, currentStates);
hoverSurfaceStates = hoverSurfaceStateMap(currentStates);
if (!surfaceState) return undefined;
return nodeForUnownedSurface(surfaceState, currentStates, now);
};
const pruneGoneHoverSurfaces = (currentStates: HoverSurfaceState[]) => {
const current = new Set(currentStates.map((state) => state.element));
for (const element of hoverSurfaceNodes.keys()) {
if (!current.has(element)) hoverSurfaceNodes.delete(element);
}
};
const parentSurfaceNodeForOwner = (
owner: HoverCandidate,
currentStates: HoverSurfaceState[],
now: number,
): HoverSurfaceNode | undefined => {
const ownedParent = ownedSurfaceContaining(owner.element);
if (ownedParent) return ownedParent;
const parentState = surfaceStateContaining(owner.element, currentStates);
if (!parentState) return undefined;
const parentOwner = inferOwnerForSurface(parentState.element, now, owner);
if (!parentOwner) return undefined;
const parentNode: HoverSurfaceNode = {
element: parentState.element,
signature: parentState.signature,
owner: parentOwner,
};
hoverSurfaceNodes.set(parentState.element, parentNode);
return parentNode;
};
const createSurfaceNode = (
state: HoverSurfaceState,
owner: HoverCandidate,
currentStates: HoverSurfaceState[],
now: number,
): HoverSurfaceNode | undefined => {
const parent = parentSurfaceNodeForOwner(owner, currentStates, now);
if (parent?.element === state.element) return undefined;
if (now - owner.recordedAt > HOVER_SURFACE_CONTEXT_MAX_MS) return undefined;
if (!isLikelyHoverSurfaceOwner(owner.element, state.element)) return undefined;
const node: HoverSurfaceNode = {
element: state.element,
signature: state.signature,
owner,
...(parent ? { parent } : {}),
};
hoverSurfaceNodes.set(state.element, node);
return node;
};
const bindChangedHoverSurfaces = (
previousStates: Map<Element, string>,
currentStates: HoverSurfaceState[],
now: number,
) => {
for (const state of currentStates) {
if (previousStates.get(state.element) === state.signature) continue;
const owner = inferOwnerForSurface(state.element, now);
if (!owner) continue;
createSurfaceNode(state, owner, currentStates, now);
}
};
const refreshHoverSurfaces = (previousStates = hoverSurfaceStates) => {
const now = Date.now();
const currentStates = collectHoverSurfaceStates();
pruneGoneHoverSurfaces(currentStates);
bindChangedHoverSurfaces(previousStates, currentStates, now);
hoverSurfaceStates = hoverSurfaceStateMap(currentStates);
};
const scheduleHoverSurfaceRefresh = (previousStates: Map<Element, string>) => {
setTimeout(() => refreshHoverSurfaces(previousStates), 0);
};
const emitHoverStep = (hover: HoverCandidate) => {
if (emittedHoverElements.has(hover.element)) return;
emitStep({
op: "hover",
target: hover.target,
});
emittedHoverElements.add(hover.element);
};
const surfaceOwnerChain = (
surface: HoverSurfaceNode,
actionElement: Element,
now: number,
): HoverCandidate[] => {
const chain: HoverCandidate[] = [];
const selected = new WeakSet<Element>();
const surfaces: HoverSurfaceNode[] = [];
let node: HoverSurfaceNode | undefined = surface;
while (node && surfaces.length < 4) {
surfaces.unshift(node);
node = node.parent;
}
for (const owned of surfaces) {
const owner = owned.owner;
if (owner.element === actionElement || actionElement.contains(owner.element)) {
continue;
}
if (selected.has(owner.element)) continue;
if (emittedHoverElements.has(owner.element)) continue;
if (now - owner.recordedAt > HOVER_SURFACE_CONTEXT_MAX_MS) continue;
selected.add(owner.element);
chain.push(owner);
}
return chain;
};
const containedHoverOwnerForAction = (
actionElement: Element,
now: number,
): HoverCandidate | undefined => {
for (let index = recentHoverCandidates.length - 1; index >= 0; index -= 1) {
const candidate = recentHoverCandidates[index];
if (!candidate) continue;
if (candidate.element === actionElement) continue;
if (!candidate.element.contains(actionElement)) continue;
if (now - candidate.recordedAt > HOVER_SURFACE_CONTEXT_MAX_MS) continue;
return candidate;
}
return undefined;
};
const emitClick = (event: MouseEvent) => {
// Only record clicks an LLM can re-identify (named interactive controls).
const target = describeEventTarget(eventTarget(event));
@@ -682,54 +249,6 @@ export function startRecordCapture(
});
};
const emitHoverCandidateBeforeAction = (actionTarget: EventTarget | null) => {
if (!(actionTarget instanceof Element)) return;
const actionElement = resolveClickableElement(actionTarget) ?? actionTarget;
const surface = ownedSurfaceForAction(actionElement);
const now = Date.now();
const containedOwner = surface ? undefined : containedHoverOwnerForAction(actionElement, now);
const hoverChain = surface
? surfaceOwnerChain(surface, actionElement, now)
: containedOwner
? [containedOwner]
: [];
if (hoverChain.length === 0) {
pendingHover = null;
return;
}
for (const hover of hoverChain) emitHoverStep(hover);
pendingHover = null;
};
const onMouseOver = (event: MouseEvent) => {
const target = eventTarget(event);
if (isOverlayTarget(target)) return;
const now = Date.now();
if (pendingHover && target instanceof Element && pendingHover.element.contains(target)) {
if (now - pendingHover.recordedAt <= HOVER_BEFORE_CLICK_MAX_MS) return;
pendingHover = null;
}
const candidate = hoverCandidateFromEvent(target);
if (candidate) {
const previousSurfaceStates = new Map(hoverSurfaceStates);
rememberHoverCandidate(candidate);
refreshHoverSurfaces(previousSurfaceStates);
scheduleHoverSurfaceRefresh(previousSurfaceStates);
}
const hover = candidate?.eligible ? candidate : null;
if (pendingHover && target instanceof Element) {
const relation = decideHoverSurfaceRelation({ triggerElement: pendingHover.element }, target);
if (relation.related && now - pendingHover.recordedAt <= HOVER_BEFORE_CLICK_MAX_MS) {
return;
}
}
if (!hover) return;
if (pendingHover && !shouldReplaceHoverCandidate(pendingHover, hover, now)) {
return;
}
pendingHover = hover;
};
const onClick = (event: MouseEvent) => {
if (event.button !== 0) return;
const target = eventTarget(event);
@@ -759,7 +278,6 @@ export function startRecordCapture(
const fillable = fillableFromTarget(target);
if (fillable) {
emitHoverCandidateBeforeAction(fillable);
ensureFillSession(fillable);
return;
}
@@ -767,7 +285,6 @@ export function startRecordCapture(
if (target instanceof Element) {
const nearbyFillable = nearbyFillableFromSearchChrome(target);
if (nearbyFillable) {
emitHoverCandidateBeforeAction(nearbyFillable);
ensureFillSession(nearbyFillable);
return;
}
@@ -787,7 +304,6 @@ export function startRecordCapture(
commitFillSession();
if (target instanceof Element && target.closest("select")) return;
emitHoverCandidateBeforeAction(target);
emitClick(event);
};
@@ -829,7 +345,6 @@ export function startRecordCapture(
commitFillSession();
const target = eventTarget(event);
if (target instanceof HTMLSelectElement) {
emitHoverCandidateBeforeAction(target);
const values = Array.from(target.selectedOptions).map((opt) => opt.value);
const labels = Array.from(target.selectedOptions).map((opt) =>
(opt.label || opt.textContent || opt.value).trim(),
@@ -883,7 +398,6 @@ export function startRecordCapture(
}
const desc = describeEventTarget(target);
if (!desc && !event.key) return;
emitHoverCandidateBeforeAction(target);
emitStep({
op: "press",
key: event.key,
@@ -894,7 +408,6 @@ export function startRecordCapture(
};
document.addEventListener("click", onClick, true);
document.addEventListener("mouseover", onMouseOver, true);
document.addEventListener("focusin", onFocusIn, true);
document.addEventListener("focusout", onFocusOut, true);
document.addEventListener("input", onInput, true);
@@ -925,7 +438,6 @@ export function startRecordCapture(
dispose() {
commitFillSession();
document.removeEventListener("click", onClick, true);
document.removeEventListener("mouseover", onMouseOver, true);
document.removeEventListener("focusin", onFocusIn, true);
document.removeEventListener("focusout", onFocusOut, true);
document.removeEventListener("input", onInput, true);
@@ -956,7 +468,7 @@ export function handleRecordContentMessage(
capture: RecordCaptureController | null;
setActiveRequestId(id: string | null): void;
setCapture(capture: RecordCaptureController | null): void;
onStart(requestId: string): void;
onStart(requestId: string, startedAtMs?: number): void;
onStop(): void;
},
sendResponse?: (response: RecordStartAck | RecordStopAck) => void,
@@ -973,7 +485,7 @@ export function handleRecordContentMessage(
}),
);
state.setActiveRequestId(message.requestId);
state.onStart(message.requestId);
state.onStart(message.requestId, message.startedAtMs);
sendResponse?.({ ok: true });
return sendResponse !== undefined;
}
@@ -1,237 +0,0 @@
export interface HoverSurfaceDecision {
related: boolean;
reason?: string;
surface?: Element;
}
export interface HoverSurfaceContext {
triggerElement: Element;
}
export interface HoverSurfaceState {
element: Element;
signature: string;
}
const HOVER_SURFACE_SELECTOR = [
'[role="menu"]',
'[role="menubar"]',
'[role="listbox"]',
'[role="dialog"]',
'[role="tooltip"]',
'[aria-modal="true"]',
"[data-popper-placement]",
"[data-floating-ui-placement]",
"[data-headlessui-state]",
"[data-radix-popper-content-wrapper]",
].join(",");
const HOVER_SURFACE_WORD_RE =
/(^|[^a-z0-9])(dropdown-menu|drop-down-menu|menu-list|popover__(popper|content|content-body)|popover-content|popper|popup|flyout|submenu|submenu-wrapper|context-menu|account-menu|user-menu|avatar-menu)(?=[^a-z0-9]|$)/i;
const HOVER_SURFACE_ITEM_RE =
/(^|[^a-z0-9])((dropdown|menu|submenu|context-menu)(-|__?)?(item|link)|item-(text|label|content)|item__?(text|label|content))(?=[^a-z0-9]|$)/i;
const HOVER_SURFACE_MAX_AXIS_GAP = 96;
const HOVER_SURFACE_MAX_DIAGONAL_GAP = 16;
const HOVER_SURFACE_MAX_VIEWPORT_AREA_RATIO = 0.5;
function elementTokenText(el: Element): string {
return [
el.id,
typeof el.className === "string" ? el.className : "",
el.getAttribute("data-testid"),
el.getAttribute("data-test"),
el.getAttribute("data-cy"),
el.getAttribute("aria-label"),
el.getAttribute("role"),
]
.filter((value): value is string => typeof value === "string" && value.length > 0)
.join(" ");
}
export function isRecognizedHoverSurfaceElement(el: Element): boolean {
if (el.matches(HOVER_SURFACE_SELECTOR)) return true;
const tokenText = elementTokenText(el);
if (HOVER_SURFACE_WORD_RE.test(tokenText) && !HOVER_SURFACE_ITEM_RE.test(tokenText)) {
return true;
}
return false;
}
function isVisibleSurfaceElement(el: Element): boolean {
if (!(el instanceof HTMLElement)) return false;
const style = getComputedStyle(el);
if (style.pointerEvents === "none" || style.visibility === "hidden" || style.display === "none") {
return false;
}
return rectFor(el) !== null;
}
export function closestRecognizedHoverSurface(element: Element): Element | null {
let node: Element | null = element;
let depth = 0;
while (node && node !== document.body && node !== document.documentElement && depth < 10) {
if (isRecognizedHoverSurfaceElement(node)) return node;
node = node.parentElement;
depth += 1;
}
return null;
}
export function isPositionedFloatingElement(el: Element): boolean {
if (!(el instanceof HTMLElement)) return false;
const style = getComputedStyle(el);
if (style.position !== "absolute") return false;
if (style.pointerEvents === "none" || style.visibility === "hidden" || style.display === "none") {
return false;
}
const rect = rectFor(el);
if (!rect) return false;
const viewportArea = window.innerWidth * window.innerHeight;
const area = rect.width * rect.height;
return (
area > 0 && (!viewportArea || area <= viewportArea * HOVER_SURFACE_MAX_VIEWPORT_AREA_RATIO)
);
}
export function isHoverSurfaceCandidateElement(el: Element): boolean {
return isRecognizedHoverSurfaceElement(el) || isPositionedFloatingElement(el);
}
export function closestHoverSurfaceCandidate(element: Element): Element | null {
let node: Element | null = element;
let depth = 0;
while (node && node !== document.body && node !== document.documentElement && depth < 10) {
if (isHoverSurfaceCandidateElement(node)) return node;
node = node.parentElement;
depth += 1;
}
return null;
}
function closestFloatingSurface(clickElement: Element): Element | null {
let node: Element | null = clickElement;
let depth = 0;
while (node && node !== document.body && node !== document.documentElement && depth < 10) {
if (isPositionedFloatingElement(node)) return node;
node = node.parentElement;
depth += 1;
}
return null;
}
function hoverSurfaceSignature(el: Element): string {
const rect = rectFor(el);
return [
rect
? `${Math.round(rect.left)},${Math.round(rect.top)},${Math.round(rect.width)},${Math.round(rect.height)}`
: "",
(el.textContent ?? "").trim().slice(0, 512),
].join("|");
}
export function collectHoverSurfaceStates(): HoverSurfaceState[] {
const states: HoverSurfaceState[] = [];
for (const el of document.querySelectorAll("*")) {
if (isHoverSurfaceCandidateElement(el) && isVisibleSurfaceElement(el)) {
states.push({ element: el, signature: hoverSurfaceSignature(el) });
}
}
return states;
}
function rectFor(el: Element): DOMRect | null {
const rect = el.getBoundingClientRect();
if (rect.width <= 0 || rect.height <= 0) return null;
return rect;
}
function surfaceRelationMetrics(surface: Element, trigger: Element) {
const surfaceRect = rectFor(surface);
const triggerRect = rectFor(trigger);
if (!surfaceRect || !triggerRect) return null;
const horizontalOverlap =
Math.min(triggerRect.right, surfaceRect.right) - Math.max(triggerRect.left, surfaceRect.left);
const verticalOverlap =
Math.min(triggerRect.bottom, surfaceRect.bottom) - Math.max(triggerRect.top, surfaceRect.top);
const horizontalGap = Math.max(
0,
triggerRect.left - surfaceRect.right,
surfaceRect.left - triggerRect.right,
);
const verticalGap = Math.max(
0,
triggerRect.top - surfaceRect.bottom,
surfaceRect.top - triggerRect.bottom,
);
return {
surfaceRect,
triggerRect,
horizontalOverlap,
verticalOverlap,
horizontalGap,
verticalGap,
};
}
function isNearTrigger(surface: Element, trigger: Element): boolean {
const metrics = surfaceRelationMetrics(surface, trigger);
if (!metrics) return true;
const { horizontalOverlap, verticalOverlap, horizontalGap, verticalGap } = metrics;
if (horizontalOverlap > 0 && verticalGap <= HOVER_SURFACE_MAX_AXIS_GAP) return true;
if (verticalOverlap > 0 && horizontalGap <= HOVER_SURFACE_MAX_AXIS_GAP) return true;
return (
horizontalGap <= HOVER_SURFACE_MAX_DIAGONAL_GAP && verticalGap <= HOVER_SURFACE_MAX_DIAGONAL_GAP
);
}
export function isLikelyHoverSurfaceOwner(trigger: Element, surface: Element): boolean {
const metrics = surfaceRelationMetrics(surface, trigger);
if (!metrics) return true;
const {
surfaceRect,
triggerRect,
horizontalOverlap,
verticalOverlap,
horizontalGap,
verticalGap,
} = metrics;
const opensVertically =
horizontalOverlap > 0 &&
verticalGap <= HOVER_SURFACE_MAX_AXIS_GAP &&
(triggerRect.bottom <= surfaceRect.top + 8 || surfaceRect.bottom <= triggerRect.top + 8);
const opensSideways =
verticalOverlap > 0 &&
horizontalGap <= HOVER_SURFACE_MAX_AXIS_GAP &&
(triggerRect.right <= surfaceRect.left + 8 || surfaceRect.right <= triggerRect.left + 8);
const opensFromCorner =
horizontalGap > 0 &&
verticalGap > 0 &&
horizontalGap <= HOVER_SURFACE_MAX_DIAGONAL_GAP &&
verticalGap <= HOVER_SURFACE_MAX_DIAGONAL_GAP;
return opensVertically || opensSideways || opensFromCorner;
}
export function decideHoverSurfaceRelation(
context: HoverSurfaceContext,
clickElement: Element,
): HoverSurfaceDecision {
const clickSurface = closestRecognizedHoverSurface(clickElement);
if (clickSurface && isNearTrigger(clickSurface, context.triggerElement)) {
return { related: true, reason: "click-inside-hover-surface", surface: clickSurface };
}
const floatingSurface = closestFloatingSurface(clickElement);
if (floatingSurface && isNearTrigger(floatingSurface, context.triggerElement)) {
return {
related: true,
reason: "click-inside-near-floating-surface",
surface: floatingSurface,
};
}
if (context.triggerElement.contains(clickElement) && clickElement !== context.triggerElement) {
return { related: true, reason: "click-inside-trigger-contained-surface" };
}
return { related: false, reason: "click-outside-hover-surface" };
}
+22 -3
View File
@@ -5,6 +5,7 @@ import { flushSync } from "react-dom";
import ReactDOM from "react-dom/client";
import { BorrowConfirmationOverlay } from "@/content/BorrowConfirmationOverlay";
import { ControlOverlay } from "@/content/ControlOverlay";
import { createCaptureSuppressController } from "@/content/capture-suppress";
import { HelpRequestOverlay } from "@/content/HelpRequestOverlay";
import overlayCss from "@/content/overlay.css?inline";
import { OverlayController, shouldShowAgentControlOverlay } from "@/content/overlay-controller";
@@ -14,6 +15,11 @@ import {
isRecordContentMessage,
type RecordCaptureController,
} from "@/content/record-capture";
import {
type CaptureSuppressAck,
type CaptureSuppressMessage,
isCaptureSuppressMessage,
} from "@/lib/capture-suppress-bridge";
import {
HELP_ACK,
HELP_FINISH,
@@ -39,6 +45,7 @@ import { sendInterrupt } from "@/lib/overlay-interrupt-client";
import {
RECORD_FINISH,
RECORD_QUERY,
type RecordQueryResponse,
type RecordStartAck,
type RecordStopAck,
} from "@/lib/record-bridge";
@@ -69,6 +76,8 @@ export default defineContentScript({
let hostLossReported = false;
let remountInProgress = false;
const captureSuppress = createCaptureSuppressController(() => overlayHost);
const ui = await createShadowRootUi(ctx, {
name: "browser-skill-overlay",
position: "inline",
@@ -80,6 +89,8 @@ export default defineContentScript({
overlayHost = shadowHost;
overlayContainer = container;
hostLossReported = false;
// A host rebuilt mid-capture must stay hidden until `end` arrives.
captureSuppress.onHostMounted(shadowHost);
const app = document.createElement("div");
app.className = "bsk-overlay-root";
container.append(app);
@@ -237,12 +248,17 @@ export default defineContentScript({
| BorrowCancelMessage
| HelpRequestMessage
| HelpCancelMessage
| CaptureSuppressMessage
| OverlayAgentOverlayResetMessage
| OverlayAgentStateMessage
| OverlayAutomationBypassMessage,
_sender: chrome.runtime.MessageSender,
sendResponse: (response: BorrowResponseMessage | HelpAckMessage) => void,
sendResponse: (response: BorrowResponseMessage | HelpAckMessage | CaptureSuppressAck) => void,
) => {
if (isCaptureSuppressMessage(message)) {
return captureSuppress.handleMessage(message, sendResponse);
}
if (isRecordContentMessage(message)) {
const needsAsync = handleRecordContentMessage(
message,
@@ -255,9 +271,10 @@ export default defineContentScript({
setCapture: (capture) => {
recordCapture = capture;
},
onStart: (requestId) => {
onStart: (requestId, startedAtMs) => {
overlays.setAgentRecordRequest({
id: requestId,
...(typeof startedAtMs === "number" ? { startedAtMs } : {}),
onFinish: () => {
void chrome.runtime.sendMessage({
type: RECORD_FINISH,
@@ -415,15 +432,17 @@ export default defineContentScript({
try {
const recordQuery = (await chrome.runtime.sendMessage({
type: RECORD_QUERY,
})) as { active?: boolean; requestId?: string } | undefined;
})) as RecordQueryResponse | undefined;
if (
recordQuery?.active &&
typeof recordQuery.requestId === "string" &&
overlays.snapshot().activeRecord === null
) {
const requestId = recordQuery.requestId;
const startedAtMs = recordQuery.startedAtMs;
overlays.setAgentRecordRequest({
id: requestId,
...(typeof startedAtMs === "number" ? { startedAtMs } : {}),
onFinish: () => {
void chrome.runtime.sendMessage({
type: RECORD_FINISH,
+9 -9
View File
@@ -1,14 +1,9 @@
import { useTranslation } from "@browser-skill/i18n/react";
import { Badge, Button, cn, Input, Label } from "@browser-skill/ui";
import {
RiApps2Line,
RiArrowLeftLine,
RiArrowRightSLine,
RiCheckLine,
RiFileCopyLine,
} from "@remixicon/react";
import { RiArrowLeftLine, RiArrowRightSLine, RiCheckLine, RiFileCopyLine } from "@remixicon/react";
import { type ChangeEvent, useEffect, useState } from "react";
import { PROTOCOL_VERSION } from "@/transport/handshake";
import functionIconUrl from "../../../assets/function.svg";
import { ConnectionStatusIndicator } from "./connection-status-indicator";
import { POPUP_FEATURES, type PopupView } from "./features";
import { type PopupStatusState, useConnectionState } from "./use-connection-state";
@@ -156,12 +151,17 @@ export function App() {
type="button"
variant="ghost"
size="icon"
className="size-7 shrink-0 rounded-md"
className="group size-8 shrink-0 rounded-md"
aria-label={t("popup.launcher.title")}
onClick={() => setView("features")}
data-slot="popup-launcher"
>
<RiApps2Line className="size-4" aria-hidden />
<img
src={functionIconUrl}
alt=""
className="size-6 opacity-35 transition-opacity group-hover:opacity-100 group-focus-visible:opacity-100"
data-slot="popup-launcher-icon"
/>
</Button>
)}
</header>
@@ -0,0 +1,77 @@
import { describe, expect, it, vi } from "vitest";
import {
CAPTURE_SUPPRESS,
type CaptureSuppressMessage,
withOverlaysHiddenForCapture,
} from "../capture-suppress-bridge";
function recordingSendToTab(events: string[]) {
return vi.fn(async (_tabId: number, message: CaptureSuppressMessage) => {
events.push(message.phase);
return { type: CAPTURE_SUPPRESS, ok: true };
});
}
describe("withOverlaysHiddenForCapture", () => {
it("sends begin → fn → end and returns fn's result", async () => {
const events: string[] = [];
const sendToTab = recordingSendToTab(events);
const result = await withOverlaysHiddenForCapture(
7,
async () => {
events.push("capture");
return "shot";
},
sendToTab,
);
expect(result).toBe("shot");
expect(events).toEqual(["begin", "capture", "end"]);
expect(sendToTab).toHaveBeenCalledWith(7, { type: CAPTURE_SUPPRESS, phase: "begin" });
expect(sendToTab).toHaveBeenCalledWith(7, { type: CAPTURE_SUPPRESS, phase: "end" });
});
it("still sends end when fn throws, then rethrows", async () => {
const events: string[] = [];
const sendToTab = recordingSendToTab(events);
await expect(
withOverlaysHiddenForCapture(
7,
async () => {
events.push("capture");
throw new Error("capture exploded");
},
sendToTab,
),
).rejects.toThrow("capture exploded");
expect(events).toEqual(["begin", "capture", "end"]);
});
it("captures directly when the tab has no content script", async () => {
const sendToTab = vi.fn(async () => {
throw new Error("Could not establish connection. Receiving end does not exist.");
});
const fn = vi.fn(async () => "shot");
const result = await withOverlaysHiddenForCapture(7, fn, sendToTab);
expect(result).toBe("shot");
expect(fn).toHaveBeenCalledTimes(1);
// begin failed → no matching end may be sent.
expect(sendToTab).toHaveBeenCalledTimes(1);
});
it("swallows end failures so the capture result stands", async () => {
const sendToTab = vi.fn(async (_tabId: number, message: CaptureSuppressMessage) => {
if (message.phase === "end") throw new Error("tab navigated away");
return { type: CAPTURE_SUPPRESS, ok: true };
});
const result = await withOverlaysHiddenForCapture(7, async () => "shot", sendToTab);
expect(result).toBe("shot");
expect(sendToTab).toHaveBeenCalledTimes(2);
});
});
@@ -1,125 +0,0 @@
import { describe, expect, it } from "vitest";
import { evaluateHoverTrigger } from "../hover-trigger-policy";
describe("evaluateHoverTrigger", () => {
it("accepts explicit popup triggers", () => {
const decision = evaluateHoverTrigger({
tag: "button",
role: "button",
label: "Open user navigation menu",
attrs: { "aria-haspopup": "menu", "aria-expanded": "false" },
rect: { x: 900, y: 8, w: 32, h: 32 },
cursor: "pointer",
pointerEvents: "auto",
});
expect(decision.eligible).toBe(true);
expect(decision.reasons).toEqual(
expect.arrayContaining(["aria-haspopup", "collapsed", "popup-signal"]),
);
});
it("accepts compact topbar image buttons without popup attributes", () => {
const decision = evaluateHoverTrigger({
tag: "button",
role: "button",
label: "image",
attrs: {},
rect: { x: 900, y: 8, w: 32, h: 32 },
pointerEvents: "auto",
hasGraphicDescendant: true,
});
expect(decision.eligible).toBe(true);
expect(decision.reasons).toEqual(expect.arrayContaining(["icon-only", "topbar", "compact"]));
});
it("accepts compact topbar image links without popup attributes", () => {
const decision = evaluateHoverTrigger({
tag: "a",
role: "link",
label: "image",
attrs: {},
rect: { x: 900, y: 8, w: 32, h: 32 },
pointerEvents: "auto",
hasGraphicDescendant: true,
});
expect(decision.eligible).toBe(true);
expect(decision.reasons).toEqual(expect.arrayContaining(["icon-only", "topbar", "compact"]));
});
it("rejects topbar menu links without surface trigger evidence", () => {
const decision = evaluateHoverTrigger({
tag: "a",
role: "link",
label: "My profile",
attrs: {},
rect: { x: 820, y: 72, w: 96, h: 32 },
cursor: "pointer",
pointerEvents: "auto",
});
expect(decision.eligible).toBe(false);
});
it("rejects text navigation items even when their attributes contain popup words", () => {
const decision = evaluateHoverTrigger({
tag: "a",
role: "link",
label: "My profile",
attrs: { class: "dropdown-item profile-link" },
rect: { x: 820, y: 72, w: 96, h: 32 },
cursor: "pointer",
pointerEvents: "auto",
});
expect(decision.eligible).toBe(false);
});
it("rejects labelled dropdown list items as hover triggers", () => {
const decision = evaluateHoverTrigger({
tag: "li",
label: "文档C+D",
attrs: { class: "dropdown-item", tabindex: "0" },
rect: { x: 820, y: 48, w: 160, h: 32 },
cursor: "pointer",
pointerEvents: "auto",
});
expect(decision.eligible).toBe(false);
});
it("rejects ordinary controls without hover popup signals", () => {
const decision = evaluateHoverTrigger({
tag: "button",
role: "button",
label: "Search",
attrs: {},
rect: { x: 100, y: 240, w: 120, h: 36 },
pointerEvents: "auto",
});
expect(decision.eligible).toBe(false);
});
it("rejects unsafe or non-visible triggers", () => {
expect(
evaluateHoverTrigger({
tag: "input",
attrs: {},
rect: { x: 10, y: 10, w: 80, h: 24 },
pointerEvents: "auto",
}).eligible,
).toBe(false);
expect(
evaluateHoverTrigger({
tag: "button",
role: "button",
attrs: { hidden: "" },
rect: { x: 10, y: 10, w: 80, h: 24 },
pointerEvents: "auto",
}).eligible,
).toBe(false);
});
});
@@ -130,30 +130,6 @@ describe("reduceTraceSteps", () => {
});
});
it("keeps hover steps before menu clicks", () => {
const { steps } = reduceTraceSteps(
[
{
op: "hover",
target: { tag: "span", role: "button", name: "Account" },
page_url: "https://example.com/app",
},
{
op: "click",
target: { tag: "a", role: "link", name: "Profile" },
page_url: "https://example.com/app",
},
],
"https://example.com/app",
);
expect(steps.map((s) => s.op)).toEqual(["hover", "click"]);
expect(steps[0]).toMatchObject({
op: "hover",
page: "p1",
target: { name: "Account" },
});
});
it("resolveTraceStartUrl prefers explicit start URL", () => {
expect(
resolveTraceStartUrl(
@@ -0,0 +1,71 @@
/**
* Wire protocol and background-side helper for hiding the extension's
* in-page overlay while a screenshot is captured, so captured frames only
* contain the page itself.
*
* The background wraps every capture in `withOverlaysHiddenForCapture`,
* which sends `begin` (the content script hides the overlay host and only
* acks once the compositor has produced an overlay-free frame) and always
* follows up with `end`, even when the capture throws. Tabs without the
* content script (chrome://, the Web Store, ...) host no overlay, so a
* failed `begin` falls through to capturing directly.
*/
export const CAPTURE_SUPPRESS = "bsk/capture-suppress";
export type CaptureSuppressPhase = "begin" | "end";
export interface CaptureSuppressMessage {
type: typeof CAPTURE_SUPPRESS;
phase: CaptureSuppressPhase;
}
export interface CaptureSuppressAck {
type: typeof CAPTURE_SUPPRESS;
ok: true;
}
export function isCaptureSuppressMessage(msg: unknown): msg is CaptureSuppressMessage {
if (typeof msg !== "object" || msg === null) return false;
const m = msg as Record<string, unknown>;
return m.type === CAPTURE_SUPPRESS && (m.phase === "begin" || m.phase === "end");
}
/** Minimal `chrome.tabs.sendMessage` surface so tests can inject a fake. */
export type CaptureSuppressSendToTab = (
tabId: number,
message: CaptureSuppressMessage,
) => Promise<unknown>;
const defaultSendToTab: CaptureSuppressSendToTab = (tabId, message) =>
chrome.tabs.sendMessage(tabId, message);
/**
* Run `fn` with the target tab's overlay hidden. `end` is sent from a
* `finally` so the overlay always reappears, and a missing content script
* (or any other `begin` failure) simply skips suppression — there is no
* overlay to hide in that tab.
*/
export async function withOverlaysHiddenForCapture<T>(
tabId: number,
fn: () => Promise<T>,
sendToTab: CaptureSuppressSendToTab = defaultSendToTab,
): Promise<T> {
let began = false;
try {
await sendToTab(tabId, { type: CAPTURE_SUPPRESS, phase: "begin" });
began = true;
} catch {
// No content script in the target tab → no overlay can leak into the
// capture; proceed without suppression.
}
try {
return await fn();
} finally {
if (began) {
await sendToTab(tabId, { type: CAPTURE_SUPPRESS, phase: "end" }).catch((err) => {
console.debug("[bsk capture] overlay restore failed", err);
});
}
}
}
-14
View File
@@ -228,20 +228,6 @@ export function resolveClickableElement(target: Element): Element | null {
return null;
}
export function resolveHoverElement(target: Element): Element | null {
const clickable = resolveClickableElement(target);
if (clickable) return clickable;
let node: Element | null = target;
let depth = 0;
while (node && node !== document.body && node !== document.documentElement && depth < 8) {
if (looksClickable(node)) return node;
node = node.parentElement;
depth += 1;
}
return null;
}
/**
* Teachable clicks: the LLM must get a label it can later find via snapshot
* (visible name), or at least a form `name_attr` for checkbox/radio.
@@ -1,178 +0,0 @@
export interface HoverTriggerRect {
x: number;
y: number;
w: number;
h: number;
}
export interface HoverTriggerSignals {
tag: string;
role?: string;
label?: string;
attrs?: Record<string, string | undefined>;
rect?: HoverTriggerRect | null;
cursor?: string;
pointerEvents?: string;
hasGraphicDescendant?: boolean;
cssHoverMatch?: boolean;
}
export interface HoverTriggerDecision {
eligible: boolean;
score: number;
reasons: string[];
}
const HOVER_TRIGGER_MIN_SCORE = 45;
const HOVER_POPUP_SIGNAL_RE =
/(menu|dropdown|popover|popup|avatar|profile|account|user|more|ellipsis|caret)/;
const HOVER_TEXT_ITEM_RE = /(^|[\s_-])(dropdown-item|menu-item|context-menu-item|option)($|[\s_-])/;
function attr(signals: HoverTriggerSignals, name: string): string | undefined {
return signals.attrs?.[name.toLowerCase()];
}
export function isUnsafeHoverTrigger(signals: HoverTriggerSignals): boolean {
const tag = signals.tag.toLowerCase();
if (["input", "textarea", "select", "option"].includes(tag)) return true;
if ((attr(signals, "contenteditable") ?? "").toLowerCase() === "true") return true;
if (attr(signals, "disabled") !== undefined || attr(signals, "inert") !== undefined) return true;
return (attr(signals, "aria-disabled") ?? "").toLowerCase() === "true";
}
export function hasHoverPopupSignal(signals: HoverTriggerSignals): boolean {
const attrs = signals.attrs ?? {};
const haystack = [
attrs.id,
attrs.class,
attrs["data-testid"],
attrs["data-test"],
attrs["data-cy"],
attrs["aria-label"],
attrs["aria-haspopup"],
attrs["aria-controls"],
attrs.title,
signals.role === "button" ? signals.label : undefined,
]
.filter(Boolean)
.join(" ")
.toLowerCase();
return HOVER_POPUP_SIGNAL_RE.test(haystack);
}
export function hasStrongHoverExpansionSignal(signals: HoverTriggerSignals): boolean {
return (
signals.cssHoverMatch === true ||
attr(signals, "aria-haspopup") !== undefined ||
(attr(signals, "aria-expanded") ?? "").toLowerCase() === "false"
);
}
export function hasDirectHoverInteractiveSignal(signals: HoverTriggerSignals): boolean {
const tag = signals.tag.toLowerCase();
const role = (signals.role ?? "").toLowerCase();
return (
["button", "a", "summary"].includes(tag) ||
["button", "link", "menuitem", "tab", "combobox"].includes(role) ||
attr(signals, "tabindex") !== undefined ||
signals.cursor === "pointer" ||
attr(signals, "onclick") !== undefined ||
attr(signals, "onmouseenter") !== undefined ||
attr(signals, "onmouseover") !== undefined ||
attr(signals, "aria-haspopup") !== undefined ||
attr(signals, "aria-expanded") === "false"
);
}
export function evaluateHoverTrigger(signals: HoverTriggerSignals): HoverTriggerDecision {
const rect = signals.rect;
if (
!rect ||
rect.w <= 0 ||
rect.h <= 0 ||
signals.pointerEvents === "none" ||
attr(signals, "hidden") !== undefined ||
attr(signals, "inert") !== undefined ||
(attr(signals, "aria-hidden") ?? "").toLowerCase() === "true" ||
isUnsafeHoverTrigger(signals)
) {
return { eligible: false, score: 0, reasons: [] };
}
const area = rect.w * rect.h;
if (area <= 0 || area > 160_000) {
return { eligible: false, score: 0, reasons: [] };
}
const reasons: string[] = [];
let score = 0;
const tag = signals.tag.toLowerCase();
const role = (signals.role ?? "").toLowerCase();
const label = signals.label?.trim();
const graphic = signals.hasGraphicDescendant === true;
const directInteractive = hasDirectHoverInteractiveSignal(signals);
const popupSignal = hasHoverPopupSignal(signals);
const isTextNavigationItem =
(tag === "a" ||
tag === "li" ||
role === "link" ||
role === "option" ||
role.startsWith("menuitem") ||
HOVER_TEXT_ITEM_RE.test(attr(signals, "class") ?? "")) &&
!graphic &&
!!label &&
label.toLowerCase() !== "image";
const compactTopbarIcon =
(role === "button" || role === "link" || tag === "button" || tag === "a") &&
graphic &&
rect.y < 120 &&
rect.w <= 96 &&
rect.h <= 96 &&
(!label || label.toLowerCase() === "image");
const surfaceTriggerEvidence =
(popupSignal && !isTextNavigationItem) ||
compactTopbarIcon ||
hasStrongHoverExpansionSignal(signals);
if (!directInteractive || !surfaceTriggerEvidence) {
return { eligible: false, score: 0, reasons: [] };
}
if (attr(signals, "aria-haspopup") !== undefined) {
score += 80;
reasons.push("aria-haspopup");
}
if ((attr(signals, "aria-expanded") ?? "").toLowerCase() === "false") {
score += 55;
reasons.push("collapsed");
}
if (popupSignal) {
score += 45;
reasons.push("popup-signal");
}
if (tag === "button" || role === "button") {
score += 30;
reasons.push("button");
}
if (signals.cursor === "pointer") {
score += 25;
reasons.push("pointer");
}
if (graphic && (!label || label.toLowerCase() === "image")) {
score += 40;
reasons.push("icon-only");
}
if (rect.y < 120) {
score += 25;
reasons.push("topbar");
}
if (rect.w <= 80 && rect.h <= 80) {
score += 20;
reasons.push("compact");
}
if (signals.cssHoverMatch) {
score += 40;
reasons.push("css-hover");
}
return { eligible: score >= HOVER_TRIGGER_MIN_SCORE, score, reasons };
}
+9 -1
View File
@@ -25,15 +25,23 @@ export interface RecordQueryMessage {
export interface RecordQueryResponse {
active: boolean;
requestId?: string;
/** Epoch ms when the recording began; see `RecordStartMessage.startedAtMs`. */
startedAtMs?: number;
}
export interface RecordStartMessage {
type: typeof RECORD_START;
requestId: string;
/**
* Epoch ms when the whole recording began, not when this tab was armed.
* The overlay timer must span the session, so it survives navigations and
* content-script remounts instead of restarting per page.
*/
startedAtMs?: number;
}
export interface RecordStepPayload {
op: "click" | "hover" | "fill" | "press" | "select" | "navigate";
op: "click" | "fill" | "press" | "select" | "navigate";
target?: TargetDescriptor;
value?: string;
key?: string;
@@ -21,14 +21,6 @@ function toDraftStep(payload: RecordStepPayload): DraftTraceStep | null {
...(pageUrl ? { page_url: pageUrl } : {}),
}
: null;
case "hover":
return payload.target
? {
op: "hover",
target: payload.target,
...(pageUrl ? { page_url: pageUrl } : {}),
}
: null;
case "fill":
return payload.target
? {
-7
View File
@@ -139,13 +139,6 @@ function toV2Step(
},
effectForNavigation(step.navigated_to, urlToId),
);
case "hover":
return {
op: "hover",
id,
page,
target: step.target,
};
case "fill":
return {
op: "fill",
@@ -9,8 +9,6 @@ import type {
} from "@/transport/types";
import { ToolDispatcher } from "../dispatcher";
type TestDispatcherCdp = NonNullable<ConstructorParameters<typeof ToolDispatcher>[0]["cdp"]>;
function fakeTransport() {
const handlers = new Set<FrameHandler>();
const stateHandlers = new Set<ConnectionStateHandler>();
@@ -131,7 +129,7 @@ describe("ToolDispatcher", () => {
setUserAgentOverride: vi.fn(async () => {}),
setTouchEmulationEnabled: vi.fn(async () => {}),
};
const dispatcher = new ToolDispatcher({ transport, sessions, cdp: cdp as TestDispatcherCdp });
const dispatcher = new ToolDispatcher({ transport, sessions, cdp });
dispatcher.start();
deliver(makeRequest("tool.console", { session_id: "aa11" }));
@@ -174,7 +172,7 @@ describe("ToolDispatcher", () => {
setUserAgentOverride: vi.fn(async () => {}),
setTouchEmulationEnabled: vi.fn(async () => {}),
};
const dispatcher = new ToolDispatcher({ transport, sessions, cdp: cdp as TestDispatcherCdp });
const dispatcher = new ToolDispatcher({ transport, sessions, cdp });
dispatcher.start();
deliver(makeRequest("tool.session_stop", { session_id: "aa11" }));
@@ -313,277 +311,6 @@ describe("ToolDispatcher", () => {
expect(onBrowserControlResumed).toHaveBeenCalledWith("aa11");
});
it("reasserts remembered hover before follow-up work and releases only after actions", async () => {
vi.stubGlobal("chrome", {
tabs: {
sendMessage: vi.fn(async () => undefined),
},
});
const { transport } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
},
});
const order: string[] = [];
const cdp = {
send: vi.fn(async <T>() => {
order.push("hover");
return {} as T;
}),
detachSession: vi.fn(async () => {}),
ensureNetworkCapture: vi.fn(async () => {}),
networkEntriesSince: vi.fn(() => ({
tab_id: 7,
entries: [],
next_since: 0,
truncated: false,
})),
setDeviceMetricsOverride: vi.fn(async () => {}),
clearDeviceMetricsOverride: vi.fn(async () => {}),
setUserAgentOverride: vi.fn(async () => {}),
setTouchEmulationEnabled: vi.fn(async () => {}),
};
const dispatcher = new ToolDispatcher({ transport, sessions, cdp: cdp as TestDispatcherCdp });
(
dispatcher as unknown as { rememberHover: (sessionId: string, result: unknown) => unknown }
).rememberHover("aa11", {
tab_id: 7,
x: 10,
y: 20,
});
const helpers = dispatcher as unknown as {
withHoverReassert: <T>(
params: { session_id: string; tab_id?: number },
work: () => Promise<T>,
options?: { releaseAfter?: boolean },
) => Promise<T>;
setHoverBypass: (sessionId: string, tabId: number, enabled: boolean) => Promise<void>;
};
await helpers.withHoverReassert({ session_id: "aa11", tab_id: 7 }, async () => {
order.push("observe");
return {};
});
expect(order).toEqual(["hover", "observe"]);
expect(chrome.tabs.sendMessage).not.toHaveBeenCalledWith(
7,
expect.objectContaining({ enabled: false }),
);
await helpers.setHoverBypass("aa11", 7, true);
await helpers.withHoverReassert(
{ session_id: "aa11", tab_id: 7 },
async () => {
order.push("click");
return {};
},
{ releaseAfter: true },
);
expect(cdp.send).toHaveBeenLastCalledWith(7, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: 10,
y: 20,
});
expect(chrome.tabs.sendMessage).toHaveBeenCalledWith(
7,
expect.objectContaining({ enabled: false }),
);
});
it("does not reassert or disable hover latches from another session", async () => {
vi.stubGlobal("chrome", {
tabs: {
sendMessage: vi.fn(async () => undefined),
},
});
const { transport } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
},
});
const cdp = {
send: vi.fn(async <T>() => ({}) as T),
detachSession: vi.fn(async () => {}),
ensureNetworkCapture: vi.fn(async () => {}),
networkEntriesSince: vi.fn(() => ({
tab_id: 7,
entries: [],
next_since: 0,
truncated: false,
})),
setDeviceMetricsOverride: vi.fn(async () => {}),
clearDeviceMetricsOverride: vi.fn(async () => {}),
setUserAgentOverride: vi.fn(async () => {}),
setTouchEmulationEnabled: vi.fn(async () => {}),
};
const dispatcher = new ToolDispatcher({ transport, sessions, cdp: cdp as TestDispatcherCdp });
const helpers = dispatcher as unknown as {
rememberHover: (sessionId: string, result: unknown) => unknown;
setHoverBypass: (sessionId: string, tabId: number, enabled: boolean) => Promise<void>;
withHoverReassert: <T>(
params: { session_id: string; tab_id?: number },
work: () => Promise<T>,
options?: { releaseAfter?: boolean },
) => Promise<T>;
releaseHoverLatch: (sessionId?: string, tabId?: number) => Promise<void>;
};
helpers.rememberHover("aa11", { tab_id: 7, x: 10, y: 20 });
await helpers.setHoverBypass("aa11", 7, true);
await helpers.withHoverReassert({ session_id: "bb22", tab_id: 7 }, async () => ({}), {
releaseAfter: true,
});
expect(cdp.send).not.toHaveBeenCalled();
expect(chrome.tabs.sendMessage).not.toHaveBeenCalledWith(
7,
expect.objectContaining({ enabled: false }),
);
await helpers.releaseHoverLatch("aa11", 7);
expect(chrome.tabs.sendMessage).toHaveBeenCalledWith(
7,
expect.objectContaining({ enabled: false }),
);
});
it("transfers hover overlay bypass ownership between sessions", async () => {
vi.stubGlobal("chrome", {
tabs: {
sendMessage: vi.fn(async () => undefined),
},
});
const { transport } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
},
});
const dispatcher = new ToolDispatcher({ transport, sessions });
const helpers = dispatcher as unknown as {
setHoverBypass: (sessionId: string, tabId: number, enabled: boolean) => Promise<void>;
};
await helpers.setHoverBypass("aa11", 7, true);
await helpers.setHoverBypass("bb22", 7, true);
await helpers.setHoverBypass("aa11", 7, false);
expect(chrome.tabs.sendMessage).not.toHaveBeenCalledWith(
7,
expect.objectContaining({ enabled: false }),
);
await helpers.setHoverBypass("bb22", 7, false);
expect(chrome.tabs.sendMessage).toHaveBeenCalledWith(
7,
expect.objectContaining({ enabled: false }),
);
expect(chrome.tabs.sendMessage).toHaveBeenCalledTimes(2);
});
it("limits default-target hover reassertion to the active tab", async () => {
vi.stubGlobal("chrome", {
tabs: {
query: vi.fn(async () => [{ id: 9, windowId: 4242, active: true }]),
get: vi.fn(async (tabId: number) => ({ id: tabId, windowId: 4242, active: tabId === 9 })),
sendMessage: vi.fn(async () => undefined),
},
});
const { transport } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
},
});
await sessions.start("aa11");
const cdp = {
send: vi.fn(async <T>() => ({}) as T),
detachSession: vi.fn(async () => {}),
ensureNetworkCapture: vi.fn(async () => {}),
networkEntriesSince: vi.fn(() => ({
tab_id: 9,
entries: [],
next_since: 0,
truncated: false,
})),
setDeviceMetricsOverride: vi.fn(async () => {}),
clearDeviceMetricsOverride: vi.fn(async () => {}),
setUserAgentOverride: vi.fn(async () => {}),
setTouchEmulationEnabled: vi.fn(async () => {}),
};
const dispatcher = new ToolDispatcher({ transport, sessions, cdp: cdp as TestDispatcherCdp });
const helpers = dispatcher as unknown as {
rememberHover: (sessionId: string, result: unknown) => unknown;
withHoverReassert: <T>(
params: { session_id: string; tab_id?: number },
work: () => Promise<T>,
) => Promise<T>;
};
helpers.rememberHover("aa11", { tab_id: 7, x: 10, y: 20 });
helpers.rememberHover("aa11", { tab_id: 9, x: 30, y: 40 });
await helpers.withHoverReassert({ session_id: "aa11" }, async () => ({}));
expect(cdp.send).toHaveBeenCalledTimes(1);
expect(cdp.send).toHaveBeenCalledWith(9, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: 30,
y: 40,
});
});
it("releases the current session hover latch before navigation-style work", async () => {
vi.stubGlobal("chrome", {
tabs: {
sendMessage: vi.fn(async () => undefined),
},
});
const { transport } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
},
});
const dispatcher = new ToolDispatcher({ transport, sessions });
const helpers = dispatcher as unknown as {
rememberHover: (sessionId: string, result: unknown) => unknown;
setHoverBypass: (sessionId: string, tabId: number, enabled: boolean) => Promise<void>;
withHoverReleaseForRequest: <T>(
params: { session_id: string; tab_id?: number },
work: () => Promise<T>,
) => Promise<T>;
withHoverReassert: <T>(
params: { session_id: string; tab_id?: number },
work: () => Promise<T>,
) => Promise<T>;
};
helpers.rememberHover("aa11", { tab_id: 7, x: 10, y: 20 });
await helpers.setHoverBypass("aa11", 7, true);
await helpers.withHoverReleaseForRequest({ session_id: "aa11", tab_id: 7 }, async () => ({}));
await helpers.withHoverReassert({ session_id: "aa11", tab_id: 7 }, async () => ({}));
expect(chrome.tabs.sendMessage).toHaveBeenCalledWith(
7,
expect.objectContaining({ enabled: false }),
);
});
it("disconnects the transport when send() fails so keepalive can rebuild", async () => {
const { transport, deliver } = fakeTransport();
const sessions = new SessionManager({
@@ -4,7 +4,6 @@ import type { CdpRunner } from "@/tools/shared";
import {
handleClick,
handleFill,
handleHover,
handlePress,
handleSelect,
modifiersBitfield,
@@ -436,50 +435,6 @@ describe("handleClick", () => {
});
});
describe("handleHover", () => {
it("keeps overlay bypass enabled after a successful hover when requested", async () => {
const bypassOverlay = vi.fn().mockResolvedValue(undefined);
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
ctx.refStore.set("e3", 1234, { tabId: 4 });
const fake = makeFakeCdp({
"DOM.scrollIntoViewIfNeeded": () => ({}),
"DOM.getContentQuads": () => ({ quads: [[10, 20, 110, 20, 110, 60, 10, 60]] }),
"Runtime.evaluate": (params: unknown) => {
const expr = String((params as { expression?: string })?.expression ?? "");
if (expr.includes("overlayHostPresent") && !expr.includes("hitIndex")) {
return { result: { value: { overlayHostPresent: true, overlayHostConnected: true } } };
}
if (expr.includes("hitIndex")) {
return {
result: {
value: { overlayHostPresent: true, overlayHostConnected: true, hitIndex: 0 },
},
};
}
throw new Error(`unexpected Runtime.evaluate: ${expr.slice(0, 80)}`);
},
"Input.dispatchMouseEvent": () => ({}),
});
const res = await handleHover(
sm,
{ session_id: "aa11", ref: "@e3", settle_ms: 0 },
{
cdp: fake.cdp,
tabsApi: fake.tabsApi,
bypassOverlay,
keepOverlayBypassAfterHover: true,
},
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res).toMatchObject({ tab_id: 4, used_ref: "e3", x: 60, y: 40 });
expect(bypassOverlay).toHaveBeenCalledTimes(1);
expect(bypassOverlay).toHaveBeenCalledWith(4, true);
});
});
describe("handleFill", () => {
it("returns not_found for unknown ref", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
@@ -1,5 +1,6 @@
import { renderVom } from "@browser-skill/vom";
import { describe, expect, it, vi } from "vitest";
import { CAPTURE_SUPPRESS, type CaptureSuppressMessage } from "@/lib/capture-suppress-bridge";
import { OVERLAY_HOST_MARKER_ATTR, OVERLAY_HOST_NAME } from "@/lib/overlay-bridge";
import { SessionManager } from "@/session-manager/manager";
import type { CdpRunner } from "@/tools/shared";
@@ -345,6 +346,98 @@ describe("handleScreenshot", () => {
});
});
describe("handleScreenshot overlay suppression", () => {
/**
* Fake background→content bridge that mirrors the content script's
* contract: `begin` hides the overlay, `end` restores it. Records the
* phase order so tests can assert begin → capture → end.
*/
function fakeSuppressBridge(events: string[]) {
let overlayHidden = false;
const sendToTab = vi.fn(async (_tabId: number, message: CaptureSuppressMessage) => {
events.push(message.phase);
overlayHidden = message.phase === "begin";
return { type: CAPTURE_SUPPRESS, ok: true };
});
return { sendToTab, isHidden: () => overlayHidden };
}
it("hides the overlay around a visible-tab capture", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
const events: string[] = [];
const bridge = fakeSuppressBridge(events);
const capture = vi.fn(async (_w: number) => {
events.push("capture");
// The overlay must already be hidden when the capture runs.
expect(bridge.isHidden()).toBe(true);
return `data:image/png;base64,${TINY_PNG}`;
});
const res = await handleScreenshot(
sm,
{ session_id: "aa11" },
{ ...makeScreenshotDeps({ captureVisibleTab: capture }), sendToTab: bridge.sendToTab },
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.image_base64).toBe(TINY_PNG);
expect(events).toEqual(["begin", "capture", "end"]);
expect(bridge.isHidden()).toBe(false);
});
it("hides the overlay around a CDP element capture", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
ctx.refStore.set("e5", 999, { tabId: 7 });
const events: string[] = [];
const bridge = fakeSuppressBridge(events);
const { cdp } = makeFakeCdp({
"DOM.scrollIntoViewIfNeeded": () => ({}),
"DOM.getContentQuads": () => ({ quads: [[10, 20, 110, 20, 110, 60, 10, 60]] }),
"Page.captureScreenshot": () => {
events.push("capture");
expect(bridge.isHidden()).toBe(true);
return { data: TINY_PNG };
},
});
const res = await handleScreenshot(
sm,
{ session_id: "aa11", ref: "@e5", tab_id: 7 },
{
...makeScreenshotDeps({
cdp,
get: vi.fn(async () => ({ id: 7, windowId: 100, active: false }) as chrome.tabs.Tab),
query: vi.fn(),
captureVisibleTab: vi.fn(),
}),
sendToTab: bridge.sendToTab,
},
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.image_base64).toBe(TINY_PNG);
expect(events).toEqual(["begin", "capture", "end"]);
expect(bridge.isHidden()).toBe(false);
});
it("still captures when the tab has no content script to ack", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
const sendToTab = vi.fn(async () => {
throw new Error("Could not establish connection. Receiving end does not exist.");
});
const capture = vi.fn(async (_w: number) => `data:image/png;base64,${TINY_PNG}`);
const res = await handleScreenshot(
sm,
{ session_id: "aa11" },
{ ...makeScreenshotDeps({ captureVisibleTab: capture }), sendToTab },
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.image_base64).toBe(TINY_PNG);
expect(capture).toHaveBeenCalledTimes(1);
// The failed begin must not be followed by an end.
expect(sendToTab).toHaveBeenCalledTimes(1);
});
});
// ---------------------------------------------------------------------------
// buildVomScene
// ---------------------------------------------------------------------------
@@ -1430,7 +1523,7 @@ describe("buildVomScene", () => {
expect(renderVom(scene).text).toContain("[§ active: Reviews (12)]");
});
it("maps hover surface probes to matching backend node ids", () => {
it("maps hover surface probes to matching node labels", () => {
const axNodes: CdpAxNode[] = [
{
nodeId: "1",
@@ -1451,7 +1544,7 @@ describe("buildVomScene", () => {
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
surfaceProbes: [
{ triggerBackendNodeId: 20, triggerAction: "hover", subItems: ["Shoes", "Bags"] },
{ triggerLabel: "Products", triggerAction: "hover", subItems: ["Shoes", "Bags"] },
],
nodes: [
{
@@ -1480,260 +1573,7 @@ describe("buildVomScene", () => {
expect(scene.surfaces).toEqual([
{ triggerId: 20, triggerAction: "hover", subItems: ["Shoes", "Bags"] },
]);
expect(renderVom(scene).text).toContain('@e1 button "Products" [hover first: Shoes | Bags]');
});
it("maps hover surface probes to rendered descendants in the same trigger subtree", () => {
const axNodes: CdpAxNode[] = [
{
nodeId: "1",
role: { type: "role", value: "RootWebArea" },
backendDOMNodeId: 10,
childIds: ["2"],
},
{
nodeId: "2",
parentId: "1",
role: { type: "role", value: "button" },
name: { type: "computedString", value: "image" },
backendDOMNodeId: 21,
},
];
const scene = buildVomScene(axNodes, {
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
surfaceProbes: [
{ triggerBackendNodeId: 20, triggerAction: "hover", subItems: ["My profile"] },
],
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 20,
parentBackendNodeId: 10,
tag: "div",
attrs: { class: "tg-avatar" },
rect: { x: 900, y: 10, w: 30, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 21,
parentBackendNodeId: 20,
tag: "div",
attrs: { class: "tg-avatar__inner" },
rect: { x: 902, y: 12, w: 26, h: 26 },
paintOrder: 2,
position: "static",
pointerEvents: "auto",
},
],
});
expect(renderVom(scene).text).toContain('@e1 button "image" [hover first: My profile]');
});
it("deduplicates hover surface probes by original trigger backend id", () => {
const axNodes: CdpAxNode[] = [
{
nodeId: "1",
role: { type: "role", value: "RootWebArea" },
backendDOMNodeId: 10,
childIds: ["2"],
},
{
nodeId: "2",
parentId: "1",
role: { type: "role", value: "button" },
name: { type: "computedString", value: "image" },
backendDOMNodeId: 21,
},
];
const scene = buildVomScene(axNodes, {
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
surfaceProbes: [
{ triggerBackendNodeId: 20, triggerAction: "hover", subItems: ["My profile"] },
{ triggerBackendNodeId: 20, triggerAction: "hover", subItems: ["Sign out"] },
{ triggerBackendNodeId: 21, triggerAction: "hover", subItems: ["Settings"] },
],
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 20,
parentBackendNodeId: 10,
tag: "div",
attrs: { class: "tg-avatar" },
rect: { x: 900, y: 10, w: 30, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 21,
parentBackendNodeId: 20,
tag: "div",
attrs: { class: "tg-avatar__inner" },
rect: { x: 902, y: 12, w: 26, h: 26 },
paintOrder: 2,
position: "static",
pointerEvents: "auto",
},
],
});
expect(scene.surfaces).toEqual([
{ triggerId: 21, triggerAction: "hover", subItems: ["My profile"] },
]);
});
it("maps hover surface probes to DOM-recovered custom controls", () => {
const axNodes: CdpAxNode[] = [
{
nodeId: "1",
role: { type: "role", value: "RootWebArea" },
backendDOMNodeId: 10,
childIds: ["2"],
},
{
nodeId: "2",
parentId: "1",
role: { type: "role", value: "img" },
name: { type: "computedString", value: "image" },
backendDOMNodeId: 21,
},
];
const scene = buildVomScene(axNodes, {
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
surfaceProbes: [
{
triggerBackendNodeId: 20,
triggerPoint: { x: 915, y: 25 },
triggerAction: "hover",
subItems: ["My profile", "Sign out"],
},
],
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
cursor: "auto",
},
{
backendNodeId: 20,
parentBackendNodeId: 10,
tag: "div",
attrs: { class: "tg-avatar" },
rect: { x: 900, y: 10, w: 30, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
cursor: "pointer",
},
{
backendNodeId: 21,
parentBackendNodeId: 20,
tag: "div",
attrs: { class: "tg-avatar__inner" },
rect: { x: 902, y: 12, w: 26, h: 26 },
paintOrder: 2,
position: "static",
pointerEvents: "auto",
cursor: "pointer",
},
],
});
expect(scene.surfaces).toEqual([
{ triggerId: 21, triggerAction: "hover", subItems: ["My profile", "Sign out"] },
]);
expect(renderVom(scene).text).toContain(
'@e1 button "image" [hover first: My profile | Sign out]',
);
});
it("does not attach hover probes to distant geometry matches", () => {
const axNodes: CdpAxNode[] = [
{
nodeId: "1",
role: { type: "role", value: "RootWebArea" },
backendDOMNodeId: 10,
childIds: ["2"],
},
{
nodeId: "2",
parentId: "1",
role: { type: "role", value: "link" },
name: { type: "computedString", value: "0" },
backendDOMNodeId: 42,
},
];
const scene = buildVomScene(axNodes, {
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
surfaceProbes: [
{
triggerBackendNodeId: 999,
triggerPoint: { x: 900, y: 20 },
triggerAction: "hover",
subItems: ["My profile"],
},
],
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 42,
parentBackendNodeId: 10,
tag: "a",
attrs: {},
rect: { x: 860, y: 120, w: 40, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
},
],
});
expect(scene.surfaces).toBeUndefined();
expect(renderVom(scene).text).not.toContain("[hover first:");
expect(renderVom(scene).text).toContain('@e1 button "Products" [hover: Shoes | Bags]');
});
it("enriches names and active scope signals from AX properties", () => {
@@ -2206,7 +2046,7 @@ describe("handleSnapshot", () => {
});
const res = await handleObserve(
sm,
{ session_id: "aa11", debug_surfaces: true },
{ session_id: "aa11" },
{
cdp: {
send: send as unknown as <T = unknown>(
@@ -2227,7 +2067,6 @@ describe("handleSnapshot", () => {
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.debug).toEqual({ surface_probes: [] });
expect(send).toHaveBeenCalledWith(
4,
"Runtime.evaluate",
+53 -234
View File
@@ -8,8 +8,6 @@ import type {
EvaluateParams,
FillParams,
GetHtmlParams,
HoverParams,
HoverResult,
NavigateBackParams,
NavigateForwardParams,
NavigateParams,
@@ -35,7 +33,7 @@ import { handleConsole } from "./console";
import { type EmulateCdpRunner, handleEmulate } from "./emulate";
import { handleEvaluate } from "./evaluate";
import { handleRequestHelp } from "./human-loop";
import { handleClick, handleFill, handleHover, handlePress, handleSelect } from "./interaction";
import { handleClick, handleFill, handlePress, handleSelect } from "./interaction";
import {
handleNavigate,
handleNavigateBack,
@@ -58,7 +56,7 @@ import {
type SessionStartParams,
type SessionStopParams,
} from "./session";
import { chromeTabsApi, lookupSession, resolveTargetTab } from "./shared";
import { chromeTabsApi } from "./shared";
import {
type BorrowConfirmationApprover,
handleTabBorrow,
@@ -83,18 +81,6 @@ type DispatcherCdpRunner = CdpRunner &
detachSession(sessionId: string): Promise<void>;
};
interface HoverLatch {
sessionId: string;
tabId: number;
x: number;
y: number;
}
interface HoverLatchScope {
session_id: string;
tab_id?: number;
}
export interface DispatcherDeps {
transport: Transport;
sessions: SessionManager;
@@ -140,8 +126,6 @@ export class ToolDispatcher {
private readonly approveBorrow?: BorrowConfirmationApprover;
private readonly helpNotificationCopy?: () => { title: string; body: string };
private subscription: { dispose(): void } | null = null;
private readonly hoverBypassTabs = new Map<number, string>();
private readonly hoverLatches = new Map<number, HoverLatch>();
/**
* Per-rpc-id `AbortController` registry. Populated inside
* [`dispatch`] before we await the tool handler and torn down in
@@ -287,20 +271,16 @@ export class ToolDispatcher {
switch (req.method) {
case "tool.session_start":
return handleSessionStart(this.sessions, req.params as SessionStartParams);
case "tool.session_stop": {
await this.releaseHoverLatch((req.params as SessionStopParams).session_id);
case "tool.session_stop":
return handleSessionStop(this.sessions, req.params as SessionStopParams, {
cdp: this.cdp,
});
}
case "tool.tab_list":
return handleTabList(this.sessions, req.params as TabListParams);
case "tool.tab_create":
return handleTabCreate(this.sessions, req.params as TabCreateParams);
case "tool.tab_close":
return this.withHoverReleaseForRequest(req.params as TabCloseParams, () =>
handleTabClose(this.sessions, req.params as TabCloseParams),
);
return handleTabClose(this.sessions, req.params as TabCloseParams);
case "tool.tab_select":
return handleTabSelect(this.sessions, req.params as TabSelectParams);
case "tool.tab_borrow":
@@ -339,31 +319,17 @@ export class ToolDispatcher {
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi } : undefined,
);
case "tool.snapshot":
return this.withHoverReassert(req.params as SnapshotParams, () =>
handleSnapshot(
this.sessions,
req.params as SnapshotParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsCaptureApi } : undefined,
),
return handleSnapshot(
this.sessions,
req.params as SnapshotParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsCaptureApi } : undefined,
);
case "tool.observe": {
const params = req.params as ObserveParams;
const hoverScope = await this.resolveHoverLatchScope(params);
return this.withHoverReassert(params, () =>
handleObserve(
this.sessions,
params,
this.cdp
? {
cdp: this.cdp,
tabsApi: chromeTabsCaptureApi,
conditionalSurfaceProbe: !this.hasHoverLatchForScope(hoverScope),
hoverProbeBypassOverlay: bypassOverlay,
}
: undefined,
),
case "tool.observe":
return handleObserve(
this.sessions,
req.params as ObserveParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsCaptureApi } : undefined,
);
}
case "tool.get_html":
return handleGetHtml(
this.sessions,
@@ -371,95 +337,68 @@ export class ToolDispatcher {
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsCaptureApi } : undefined,
);
case "tool.navigate":
return this.withHoverReleaseForRequest(req.params as NavigateParams, () =>
handleNavigate(
this.sessions,
req.params as NavigateParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
),
return handleNavigate(
this.sessions,
req.params as NavigateParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
);
case "tool.navigate_back":
return this.withHoverReleaseForRequest(req.params as NavigateBackParams, () =>
handleNavigateBack(
this.sessions,
req.params as NavigateBackParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
),
return handleNavigateBack(
this.sessions,
req.params as NavigateBackParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
);
case "tool.navigate_forward":
return this.withHoverReleaseForRequest(req.params as NavigateForwardParams, () =>
handleNavigateForward(
this.sessions,
req.params as NavigateForwardParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
),
return handleNavigateForward(
this.sessions,
req.params as NavigateForwardParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
);
case "tool.reload":
return this.withHoverReleaseForRequest(req.params as ReloadParams, () =>
handleReload(
this.sessions,
req.params as ReloadParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
),
return handleReload(
this.sessions,
req.params as ReloadParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
);
case "tool.click":
return this.withHoverReassert(
req.params as ClickParams,
() =>
handleClick(
this.sessions,
req.params as ClickParams,
this.cdp
? {
cdp: this.cdp,
tabsApi: chromeTabsApi,
signal,
bypassOverlay,
}
: undefined,
),
{ releaseAfter: true },
);
case "tool.hover": {
const result = await handleHover(
return handleClick(
this.sessions,
req.params as HoverParams,
req.params as ClickParams,
this.cdp
? {
cdp: this.cdp,
tabsApi: chromeTabsApi,
signal,
bypassOverlay: (tabId, enabled) =>
this.setHoverBypass((req.params as HoverParams).session_id, tabId, enabled),
keepOverlayBypassAfterHover: true,
bypassOverlay: async (tabId, enabled) => {
try {
await chrome.tabs.sendMessage(tabId, {
type: OVERLAY_AUTOMATION_BYPASS,
enabled,
});
} catch {
// Content script may be unavailable on restricted pages.
}
},
}
: undefined,
);
return this.rememberHover((req.params as HoverParams).session_id, result);
}
case "tool.fill":
return this.withHoverReleaseForRequest(req.params as FillParams, () =>
handleFill(
this.sessions,
req.params as FillParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
),
return handleFill(
this.sessions,
req.params as FillParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
);
case "tool.press":
return this.withHoverReleaseForRequest(req.params as PressParams, () =>
handlePress(
this.sessions,
req.params as PressParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
),
return handlePress(
this.sessions,
req.params as PressParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
);
case "tool.select":
return this.withHoverReleaseForRequest(req.params as SelectParams, () =>
handleSelect(
this.sessions,
req.params as SelectParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
),
return handleSelect(
this.sessions,
req.params as SelectParams,
this.cdp ? { cdp: this.cdp, tabsApi: chromeTabsApi, signal } : undefined,
);
case "tool.evaluate":
return handleEvaluate(
@@ -532,114 +471,6 @@ export class ToolDispatcher {
} satisfies RpcError;
}
}
private async setHoverBypass(sessionId: string, tabId: number, enabled: boolean): Promise<void> {
const owner = this.hoverBypassTabs.get(tabId);
if (enabled) {
if (owner === sessionId) return;
if (owner === undefined) await bypassOverlay(tabId, true);
this.hoverBypassTabs.set(tabId, sessionId);
} else {
if (owner !== sessionId) return;
await bypassOverlay(tabId, false);
this.hoverBypassTabs.delete(tabId);
}
}
private rememberHover(sessionId: string, result: HoverResult | RpcError): HoverResult | RpcError {
if (!isRpcError(result)) {
this.hoverLatches.set(result.tab_id, {
sessionId,
tabId: result.tab_id,
x: result.x,
y: result.y,
});
}
return result;
}
private hasHoverLatchForScope(scope: HoverLatchScope): boolean {
return this.hoverLatchesForRequest(scope).length > 0;
}
private async withHoverReassert<T>(
params: { session_id: string; tab_id?: number },
work: () => Promise<T>,
options: { releaseAfter?: boolean } = {},
): Promise<T> {
const scope = await this.resolveHoverLatchScope(params);
await this.reassertHover(scope);
try {
return await work();
} finally {
if (options.releaseAfter) {
await this.releaseHoverLatch(scope.session_id, scope.tab_id);
}
}
}
private async withHoverReleaseForRequest<T>(
params: { session_id: string; tab_id?: number },
work: () => Promise<T>,
): Promise<T> {
const scope = await this.resolveHoverLatchScope(params);
await this.releaseHoverLatch(scope.session_id, scope.tab_id);
return work();
}
private async resolveHoverLatchScope(params: {
session_id: string;
tab_id?: number;
}): Promise<HoverLatchScope> {
if (params.tab_id !== undefined) return params;
const ctx = lookupSession(this.sessions, params, "hover latch");
if (isRpcError(ctx)) return params;
const target = await resolveTargetTab(this.sessions, ctx, undefined, chromeTabsApi);
if (isRpcError(target)) return params;
return { session_id: params.session_id, tab_id: target.tabId };
}
private hoverLatchesForRequest(params: { session_id: string; tab_id?: number }): HoverLatch[] {
return [...this.hoverLatches.values()].filter((latch) => {
if (latch.sessionId !== params.session_id) return false;
return params.tab_id === undefined || latch.tabId === params.tab_id;
});
}
private async reassertHover(params: { session_id: string; tab_id?: number }): Promise<void> {
if (!this.cdp) return;
await Promise.all(
this.hoverLatchesForRequest(params).map((latch) =>
this.cdp!.send(latch.tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: latch.x,
y: latch.y,
}).catch((err) => {
console.debug("[bsk dispatcher] hover reassert failed", err);
this.hoverLatches.delete(latch.tabId);
}),
),
);
}
private async releaseHoverLatch(sessionId?: string, tabId?: number): Promise<void> {
const matchesScope = (entrySessionId: string, entryTabId: number): boolean => {
if (sessionId !== undefined && entrySessionId !== sessionId) return false;
return tabId === undefined || entryTabId === tabId;
};
const tabs = new Set<number>();
for (const [bypassTabId, bypassSessionId] of this.hoverBypassTabs) {
if (!matchesScope(bypassSessionId, bypassTabId)) continue;
tabs.add(bypassTabId);
this.hoverBypassTabs.delete(bypassTabId);
}
for (const latch of this.hoverLatches.values()) {
if (!matchesScope(latch.sessionId, latch.tabId)) continue;
tabs.add(latch.tabId);
this.hoverLatches.delete(latch.tabId);
}
await Promise.all([...tabs].map((tabId) => bypassOverlay(tabId, false)));
}
}
function isRpcError(v: unknown): v is RpcError {
@@ -666,7 +497,6 @@ function sessionIdForBrowserControlMethod(req: RequestFrame): string | null {
case "tool.navigate_forward":
case "tool.reload":
case "tool.click":
case "tool.hover":
case "tool.fill":
case "tool.press":
case "tool.select":
@@ -682,17 +512,6 @@ function sessionIdForBrowserControlMethod(req: RequestFrame): string | null {
}
}
async function bypassOverlay(tabId: number, enabled: boolean): Promise<void> {
try {
await chrome.tabs.sendMessage(tabId, {
type: OVERLAY_AUTOMATION_BYPASS,
enabled,
});
} catch {
// Content script may be unavailable on restricted pages.
}
}
/**
* Resolves never; rejects with `AbortLikeError` as soon as the signal
* fires (or immediately if it is already aborted). Used by the
+1 -124
View File
@@ -17,8 +17,6 @@ import type {
ClickResult,
FillParams,
FillResult,
HoverParams,
HoverResult,
KeyModifier,
MouseButton,
PressParams,
@@ -55,12 +53,9 @@ export interface InteractionDeps {
defaultTimeoutMs?: number;
/** Temporarily disable overlay click blocker during CDP automation. */
bypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>;
/** Keep hover hit-testing active for the caller's next observation/action. */
keepOverlayBypassAfterHover?: boolean;
}
const DEFAULT_TIMEOUT_MS = 30_000;
const DEFAULT_HOVER_SETTLE_MS = 200;
let defaultDeps: { cdp: ChromiumCdp; tabsApi: ChromeTabsApi } | null = null;
function getDefaultDeps(): { cdp: ChromiumCdp; tabsApi: ChromeTabsApi } {
@@ -105,27 +100,6 @@ function throwIfAborted(signal: AbortSignal | undefined): RpcError | null {
return null;
}
function isAbortLikeError(err: unknown): boolean {
return err instanceof DOMException && err.name === "AbortError";
}
async function wait(ms: number, signal?: AbortSignal): Promise<void> {
if (signal?.aborted) throw new DOMException("aborted", "AbortError");
await new Promise<void>((resolve, reject) => {
const cleanup = () => signal?.removeEventListener("abort", onAbort);
const timer = setTimeout(() => {
cleanup();
resolve();
}, ms);
const onAbort = () => {
clearTimeout(timer);
cleanup();
reject(new DOMException("aborted", "AbortError"));
};
signal?.addEventListener("abort", onAbort, { once: true });
});
}
/**
* Resolve `{ref?, selector?}` into a `backendNodeId`. Returns an
* `RpcError` if the caller supplied neither (or both), or if neither
@@ -312,104 +286,7 @@ export async function handleClick(
message: err instanceof Error ? err.message : String(err),
};
} finally {
if (automationBypassEnabled && deps.bypassOverlay && !deps.keepOverlayBypassAfterHover) {
try {
await deps.bypassOverlay(target.tabId, false);
} catch (err) {
console.debug("[bsk interaction] overlay bypass disable failed", err);
}
}
}
return attachDialogs(deps.cdp, target.tabId, dialogCursor, {
tab_id: target.tabId,
used_ref: node.usedRef,
used_selector: node.usedSelector,
x: centre.x,
y: centre.y,
});
}
// ---------------------------------------------------------------------------
// tool.hover
// ---------------------------------------------------------------------------
export async function handleHover(
manager: SessionManager,
params: HoverParams,
deps: InteractionDeps = getDefaultDeps(),
): Promise<HoverResult | RpcError> {
const ctxOrErr = lookupSession(manager, params, "hover");
if (isRpcError(ctxOrErr)) return ctxOrErr;
const ctx = ctxOrErr;
const aborted = throwIfAborted(deps.signal);
if (aborted) return aborted;
const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "hover");
if (denied) return denied;
const dialogCursor = markDialogCursor(deps.cdp, target.tabId);
const node = await resolveBackendNode(deps.cdp, ctx, target, params, "hover");
if (isRpcError(node)) return node;
try {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const scrollErr = await scrollNodeIntoView(deps.cdp, target.tabId, node.backendNodeId);
if (scrollErr) return scrollErr;
} catch (err) {
return {
code: "cdp_failed",
message: err instanceof Error ? err.message : String(err),
};
}
const centre = await nodeCentre(deps.cdp, target.tabId, node.backendNodeId);
if (isRpcError(centre)) return centre;
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "hover aborted" };
}
const modifiers = modifiersBitfield(params.modifiers);
const overlayBlocking = await checkOverlayAtPoint(deps.cdp, target.tabId, centre.x, centre.y);
let automationBypassEnabled = false;
let hoverCompleted = false;
if (overlayBlocking && deps.bypassOverlay) {
try {
await deps.bypassOverlay(target.tabId, true);
automationBypassEnabled = true;
} catch (err) {
console.debug("[bsk interaction] overlay bypass enable failed", err);
}
}
try {
await deps.cdp.send(target.tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: centre.x,
y: centre.y,
modifiers,
});
const settleMs = params.settle_ms ?? DEFAULT_HOVER_SETTLE_MS;
if (settleMs > 0) {
await wait(settleMs, deps.signal);
}
hoverCompleted = true;
} catch (err) {
if (isAbortLikeError(err)) {
return { code: "cancelled", message: "hover aborted" };
}
return {
code: "cdp_failed",
message: err instanceof Error ? err.message : String(err),
};
} finally {
if (
automationBypassEnabled &&
deps.bypassOverlay &&
(!deps.keepOverlayBypassAfterHover || !hoverCompleted)
) {
if (automationBypassEnabled && deps.bypassOverlay) {
try {
await deps.bypassOverlay(target.tabId, false);
} catch (err) {
+47 -116
View File
@@ -5,14 +5,16 @@
import {
type ActiveScopeBlock,
applyVomInteractionRecovery,
type CondSurface,
isVomReferenceNode,
renderVom,
type VomNode,
type VomScene,
} from "@browser-skill/vom";
import { ChromiumCdp } from "@/browser-driver/chromium-cdp";
import {
type CaptureSuppressSendToTab,
withOverlaysHiddenForCapture,
} from "@/lib/capture-suppress-bridge";
import type { SessionManager } from "@/session-manager/manager";
import type {
GetHtmlParams,
@@ -116,6 +118,12 @@ export interface ScreenshotDeps {
cdp?: SharedCdpRunner;
tabsApi: ChromeTabsApi;
captureApi: ChromeTabsCaptureApi;
/**
* Bridge used to hide the in-page overlay while a screenshot is taken,
* so captured frames only contain page content. Defaults to
* `chrome.tabs.sendMessage`; tests inject a fake.
*/
sendToTab?: CaptureSuppressSendToTab;
}
function defaultScreenshotDeps(): ScreenshotDeps {
@@ -194,7 +202,12 @@ export async function handleScreenshot(
if (isRpcError(node)) return node;
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
await deps.cdp.ensureAttachedToUrl?.(target.tabId, target.url);
const captured = await captureElementScreenshot(deps.cdp, target.tabId, node.backendNodeId);
const cdp = deps.cdp;
const captured = await withOverlaysHiddenForCapture(
target.tabId,
() => captureElementScreenshot(cdp, target.tabId, node.backendNodeId),
deps.sendToTab,
);
if (isRpcError(captured)) return captured;
return withShotDialogs({
image_base64: captured.image_base64,
@@ -214,7 +227,11 @@ export async function handleScreenshot(
}
try {
const dataUrl = await deps.captureApi.captureVisibleTab(target.windowId, { format: "png" });
const dataUrl = await withOverlaysHiddenForCapture(
target.tabId,
() => deps.captureApi.captureVisibleTab(target.windowId, { format: "png" }),
deps.sendToTab,
);
const image_base64 = stripDataUrlPrefix(dataUrl);
const dims = parsePngDimensions(image_base64) ?? { width: 0, height: 0 };
return withShotDialogs({
@@ -910,28 +927,39 @@ function buildActiveScopeBlocks(nodes: VomNode[], signals: VomNodeDomSignals): A
return blocks;
}
function labelForSurfaceMatch(node: VomNode): string {
return (
cleanAttr(node.name) ??
cleanAttr(node.text) ??
cleanAttr(node.attrs?.["aria-label"]) ??
cleanAttr(node.attrs?.title) ??
""
);
}
function buildConditionalSurfaces(nodes: VomNode[], captured: CapturedViewModel): CondSurface[] {
const probes = captured.surfaceProbes ?? [];
if (probes.length === 0) return [];
const surfaces: CondSurface[] = [];
const recoveredNodes = applyVomInteractionRecovery(nodes);
const signals = capturedOnlySignals(captured.nodes);
const nodesWithLabels = nodes
.map((node) => ({ node, key: normalizeProbeKey(labelForSurfaceMatch(node)) }))
.filter((entry) => entry.key.length > 0);
const used = new Set<number>();
for (const probe of probes) {
if (probe.subItems.length === 0 || used.has(probe.triggerBackendNodeId)) continue;
const match = findSurfaceTriggerNode(
probe.triggerBackendNodeId,
recoveredNodes,
signals,
probe.triggerPoint,
);
const probeKey = normalizeProbeKey(probe.triggerLabel);
if (!probeKey || probe.subItems.length === 0) continue;
const match =
nodesWithLabels.find((entry) => entry.key === probeKey && !used.has(entry.node.id)) ??
nodesWithLabels.find(
(entry) =>
!used.has(entry.node.id) &&
(probeKey.startsWith(entry.key) || entry.key.startsWith(probeKey)),
);
if (!match) continue;
if (used.has(match.id)) continue;
used.add(probe.triggerBackendNodeId);
used.add(match.id);
used.add(match.node.id);
surfaces.push({
triggerId: match.id,
triggerId: match.node.id,
triggerAction: probe.triggerAction,
subItems: probe.subItems,
});
@@ -939,77 +967,6 @@ function buildConditionalSurfaces(nodes: VomNode[], captured: CapturedViewModel)
return surfaces;
}
function findSurfaceTriggerNode(
triggerBackendNodeId: number,
nodes: VomNode[],
signals: VomNodeDomSignals,
triggerPoint?: { x: number; y: number },
): VomNode | undefined {
const renderedById = new Map(nodes.map((node) => [node.id, node]));
const exact = renderedById.get(triggerBackendNodeId);
if (exact && isSurfaceAttachableNode(exact)) return exact;
const domDescendant = nodes.find(
(node) =>
isSurfaceAttachableNode(node) &&
(node.domParentId === triggerBackendNodeId ||
node.domAncestorIds?.includes(triggerBackendNodeId) === true),
);
if (domDescendant) return domDescendant;
const queue = [...(signals.childrenByParentId.get(triggerBackendNodeId) ?? [])];
while (queue.length > 0) {
const child = queue.shift() as CapturedNode;
const rendered = renderedById.get(child.backendNodeId);
if (rendered && isSurfaceAttachableNode(rendered)) return rendered;
queue.push(...(signals.childrenByParentId.get(child.backendNodeId) ?? []));
}
let current = signals.capturedByBackendId.get(triggerBackendNodeId);
while (current?.parentBackendNodeId !== null && current?.parentBackendNodeId !== undefined) {
const parent = renderedById.get(current.parentBackendNodeId);
if (parent && isSurfaceAttachableNode(parent)) return parent;
current = signals.capturedByBackendId.get(current.parentBackendNodeId);
}
if (triggerPoint) {
return findSurfaceNodeByPoint(triggerPoint, nodes, signals);
}
return undefined;
}
function isSurfaceAttachableNode(node: VomNode): boolean {
return isVomReferenceNode(node);
}
function findSurfaceNodeByPoint(
point: { x: number; y: number },
nodes: VomNode[],
signals: VomNodeDomSignals,
): VomNode | undefined {
let best: { node: VomNode; score: number } | undefined;
for (const node of nodes) {
if (!isSurfaceAttachableNode(node)) continue;
const rect = node.rect ?? signals.capturedByBackendId.get(node.id)?.rect;
if (!rect) continue;
const contains =
point.x >= rect.x &&
point.x <= rect.x + rect.w &&
point.y >= rect.y &&
point.y <= rect.y + rect.h;
const centerX = rect.x + rect.w / 2;
const centerY = rect.y + rect.h / 2;
const distance = Math.hypot(point.x - centerX, point.y - centerY);
if (!contains && distance > 40) continue;
const score = contains ? distance : distance + 1_000;
if (!best || score < best.score) {
best = { node, score };
}
}
return best?.node;
}
function axNodeInOverlaySubtree(
axNode: CdpAxNode,
axById: Map<string, CdpAxNode>,
@@ -1134,8 +1091,6 @@ export interface SnapshotDeps {
get(tabId: number): Promise<chrome.tabs.Tab>;
query(q: chrome.tabs.QueryInfo): Promise<chrome.tabs.Tab[]>;
};
conditionalSurfaceProbe?: boolean;
hoverProbeBypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>;
}
let defaultDeps: SnapshotDeps | null = null;
@@ -1281,13 +1236,9 @@ async function captureForVom(
cdp: CdpRunner,
tabId: number,
conditionalSurfaceProbe: boolean,
hoverProbeBypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>,
): Promise<CapturedViewModel> {
try {
return await captureViewModel(cdp, tabId, {
conditionalSurfaceProbe,
hoverProbeBypassOverlay,
});
return await captureViewModel(cdp, tabId, { conditionalSurfaceProbe });
} catch {
return fallbackCapturedViewModel(cdp, tabId);
}
@@ -1326,14 +1277,7 @@ async function handleVomObservation(
{},
);
const axNodes = result.nodes ?? [];
const effectiveConditionalSurfaceProbe =
deps.conditionalSurfaceProbe ?? conditionalSurfaceProbe;
const captured = await captureForVom(
deps.cdp,
target.tabId,
effectiveConditionalSurfaceProbe,
deps.hoverProbeBypassOverlay,
);
const captured = await captureForVom(deps.cdp, target.tabId, conditionalSurfaceProbe);
const scene = buildVomScene(axNodes, captured, { pageUrl: target.url });
const rendered = renderVom(scene, {
maxDepth: params.max_depth,
@@ -1350,19 +1294,6 @@ async function handleVomObservation(
ref_count: rendered.refs.length,
tab_id: target.tabId,
truncated: rendered.truncated,
...(toolName === "observe" && (params as ObserveParams).debug_surfaces
? {
debug: {
surface_probes: (captured.surfaceProbes ?? []).map((probe) => ({
trigger_backend_node_id: probe.triggerBackendNodeId,
...(probe.triggerPoint ? { trigger_point: probe.triggerPoint } : {}),
trigger_action: probe.triggerAction,
sub_items: probe.subItems,
...(probe.confidence ? { confidence: probe.confidence } : {}),
})),
},
}
: {}),
});
} catch (err) {
return {
+15 -4
View File
@@ -49,6 +49,7 @@ interface ActiveRecording {
purpose?: string;
steps: DraftTraceStep[];
startedAt: string;
startedAtMs: number;
finishPromise: Promise<Trace>;
resolveFinish: (trace: Trace) => void;
rejectFinish: (err: Error) => void;
@@ -373,7 +374,11 @@ async function rearmRecording(
// stuck with pointer-events:none (page usable, Interrupt dead). RecordOverlay
// already hides the control chrome while activeRecord is set.
for (let attempt = 0; attempt < RECORD_REARM_MAX_ATTEMPTS; attempt += 1) {
const startMsg: RecordStartMessage = { type: RECORD_START, requestId: recording.requestId };
const startMsg: RecordStartMessage = {
type: RECORD_START,
requestId: recording.requestId,
startedAtMs: recording.startedAtMs,
};
try {
await sendRecordStartWithAck(targetTabId, startMsg, deps.sendToTab);
recording.tabId = targetTabId;
@@ -501,7 +506,11 @@ export function attachRecordQueryListener(deps: RecordDeps = getDefaultDeps()):
return;
}
await rearmRecording(recording, tabId, deps);
sendResponse({ active: true, requestId: recording.requestId });
sendResponse({
active: true,
requestId: recording.requestId,
startedAtMs: recording.startedAtMs,
});
})();
return true;
};
@@ -572,6 +581,7 @@ export async function handleRecordStart(
rejectFinish = reject;
});
const navigateUrl = params.url ?? RECORD_DEFAULT_START_URL;
const startedAtMs = Date.now();
recordings.set(params.session_id, {
requestId,
tabId: target.tabId,
@@ -579,7 +589,8 @@ export async function handleRecordStart(
startUrl: navigateUrl,
...(params.purpose ? { purpose: params.purpose } : {}),
steps: [],
startedAt: new Date().toISOString(),
startedAt: new Date(startedAtMs).toISOString(),
startedAtMs,
finishPromise,
resolveFinish,
rejectFinish,
@@ -707,7 +718,7 @@ export async function handleRecordStart(
if (cancelled) return cancelled;
}
const startMsg: RecordStartMessage = { type: RECORD_START, requestId };
const startMsg: RecordStartMessage = { type: RECORD_START, requestId, startedAtMs };
try {
await sendRecordStartWithAck(target.tabId, startMsg, deps.sendToTab);
@@ -51,156 +51,6 @@ function fakeSnapshotReply() {
};
}
function hoverTriggerSnapshotReply() {
const S = [
"html",
"body",
"button",
"class",
"user-avatar",
"position",
"static",
"pointer-events",
"auto",
"cursor",
"pointer",
];
const i = (s: string) => S.indexOf(s);
return {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1],
nodeType: [1, 1, 1],
nodeName: [i("html"), i("body"), i("button")],
backendNodeId: [10, 11, 12],
attributes: [[], [], [i("class"), i("user-avatar")]],
},
layout: {
nodeIndex: [1, 2],
styles: [
[i("static"), i("auto"), i("pointer")],
[i("static"), i("auto"), i("pointer")],
],
bounds: [
[0, 0, 1000, 800],
[940, 16, 32, 32],
],
paintOrders: [0, 1],
},
},
],
};
}
function twoHoverTriggerSnapshotReply() {
const S = [
"html",
"body",
"button",
"class",
"user-avatar",
"secondary-dropdown-trigger",
"position",
"static",
"pointer-events",
"auto",
"cursor",
"pointer",
];
const i = (s: string) => S.indexOf(s);
return {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1, 1],
nodeType: [1, 1, 1, 1],
nodeName: [i("html"), i("body"), i("button"), i("button")],
backendNodeId: [10, 11, 12, 13],
attributes: [
[],
[],
[i("class"), i("user-avatar")],
[i("class"), i("secondary-dropdown-trigger")],
],
},
layout: {
nodeIndex: [1, 2, 3],
styles: [
[i("static"), i("auto"), i("pointer")],
[i("static"), i("auto"), i("pointer")],
[i("static"), i("auto"), i("pointer")],
],
bounds: [
[0, 0, 1000, 800],
[940, 16, 32, 32],
[100, 120, 32, 32],
],
paintOrders: [0, 1, 2],
},
},
],
};
}
function nestedHoverTriggerSnapshotReply() {
const S = [
"html",
"body",
"div",
"img",
"class",
"tg-avatar",
"tg-avatar__inner",
"tg-avatar__image",
"position",
"static",
"pointer-events",
"auto",
"cursor",
"pointer",
];
const i = (s: string) => S.indexOf(s);
return {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1, 2, 3],
nodeType: [1, 1, 1, 1, 1],
nodeName: [i("html"), i("body"), i("div"), i("div"), i("img")],
backendNodeId: [10, 11, 12, 13, 14],
attributes: [
[],
[],
[i("class"), i("tg-avatar")],
[i("class"), i("tg-avatar__inner")],
[i("class"), i("tg-avatar__image")],
],
},
layout: {
nodeIndex: [1, 2, 3, 4],
styles: [
[i("static"), i("auto"), i("pointer")],
[i("static"), i("auto"), i("pointer")],
[i("static"), i("auto"), i("pointer")],
[i("static"), i("auto"), i("pointer")],
],
bounds: [
[0, 0, 1000, 800],
[940, 16, 32, 32],
[942, 18, 28, 28],
[944, 20, 24, 24],
],
paintOrders: [0, 1, 2, 3],
},
},
],
};
}
function makeCdp(snapshot: unknown) {
return {
send: vi.fn(async (_tab: number, method: string) => {
@@ -276,11 +126,11 @@ describe("captureViewModel", () => {
});
it("runs hover surface probes when explicitly enabled", async () => {
let hoverStateCalls = 0;
let runtimeCalls = 0;
const cdp = {
send: vi.fn(async (_tabId: number, method: string, params?: object) => {
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return hoverTriggerSnapshotReply();
if (method === "DOMSnapshot.captureSnapshot") return fakeSnapshotReply();
if (method === "Page.getLayoutMetrics") {
return {
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
@@ -291,23 +141,22 @@ describe("captureViewModel", () => {
if (expression.includes("input,textarea,select")) {
return { result: { value: { controls: [], childFrames: [] } } };
}
if (expression.includes("document.styleSheets")) {
return { result: { value: [] } };
}
if (expression.includes("querySelectorAll(selectors)")) {
hoverStateCalls += 1;
return hoverStateCalls === 1
? { result: { value: [] } }
: {
result: {
value: [
{ text: "My profile", role: "", tag: "a", x: 900, y: 60 },
{ text: "Sign out", role: "", tag: "a", x: 900, y: 90 },
],
},
};
}
throw new Error(`unexpected Runtime.evaluate: ${expression.slice(0, 80)}`);
runtimeCalls += 1;
return runtimeCalls === 1
? {
result: {
value: [
{
triggerSel: ".menu",
affectedSub: " .items",
label: "Products",
x: 50,
y: 20,
},
],
},
}
: { result: { value: ["Shoes", "Bags"] } };
}
if (method === "Input.dispatchMouseEvent") return {};
throw new Error(`unexpected ${method}`);
@@ -317,18 +166,12 @@ describe("captureViewModel", () => {
const result = await captureViewModel(cdp, 4, { conditionalSurfaceProbe: true });
expect(result.surfaceProbes).toEqual([
{
triggerBackendNodeId: 12,
triggerPoint: { x: 956, y: 32 },
triggerAction: "hover",
subItems: ["My profile", "Sign out"],
confidence: "high",
},
{ triggerLabel: "Products", triggerAction: "hover", subItems: ["Shoes", "Bags"] },
]);
expect(cdp.send).toHaveBeenCalledWith(
4,
"Input.dispatchMouseEvent",
expect.objectContaining({ type: "mouseMoved", x: 956, y: 32 }),
expect.objectContaining({ type: "mouseMoved", x: 50, y: 20 }),
);
expect(cdp.send).toHaveBeenCalledWith(
4,
@@ -337,91 +180,6 @@ describe("captureViewModel", () => {
);
});
it("uses a fresh baseline for each hover candidate", async () => {
let hoverStateCalls = 0;
const cdp = {
send: vi.fn(async (_tabId: number, method: string, params?: object) => {
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return twoHoverTriggerSnapshotReply();
if (method === "Page.getLayoutMetrics") {
return {
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
};
}
if (method === "Runtime.evaluate") {
const expression = (params as { expression?: string } | undefined)?.expression ?? "";
if (expression.includes("input,textarea,select")) {
return { result: { value: { controls: [], childFrames: [] } } };
}
if (expression.includes("document.styleSheets")) {
return { result: { value: [] } };
}
if (expression.includes("querySelectorAll(selectors)")) {
hoverStateCalls += 1;
const visibleMenu = [
{ text: "My profile", role: "", tag: "a", x: 900, y: 60 },
{ text: "Sign out", role: "", tag: "a", x: 900, y: 90 },
];
return hoverStateCalls === 1
? { result: { value: [] } }
: { result: { value: visibleMenu } };
}
throw new Error(`unexpected Runtime.evaluate: ${expression.slice(0, 80)}`);
}
if (method === "Input.dispatchMouseEvent") return {};
throw new Error(`unexpected ${method}`);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const result = await captureViewModel(cdp, 4, { conditionalSurfaceProbe: true });
expect(result.surfaceProbes).toEqual([
expect.objectContaining({
triggerBackendNodeId: 12,
subItems: ["My profile", "Sign out"],
}),
]);
expect(hoverStateCalls).toBeGreaterThanOrEqual(4);
});
it("deduplicates nested hover candidates for one visual trigger", async () => {
let hoverMoves = 0;
const cdp = {
send: vi.fn(async (_tabId: number, method: string, params?: object) => {
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return nestedHoverTriggerSnapshotReply();
if (method === "Page.getLayoutMetrics") {
return {
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
};
}
if (method === "Runtime.evaluate") {
const expression = (params as { expression?: string } | undefined)?.expression ?? "";
if (expression.includes("input,textarea,select")) {
return { result: { value: { controls: [], childFrames: [] } } };
}
if (expression.includes("document.styleSheets")) {
return { result: { value: [] } };
}
if (expression.includes("querySelectorAll(selectors)")) {
return { result: { value: [] } };
}
throw new Error(`unexpected Runtime.evaluate: ${expression.slice(0, 80)}`);
}
if (method === "Input.dispatchMouseEvent") {
const point = params as { x?: number; y?: number };
if (point.x !== -10 && point.x !== 0) hoverMoves += 1;
return {};
}
throw new Error(`unexpected ${method}`);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
await captureViewModel(cdp, 4, { conditionalSurfaceProbe: true });
expect(hoverMoves).toBe(1);
});
it("excludes the agent's own overlay shadow host and its inlined shadow subtree", async () => {
// DOMSnapshot inlines an open shadow root's content as descendants of the
// host. The agent's WXT overlay host carries a fixed full-viewport
+113 -277
View File
@@ -5,7 +5,6 @@
// `styles` columns are [position, pointer-events, cursor] in that order.
import type { Rect, Viewport } from "@browser-skill/vom";
import { evaluateHoverTrigger } from "@/lib/hover-trigger-policy";
import { isOverlayHostNode, OVERLAY_HOST_SELECTOR } from "../../lib/overlay-bridge";
import type { CdpRunner } from "../shared";
@@ -46,11 +45,9 @@ export interface CapturedNode {
export type CapturedIframeNodes = Map<number, CapturedNode[]>;
export interface CapturedSurfaceProbe {
triggerBackendNodeId: number;
triggerPoint?: { x: number; y: number };
triggerLabel: string;
triggerAction: "hover" | "focus" | string;
subItems: string[];
confidence?: "high" | "medium" | "low";
}
export interface CapturedViewModel {
@@ -64,7 +61,6 @@ export interface CapturedViewModel {
export interface CaptureViewModelOptions {
conditionalSurfaceProbe?: boolean;
hoverProbeBypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>;
}
/** Sparse array format Chrome uses for infrequently-set per-node fields. */
@@ -229,12 +225,11 @@ interface RuntimeEvaluateReply {
}
interface HoverCandidate {
backendNodeId: number;
label?: string;
triggerSel: string;
affectedSub: string;
label: string;
x: number;
y: number;
score: number;
reasons: string[];
}
interface CdpDomNode {
@@ -327,19 +322,17 @@ function collectBackendIdsFromDomNode(node: CdpDomNode | undefined, out: Set<num
}
const MAX_HOVER_PROBE_MS = 2_000;
const MAX_HOVER_TRIGGERS = 6;
const MAX_HOVER_SURFACES = 3;
const HOVER_SETTLE_MS = 300;
const MAX_HOVER_SUB_ITEMS = 12;
const MAX_HOVER_TRIGGERS = 8;
const HOVER_SETTLE_MS = 200;
function runtimeValue<T>(reply: RuntimeEvaluateReply): T | undefined {
return reply.result?.value as T | undefined;
}
function hoverCssTriggerScanExpression(): string {
function hoverCssScanExpression(): string {
return `(() => {
const visibilityProps = ["display", "visibility", "opacity", "maxHeight", "height", "overflow"];
const centres = [];
const pairs = [];
const seenRules = new Set();
for (const sheet of Array.from(document.styleSheets)) {
let rules;
@@ -355,73 +348,90 @@ function hoverCssTriggerScanExpression(): string {
const hoverIndex = part.indexOf(":hover");
if (hoverIndex < 0) continue;
const triggerSel = part.slice(0, hoverIndex).trim();
const affectedSub = part.slice(hoverIndex + 6).trim();
if (!triggerSel) continue;
if (seenRules.has(triggerSel)) continue;
seenRules.add(triggerSel);
let elements;
try { elements = Array.from(document.querySelectorAll(triggerSel)); } catch { continue; }
for (const el of elements) {
if (!(el instanceof HTMLElement)) continue;
const rect = el.getBoundingClientRect();
const style = getComputedStyle(el);
if (rect.width <= 0 || rect.height <= 0 || style.visibility === "hidden" || style.display === "none" || style.pointerEvents === "none") continue;
centres.push({ x: rect.left + rect.width / 2, y: rect.top + rect.height / 2 });
if (centres.length >= 24) return centres;
}
const key = triggerSel + "||" + affectedSub;
if (seenRules.has(key)) continue;
seenRules.add(key);
pairs.push({ triggerSel, affectedSub });
}
}
}
return centres;
const candidates = [];
const seenLabels = new Set();
const isUnsafeTrigger = (el) => {
if (!(el instanceof HTMLElement)) return true;
const tag = el.tagName.toLowerCase();
if (["input", "textarea", "select", "option"].includes(tag)) return true;
if (el.isContentEditable) return true;
if (el.hasAttribute("disabled") || el.hasAttribute("inert")) return true;
if ((el.getAttribute("aria-disabled") || "").toLowerCase() === "true") return true;
return false;
};
for (const pair of pairs) {
let elements;
try { elements = Array.from(document.querySelectorAll(pair.triggerSel)); } catch { continue; }
for (const el of elements) {
if (!(el instanceof Element)) continue;
if (isUnsafeTrigger(el)) continue;
const rect = el.getBoundingClientRect();
const style = getComputedStyle(el);
if (rect.width <= 0 || rect.height <= 0 || style.visibility === "hidden" || style.display === "none" || style.pointerEvents === "none") continue;
const label = (el.textContent || "").replace(/\\s+/g, " ").trim();
if (!label || seenLabels.has(label)) continue;
seenLabels.add(label);
candidates.push({
triggerSel: pair.triggerSel,
affectedSub: pair.affectedSub,
label,
x: rect.left + rect.width / 2,
y: rect.top + rect.height / 2,
});
if (candidates.length >= ${MAX_HOVER_TRIGGERS}) return candidates;
}
}
return candidates;
})()`;
}
interface HoverRuntimeItem {
text: string;
role: string;
tag: string;
x: number;
y: number;
}
function hoverStateExpression(): string {
function hoverCollectExpression(candidate: HoverCandidate): string {
return `(() => {
const triggerSel = ${JSON.stringify(candidate.triggerSel)};
const rawAffectedSub = ${JSON.stringify(candidate.affectedSub)};
const x = ${JSON.stringify(candidate.x)};
const y = ${JSON.stringify(candidate.y)};
const hit = document.elementFromPoint(x, y);
const trigger = hit instanceof Element ? hit.closest(triggerSel) : null;
if (!trigger) return [];
const affectedSub = rawAffectedSub.replace(/^[>+~]\\s*/, "").trim();
let targets = [];
if (!affectedSub) {
targets = [trigger];
} else {
try { targets = Array.from(trigger.querySelectorAll(affectedSub)); } catch { targets = []; }
}
const items = [];
const seen = new Set();
const selectors = [
"a",
"button",
"[role='menuitem']",
"[role='menuitemcheckbox']",
"[role='menuitemradio']",
"[role='option']",
"[role='tab']",
"[role='link']",
"[role='button']"
].join(",");
const push = (el) => {
if (!(el instanceof HTMLElement)) return;
const style = getComputedStyle(el);
const rect = el.getBoundingClientRect();
if (rect.width <= 0 || rect.height <= 0) return;
if (style.display === "none" || style.visibility === "hidden" || style.opacity === "0") return;
const text = String(
el.getAttribute("aria-label") ||
el.getAttribute("title") ||
el.textContent ||
""
).replace(/\\s+/g, " ").trim();
const push = (value) => {
const text = String(value || "").replace(/\\s+/g, " ").trim();
if (!text || seen.has(text)) return;
seen.add(text);
items.push({
text,
role: (el.getAttribute("role") || "").toLowerCase(),
tag: el.tagName.toLowerCase(),
x: rect.left,
y: rect.top,
});
items.push(text);
};
for (const el of Array.from(document.querySelectorAll(selectors))) push(el);
return items.slice(0, 400);
for (const target of targets) {
if (!(target instanceof Element)) continue;
const style = getComputedStyle(target);
if (style.display === "none" || style.visibility === "hidden" || style.opacity === "0") continue;
const clickable = target.querySelectorAll('a, button, [role="menuitem"], [role="option"]');
if (clickable.length > 0) {
for (const child of Array.from(clickable)) push(child.textContent);
} else {
push(target.textContent);
}
}
return items.slice(0, 12);
})()`;
}
@@ -443,217 +453,45 @@ async function clearHover(cdp: CdpRunner, tabId: number): Promise<void> {
);
}
function capturedText(node: CapturedNode): string | undefined {
const value =
node.attrs["aria-label"] ?? node.attrs.title ?? node.attrs.alt ?? node.textContent ?? "";
const clean = value.replace(/\s+/g, " ").trim();
return clean || undefined;
}
function hasGraphicDescendant(
node: CapturedNode,
childrenByParentId: Map<number, CapturedNode[]>,
depth = 0,
): boolean {
if (depth > 3) return false;
for (const child of childrenByParentId.get(node.backendNodeId) ?? []) {
const tag = child.tag.toLowerCase();
if (["img", "svg", "use", "path", "i"].includes(tag)) return true;
if (hasGraphicDescendant(child, childrenByParentId, depth + 1)) return true;
}
return false;
}
function roleOf(node: CapturedNode): string {
return (node.attrs.role ?? "").toLowerCase();
}
function scoreHoverCandidate(
node: CapturedNode,
childrenByParentId: Map<number, CapturedNode[]>,
cssHoverPoints: Array<{ x: number; y: number }>,
): HoverCandidate | null {
const rect = node.rect;
if (!rect) return null;
const label = capturedText(node);
const cssHoverMatch = cssHoverPoints.some(
(point) =>
point.x >= rect.x &&
point.x <= rect.x + rect.w &&
point.y >= rect.y &&
point.y <= rect.y + rect.h,
);
const decision = evaluateHoverTrigger({
tag: node.tag,
role: roleOf(node),
label,
attrs: node.attrs,
rect,
cursor: node.cursor,
pointerEvents: node.pointerEvents,
hasGraphicDescendant: hasGraphicDescendant(node, childrenByParentId),
cssHoverMatch,
});
if (!decision.eligible) return null;
return {
backendNodeId: node.backendNodeId,
label,
x: rect.x + rect.w / 2,
y: rect.y + rect.h / 2,
score: decision.score,
reasons: decision.reasons,
};
}
function buildHoverCandidates(
nodes: CapturedNode[],
cssHoverPoints: Array<{ x: number; y: number }>,
): HoverCandidate[] {
const childrenByParentId = new Map<number, CapturedNode[]>();
const parentByBackendId = new Map<number, number | null>();
for (const node of nodes) {
parentByBackendId.set(node.backendNodeId, node.parentBackendNodeId);
if (node.parentBackendNodeId === null) continue;
const children = childrenByParentId.get(node.parentBackendNodeId) ?? [];
children.push(node);
childrenByParentId.set(node.parentBackendNodeId, children);
}
const candidates = nodes
.map((node) => scoreHoverCandidate(node, childrenByParentId, cssHoverPoints))
.filter((candidate): candidate is HoverCandidate => candidate !== null)
.sort((a, b) => b.score - a.score);
const deduped: HoverCandidate[] = [];
const seen = new Set<number>();
for (const candidate of candidates) {
if (seen.has(candidate.backendNodeId)) continue;
if (deduped.some((existing) => sameHoverCluster(existing, candidate, parentByBackendId))) {
continue;
}
seen.add(candidate.backendNodeId);
deduped.push(candidate);
if (deduped.length >= MAX_HOVER_TRIGGERS) break;
}
return deduped;
}
function sameHoverCluster(
a: HoverCandidate,
b: HoverCandidate,
parentByBackendId: Map<number, number | null>,
): boolean {
if (Math.hypot(a.x - b.x, a.y - b.y) <= 8) return true;
return (
isBackendAncestor(a.backendNodeId, b.backendNodeId, parentByBackendId) ||
isBackendAncestor(b.backendNodeId, a.backendNodeId, parentByBackendId)
);
}
function isBackendAncestor(
ancestorId: number,
nodeId: number,
parentByBackendId: Map<number, number | null>,
): boolean {
let parentId = parentByBackendId.get(nodeId);
let guard = 0;
while (parentId !== null && parentId !== undefined && guard < parentByBackendId.size) {
if (parentId === ancestorId) return true;
parentId = parentByBackendId.get(parentId);
guard += 1;
}
return false;
}
function diffHoverItems(before: HoverRuntimeItem[], after: HoverRuntimeItem[]): string[] {
const beforeKeys = new Set(before.map((item) => item.text.toLowerCase()));
const out: string[] = [];
const seen = new Set<string>();
for (const item of after) {
const text = item.text.replace(/\s+/g, " ").trim();
const key = text.toLowerCase();
if (!text || beforeKeys.has(key) || seen.has(key)) continue;
seen.add(key);
out.push(text);
if (out.length >= MAX_HOVER_SUB_ITEMS) break;
}
return out;
}
function confidenceForHover(
candidate: HoverCandidate,
subItems: string[],
): "high" | "medium" | "low" {
if (subItems.length >= 2 && candidate.score >= 80) return "high";
if (subItems.length >= 2 || candidate.score >= 80) return "medium";
return "low";
}
async function probeHoverSurfaces(
cdp: CdpRunner,
tabId: number,
nodes: CapturedNode[],
options: CaptureViewModelOptions,
): Promise<CapturedSurfaceProbe[]> {
async function probeHoverSurfaces(cdp: CdpRunner, tabId: number): Promise<CapturedSurfaceProbe[]> {
const started = Date.now();
try {
const cssScan = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverCssTriggerScanExpression(),
const scan = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverCssScanExpression(),
returnByValue: true,
});
const cssHoverPoints = runtimeValue<Array<{ x: number; y: number }>>(cssScan) ?? [];
const candidates = buildHoverCandidates(nodes, cssHoverPoints);
if (candidates.length === 0) return [];
const candidates = runtimeValue<HoverCandidate[]>(scan) ?? [];
const results: CapturedSurfaceProbe[] = [];
const seen = new Set<number>();
await options.hoverProbeBypassOverlay?.(tabId, true).catch(() => undefined);
try {
for (const candidate of candidates.slice(0, MAX_HOVER_TRIGGERS)) {
if (Date.now() - started > MAX_HOVER_PROBE_MS) break;
if (results.length >= MAX_HOVER_SURFACES) break;
if (seen.has(candidate.backendNodeId)) continue;
try {
await clearHover(cdp, tabId);
await wait(HOVER_SETTLE_MS);
const baselineReply = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
const baselineItems = runtimeValue<HoverRuntimeItem[]>(baselineReply) ?? [];
await cdp.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: candidate.x,
y: candidate.y,
});
await wait(HOVER_SETTLE_MS);
const collected = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
const subItems = diffHoverItems(
baselineItems,
runtimeValue<HoverRuntimeItem[]>(collected) ?? [],
);
if (subItems.length === 0) continue;
seen.add(candidate.backendNodeId);
results.push({
triggerBackendNodeId: candidate.backendNodeId,
triggerPoint: { x: candidate.x, y: candidate.y },
triggerAction: "hover",
subItems,
confidence: confidenceForHover(candidate, subItems),
});
} catch {
continue;
} finally {
await clearHover(cdp, tabId);
}
const seen = new Set<string>();
for (const candidate of candidates.slice(0, MAX_HOVER_TRIGGERS)) {
if (Date.now() - started > MAX_HOVER_PROBE_MS) break;
if (!candidate.label || seen.has(candidate.label)) continue;
try {
await cdp.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: candidate.x,
y: candidate.y,
});
await wait(HOVER_SETTLE_MS);
const collected = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverCollectExpression(candidate),
returnByValue: true,
});
const subItems = (runtimeValue<string[]>(collected) ?? [])
.map((item) => item.replace(/\s+/g, " ").trim())
.filter(Boolean);
if (subItems.length === 0) continue;
seen.add(candidate.label);
results.push({
triggerLabel: candidate.label,
triggerAction: "hover",
subItems,
});
} catch {
continue;
} finally {
await clearHover(cdp, tabId);
}
} finally {
await options.hoverProbeBypassOverlay?.(tabId, false).catch(() => undefined);
}
return results;
} catch (err) {
@@ -959,9 +797,7 @@ export async function captureViewModel(
await enrichFormControlStates(cdp, tabId, [nodes, ...iframeNodes.values()]);
const surfaceProbes = options.conditionalSurfaceProbe
? await probeHoverSurfaces(cdp, tabId, nodes, options)
: [];
const surfaceProbes = options.conditionalSurfaceProbe ? await probeHoverSurfaces(cdp, tabId) : [];
return { nodes, viewport, iframeNodes, surfaceProbes, excludedBackendNodeIds };
}
+2 -40
View File
@@ -314,21 +314,8 @@ export interface SnapshotResult {
dialogs?: JavaScriptDialogInfo[];
}
export interface ObserveParams extends SnapshotParams {
debug_surfaces?: boolean;
}
export interface ObserveResult extends SnapshotResult {
debug?: {
surface_probes?: Array<{
trigger_backend_node_id: number;
trigger_point?: { x: number; y: number };
trigger_action: string;
sub_items: string[];
confidence?: string;
}>;
};
}
export type ObserveParams = SnapshotParams;
export type ObserveResult = SnapshotResult;
export interface GetHtmlParams {
session_id: string;
@@ -423,25 +410,6 @@ export interface ClickResult {
dialogs?: JavaScriptDialogInfo[];
}
export interface HoverParams {
session_id: string;
ref?: string;
selector?: string;
tab_id?: number;
modifiers?: KeyModifier[];
settle_ms?: number;
timeout_ms?: number;
}
export interface HoverResult {
tab_id: number;
used_ref?: string;
used_selector?: string;
x: number;
y: number;
dialogs?: JavaScriptDialogInfo[];
}
export interface FillParams {
session_id: string;
value: string;
@@ -700,11 +668,6 @@ export type DraftTraceStep =
navigated_to?: string;
page_url?: string;
}
| {
op: "hover";
target: TargetDescriptor;
page_url?: string;
}
| {
op: "fill";
target: TargetDescriptor;
@@ -738,7 +701,6 @@ export type DraftTraceStep =
export type Step =
| ({ op: "navigate" } & StepCommon & { to: string })
| ({ op: "click" } & StepCommon & { target: TargetDescriptor })
| ({ op: "hover" } & StepCommon & { target: TargetDescriptor })
| ({ op: "fill" } & StepCommon & {
target: TargetDescriptor;
value: string;
+7 -11
View File
@@ -67,25 +67,22 @@ Write operations only affect tabs in the **Agent Window** (or tabs you **borrowe
```
bsk navigate <url> --session <id>
bsk observe --session <id> → primary semantic VOM view; reveals hover/focus surfaces
bsk snapshot --session <id> → static aria tree fallback when VOM is insufficient
bsk hover @e3 --session <id> → reveal hover-triggered menus before re-observing/clicking
bsk click @e4 --session <id> → or bsk fill, bsk select, bsk press
bsk observe --session <id> → again after navigation / DOM change
bsk snapshot --session <id> → aria tree with @e1, @e2, … refs
bsk observe --session <id> → semantic VOM view; may reveal hover/focus surfaces
bsk click @e3 --session <id> → or bsk fill, bsk select, bsk press
bsk snapshot --session <id> → again after navigation / DOM change
```
**Refs invalidate after navigation** — always re-snapshot before clicking, filling, or selecting on a new page.
Prefer `@eN` refs from the latest snapshot over raw CSS selectors. Use `--ref` / `--selector` when ambiguous (`bsk click --help`).
When VOM renders `[hover first: …]` on an element, the listed items are not currently clickable refs. Run `bsk hover <that-ref> --session <id>`, then immediately run `bsk snapshot` or `bsk observe` again and click the newly visible menu item ref. Do not click the trigger itself unless the user explicitly wants the trigger action.
## Observation priority
Start with `bsk observe` to understand page structure, text, controls, element refs, and conditional hover/focus surfaces. Use `bsk snapshot` only when you need the stricter static accessibility tree or VOM is insufficient. Only escalate to raw HTML or screenshots when the latest observation cannot answer the question:
Start with `bsk snapshot` to understand page structure, text, controls, and element refs. Use `bsk observe` when semantic VOM output or conditional hover/focus surfaces would materially help. Only escalate to raw HTML or screenshots when the latest observation cannot answer the question:
1. `bsk observe` — primary semantic VOM observation; may run bounded perception probes such as hover-surface discovery
2. `bsk snapshot` — strict static accessibility tree fallback
1. `bsk snapshot` — strict static page understanding and interaction planning
2. `bsk observe` — semantic VOM observation; may run bounded perception probes such as hover-surface discovery
3. `bsk get-html` — when hidden DOM, metadata, or markup details are required
4. `bsk screenshot` — when visual layout, canvas/image content, or styling cannot be inferred from the observation. Use `--ref @eN` (from the latest snapshot/observe) to crop to one element; omit `--ref` for the full visible tab.
@@ -189,7 +186,6 @@ bsk emulate --session <id> --off
| Command | Summary |
|---------|---------|
| `bsk click <ref-or-selector>` | Click element (`--button`, `--click-count`, `--modifiers`) |
| `bsk hover <ref-or-selector>` | Move the mouse to an element and wait for hover UI to settle (`--settle`, `--modifiers`) |
| `bsk fill <ref-or-selector> --value <text>` | Clear and type into input |
| `bsk select <ref-or-selector> --value <v>` | Set `<select>` option(s) by `value` (repeat `--value` for multi-select) |
| `bsk press <key>` | Key/combo (`Enter`, `Ctrl+A`, …; optional `--ref` to focus first) |
+2 -87
View File
@@ -10,8 +10,8 @@ use std::time::Duration;
use anyhow::Context;
use bsk_protocol::Method;
use bsk_protocol::tools::{
ClickParams, ClickResult, FillParams, FillResult, HoverParams, HoverResult, KeyModifier,
MouseButton, PressParams, PressResult, SelectParams, SelectResult,
ClickParams, ClickResult, FillParams, FillResult, KeyModifier, MouseButton, PressParams,
PressResult, SelectParams, SelectResult,
};
use clap::{Args, ValueEnum};
@@ -189,91 +189,6 @@ pub fn dispatch_click(args: ClickArgs, format: Format) -> Result<(), CliError> {
Ok(())
}
// ---------------------------------------------------------------------------
// bsk hover
// ---------------------------------------------------------------------------
#[derive(Debug, Clone, Args)]
pub struct HoverArgs {
/// Snapshot ref (`@e3`, `e3`) or CSS selector. Optional when `--ref`/`--selector` is used.
pub target: Option<String>,
/// Force-treat the value as a snapshot ref (overrides positional detection).
#[arg(long = "ref")]
pub ref_: Option<String>,
/// Force-treat the value as a CSS selector.
#[arg(long = "selector")]
pub selector: Option<String>,
#[arg(long)]
pub session: String,
#[arg(long = "tab-id")]
pub tab_id: Option<i64>,
/// Comma-separated modifiers (`alt,ctrl,shift,meta`).
#[arg(long, default_value = "")]
pub modifiers: String,
/// Wait after moving the mouse so hover-triggered UI can settle.
#[arg(long = "settle", default_value = "200ms", value_parser = parse_timeout_ms)]
pub settle: u32,
#[arg(long, default_value = "30s", value_parser = parse_timeout_ms)]
pub timeout: u32,
}
pub fn dispatch_hover(args: HoverArgs, format: Format) -> Result<(), CliError> {
let info = ensure_daemon().context("ensure daemon is running")?;
let (ref_, selector) = split_target(
args.target.clone(),
args.ref_.clone(),
args.selector.clone(),
)?;
let modifiers = parse_modifiers(&args.modifiers)
.map_err(|e| CliError::Local(anyhow::anyhow!("--modifiers: {e}")))?;
let params = HoverParams {
session_id: args.session.clone(),
ref_,
selector,
tab_id: args.tab_id,
modifiers: if modifiers.is_empty() {
None
} else {
Some(modifiers)
},
settle_ms: Some(args.settle),
timeout_ms: Some(args.timeout),
};
let reply: HoverResult = call(
info.sock_path,
Method::ToolHover,
params,
"hover-1",
args.timeout,
)?;
match format {
Format::Json => {
println!(
"{}",
serde_json::to_string_pretty(&reply)
.map_err(|e| CliError::Local(anyhow::anyhow!(e)))?
);
}
Format::Human => {
let target =
format_used_target(reply.used_ref.as_deref(), reply.used_selector.as_deref());
println!(
"hover ok tab={} target={target} at=({}, {})",
reply.tab_id, reply.x, reply.y
);
print_dialog_summaries(&reply.dialogs);
}
}
Ok(())
}
// ---------------------------------------------------------------------------
// bsk fill
// ---------------------------------------------------------------------------
+1 -4
View File
@@ -42,7 +42,7 @@ use crate::cli::evaluate::EvaluateArgs;
use crate::cli::get_html::GetHtmlArgs;
use crate::cli::human_loop::RequestHelpArgs;
use crate::cli::install_skill::InstallSkillArgs;
use crate::cli::interaction::{ClickArgs, FillArgs, HoverArgs, PressArgs, SelectArgs};
use crate::cli::interaction::{ClickArgs, FillArgs, PressArgs, SelectArgs};
use crate::cli::navigate::{NavigateCommand, NavigateHistoryArgs, ReloadArgs};
use crate::cli::network::NetworkArgs;
use crate::cli::observe::ObserveArgs;
@@ -164,9 +164,6 @@ pub enum Command {
/// Click a snapshot ref or CSS selector.
Click(ClickArgs),
/// Hover a snapshot ref or CSS selector.
Hover(HoverArgs),
/// Fill an input / textarea / contenteditable.
Fill(FillArgs),
-5
View File
@@ -29,10 +29,6 @@ pub struct ObserveArgs {
/// Soft cap on rendered tokens (~4 chars/token).
#[arg(long = "max-tokens")]
pub max_tokens: Option<u32>,
/// Include conditional surface probe diagnostics in JSON output.
#[arg(long = "debug-surfaces")]
pub debug_surfaces: bool,
}
pub fn dispatch(args: ObserveArgs, format: Format) -> Result<(), CliError> {
@@ -46,7 +42,6 @@ fn run(sock: PathBuf, args: ObserveArgs, format: Format) -> Result<(), CliError>
tab_id: args.tab_id,
max_depth: args.max_depth,
max_tokens: args.max_tokens,
debug_surfaces: args.debug_surfaces,
};
let reply: ObserveResult = call(sock, params)?;
match format {
-1
View File
@@ -242,7 +242,6 @@ pub fn full_handler(status: DaemonStatus, state: Arc<DaemonState>) -> RpcHandler
| Method::ToolNavigateForward
| Method::ToolReload
| Method::ToolClick
| Method::ToolHover
| Method::ToolFill
| Method::ToolPress
| Method::ToolSelect
-1
View File
@@ -93,7 +93,6 @@ fn dispatch(cli: Cli, format: Format) -> Result<(), CliError> {
Command::NavigateForward(args) => cli::navigate::dispatch_navigate_forward(args, format),
Command::Reload(args) => cli::navigate::dispatch_reload(args, format),
Command::Click(args) => cli::interaction::dispatch_click(args, format),
Command::Hover(args) => cli::interaction::dispatch_hover(args, format),
Command::Fill(args) => cli::interaction::dispatch_fill(args, format),
Command::Press(args) => cli::interaction::dispatch_press(args, format),
Command::Select(args) => cli::interaction::dispatch_select(args, format),
@@ -21,6 +21,7 @@ pub enum HarnessId {
Workbuddy,
PiAgent,
Hermes,
KimiCode,
}
impl HarnessId {
@@ -36,6 +37,7 @@ impl HarnessId {
HarnessId::Workbuddy,
HarnessId::PiAgent,
HarnessId::Hermes,
HarnessId::KimiCode,
];
pub fn index(self) -> usize {
@@ -55,6 +57,7 @@ impl HarnessId {
HarnessId::Workbuddy => "workbuddy",
HarnessId::PiAgent => "pi",
HarnessId::Hermes => "hermes",
HarnessId::KimiCode => "kimi-code",
}
}
@@ -71,6 +74,7 @@ impl HarnessId {
HarnessId::Workbuddy => "WorkBuddy",
HarnessId::PiAgent => "Pi",
HarnessId::Hermes => "Hermes Agent",
HarnessId::KimiCode => "Kimi Code",
}
}
@@ -87,6 +91,10 @@ impl HarnessId {
HarnessId::Workbuddy => home.join(".workbuddy").join("skills"),
HarnessId::PiAgent => home.join(".pi").join("agent").join("skills"),
HarnessId::Hermes => hermes_home_for_user_home(home).join("skills"),
// Kimi Code CLI 用户级 skills 目录:`$KIMI_CODE_HOME/skills`,
// 默认 ~/.kimi-code/skills,见
// https://github.com/MoonshotAI/kimi-code/blob/main/docs/en/customization/skills.md
HarnessId::KimiCode => kimi_code_home_for_user_home(home).join("skills"),
}
}
@@ -215,6 +223,23 @@ impl HarnessId {
push("`hermes` on PATH");
}
}
HarnessId::KimiCode => {
let kimi_home = kimi_code_home_for_user_home(home);
if kimi_home.is_dir() {
let label = if std::env::var("KIMI_CODE_HOME")
.ok()
.is_some_and(|value| !value.trim().is_empty())
{
format!("$KIMI_CODE_HOME ({})", kimi_home.display())
} else {
"~/.kimi-code".to_string()
};
push(&label);
}
if command_exists("kimi") {
push("`kimi` on PATH");
}
}
}
let detected = !signals.is_empty();
@@ -265,6 +290,7 @@ pub fn parse_harness_id(raw: &str) -> Result<HarnessId> {
"workbuddy" | "work-buddy" => Ok(HarnessId::Workbuddy),
"pi" | "pi-agent" | "pi_agent" | "piagent" => Ok(HarnessId::PiAgent),
"hermes" | "hermes-agent" | "hermes_agent" | "hermesagent" => Ok(HarnessId::Hermes),
"kimi-code" | "kimi" | "kimicode" | "kimi-cli" | "kimi_cli" => Ok(HarnessId::KimiCode),
other => bail!(
"unknown harness '{other}'; expected one of: {}",
HarnessId::ALL
@@ -280,6 +306,18 @@ pub(crate) fn home_dir() -> Result<PathBuf> {
dirs::home_dir().ok_or_else(|| anyhow::anyhow!("could not resolve home directory"))
}
/// Kimi Code CLI 的数据根目录:`KIMI_CODE_HOME` 设置时用之,否则 `~/.kimi-code`
/// (见 kimi-code 官方 skills 文档:用户级 skills 在 `$KIMI_CODE_HOME/skills`)。
fn kimi_code_home_for_user_home(home: &Path) -> PathBuf {
if let Ok(override_home) = std::env::var("KIMI_CODE_HOME") {
let trimmed = override_home.trim();
if !trimmed.is_empty() {
return PathBuf::from(trimmed);
}
}
home.join(".kimi-code")
}
/// Mirrors [`get_hermes_home()`](https://github.com/NousResearch/hermes-agent/blob/main/hermes_constants.py):
/// `HERMES_HOME` when set, else `%LOCALAPPDATA%\\hermes` on native Windows, else `~/.hermes`.
fn hermes_home_for_user_home(home: &Path) -> PathBuf {
@@ -360,6 +398,12 @@ fn command_exists(name: &str) -> bool {
mod tests {
use super::*;
/// 读写 KIMI_CODE_HOME 环境变量的测试共用的互斥锁(Rust 测试默认并行)。
fn kimi_env_lock() -> &'static std::sync::Mutex<()> {
static LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
&LOCK
}
#[test]
fn parse_aliases() {
assert_eq!(parse_harness_id("codebody").unwrap(), HarnessId::Codebuddy);
@@ -369,6 +413,9 @@ mod tests {
);
assert_eq!(parse_harness_id("pi-agent").unwrap(), HarnessId::PiAgent);
assert_eq!(parse_harness_id("hermes-agent").unwrap(), HarnessId::Hermes);
assert_eq!(parse_harness_id("kimi").unwrap(), HarnessId::KimiCode);
assert_eq!(parse_harness_id("kimi-code").unwrap(), HarnessId::KimiCode);
assert_eq!(parse_harness_id("kimi-cli").unwrap(), HarnessId::KimiCode);
}
#[test]
@@ -424,6 +471,52 @@ mod tests {
HarnessId::Hermes.skills_dir_for_home(home),
PathBuf::from("/home/user/.hermes/skills")
);
assert_eq!(
HarnessId::KimiCode.skills_dir_for_home(home),
PathBuf::from("/home/user/.kimi-code/skills")
);
}
#[test]
fn detects_kimi_code_from_home_layout() {
// 与改动 KIMI_CODE_HOME 的测试互斥,避免并行读写环境变量产生竞争。
let _guard = kimi_env_lock()
.lock()
.unwrap_or_else(|err| err.into_inner());
let tmp = tempfile::TempDir::new().unwrap();
let home = tmp.path();
std::fs::create_dir_all(home.join(".kimi-code")).unwrap();
let report = HarnessId::KimiCode.report_for_home(home);
assert!(report.detected);
assert_eq!(report.skills_dir, home.join(".kimi-code").join("skills"));
assert_eq!(HarnessId::KimiCode.cli_name(), "kimi-code");
assert_eq!(HarnessId::KimiCode.display_name(), "Kimi Code");
}
#[test]
fn kimi_code_skills_dir_honors_kimi_code_home_env() {
let _guard = kimi_env_lock()
.lock()
.unwrap_or_else(|err| err.into_inner());
let tmp = tempfile::TempDir::new().unwrap();
let custom = tmp.path().join("custom-kimi-code");
std::fs::create_dir_all(&custom).unwrap();
let previous = std::env::var("KIMI_CODE_HOME").ok();
// SAFETY: 持有 kimi_env_lock,覆盖所有读写 KIMI_CODE_HOME 的测试。
unsafe {
std::env::set_var("KIMI_CODE_HOME", &custom);
}
let home = Path::new("/home/user");
assert_eq!(
HarnessId::KimiCode.skills_dir_for_home(home),
custom.join("skills")
);
unsafe {
match previous {
Some(value) => std::env::set_var("KIMI_CODE_HOME", value),
None => std::env::remove_var("KIMI_CODE_HOME"),
}
}
}
#[test]
-18
View File
@@ -187,24 +187,6 @@ fn parses_click_count_alias() {
assert_eq!(args.click_count, 2);
}
#[test]
fn parses_hover_with_settle() {
let cli = parse(&[
"bsk",
"hover",
"@e1",
"--session",
"s1",
"--settle",
"300ms",
]);
let Command::Hover(args) = cli.command else {
panic!("expected hover command");
};
assert_eq!(args.target.as_deref(), Some("@e1"));
assert_eq!(args.settle, 300);
}
#[test]
fn rejects_zero_click_count() {
assert!(
-2
View File
@@ -478,7 +478,6 @@ async fn observe_returns_semantic_text_and_ref_count() {
tab_id: 13,
truncated: false,
dialogs: vec![],
debug: None,
})
.unwrap(),
)
@@ -493,7 +492,6 @@ async fn observe_returns_semantic_text_and_ref_count() {
tab_id: None,
max_depth: None,
max_tokens: None,
debug_surfaces: false,
},
)
.await
@@ -1,72 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "HoverParams",
"type": "object",
"required": [
"session_id"
],
"properties": {
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"ref": {
"description": "Optional `@e<N>` ref allocated by the last observation. Mutually exclusive with `selector`.",
"type": [
"string",
"null"
]
},
"selector": {
"type": [
"string",
"null"
]
},
"session_id": {
"type": "string"
},
"settle_ms": {
"description": "Wait after the mouse move so hover-triggered UI can settle.",
"type": [
"integer",
"null"
],
"format": "uint32",
"minimum": 0.0
},
"tab_id": {
"description": "Target tab. Defaults to the Agent Window's active tab.",
"type": [
"integer",
"null"
],
"format": "int64"
},
"timeout_ms": {
"type": [
"integer",
"null"
],
"format": "uint32",
"minimum": 1.0
}
},
"definitions": {
"KeyModifier": {
"description": "Keyboard modifier flags. Multiple flags may be combined; the extension folds them into CDP's bitfield (`alt=1, ctrl=2, meta=4, shift=8`).",
"type": "string",
"enum": [
"alt",
"ctrl",
"meta",
"shift"
]
}
}
}
@@ -1,113 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "HoverResult",
"type": "object",
"required": [
"tab_id",
"x",
"y"
],
"properties": {
"dialogs": {
"type": "array",
"items": {
"$ref": "#/definitions/JavaScriptDialogInfo"
}
},
"tab_id": {
"type": "integer",
"format": "int64"
},
"used_ref": {
"type": [
"string",
"null"
]
},
"used_selector": {
"type": [
"string",
"null"
]
},
"x": {
"description": "Viewport-relative hover coordinates (CSS pixels).",
"type": "number",
"format": "double"
},
"y": {
"type": "number",
"format": "double"
}
},
"definitions": {
"JavaScriptDialogHandledAction": {
"description": "How the extension resolved the dialog so CDP could continue.",
"type": "string",
"enum": [
"accepted",
"dismissed"
]
},
"JavaScriptDialogInfo": {
"description": "One observed + handled JavaScript dialog during a tool call.",
"type": "object",
"required": [
"handled",
"message",
"sequence",
"tab_id",
"type"
],
"properties": {
"default_prompt": {
"type": [
"string",
"null"
]
},
"handled": {
"$ref": "#/definitions/JavaScriptDialogHandledAction"
},
"has_browser_handler": {
"type": [
"boolean",
"null"
]
},
"message": {
"type": "string"
},
"sequence": {
"description": "Monotonic per-tab sequence for ordering within a session.",
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"tab_id": {
"type": "integer",
"format": "int64"
},
"type": {
"$ref": "#/definitions/JavaScriptDialogType"
},
"url": {
"type": [
"string",
"null"
]
}
}
},
"JavaScriptDialogType": {
"description": "Native JS dialog kind reported by CDP.",
"type": "string",
"enum": [
"alert",
"confirm",
"prompt",
"beforeunload"
]
}
}
}
@@ -7,11 +7,6 @@
"session_id"
],
"properties": {
"debug_surfaces": {
"description": "Include implementation diagnostics for conditional surface probes.",
"default": false,
"type": "boolean"
},
"max_depth": {
"description": "Cap the depth of the rendered VOM tree.",
"type": [
@@ -8,16 +8,6 @@
"text"
],
"properties": {
"debug": {
"anyOf": [
{
"$ref": "#/definitions/ObserveDebug"
},
{
"type": "null"
}
]
},
"dialogs": {
"type": "array",
"items": {
@@ -113,73 +103,6 @@
"prompt",
"beforeunload"
]
},
"ObserveDebug": {
"type": "object",
"properties": {
"surface_probes": {
"type": "array",
"items": {
"$ref": "#/definitions/SurfaceProbeDebug"
}
}
}
},
"SurfaceProbeDebug": {
"type": "object",
"required": [
"sub_items",
"trigger_action",
"trigger_backend_node_id"
],
"properties": {
"confidence": {
"type": [
"string",
"null"
]
},
"sub_items": {
"type": "array",
"items": {
"type": "string"
}
},
"trigger_action": {
"type": "string"
},
"trigger_backend_node_id": {
"type": "integer",
"format": "int64"
},
"trigger_point": {
"anyOf": [
{
"$ref": "#/definitions/SurfaceProbePoint"
},
{
"type": "null"
}
]
}
}
},
"SurfaceProbePoint": {
"type": "object",
"required": [
"x",
"y"
],
"properties": {
"x": {
"type": "number",
"format": "double"
},
"y": {
"type": "number",
"format": "double"
}
}
}
}
}
@@ -144,46 +144,6 @@
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
@@ -144,46 +144,6 @@
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
-40
View File
@@ -177,46 +177,6 @@
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
@@ -83,46 +83,6 @@
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
@@ -77,8 +77,6 @@ fn main() {
dump!(ClickParams, "tool_click_params");
dump!(ClickResult, "tool_click_result");
dump!(HoverParams, "tool_hover_params");
dump!(HoverResult, "tool_hover_result");
dump!(FillParams, "tool_fill_params");
dump!(FillResult, "tool_fill_result");
dump!(PressParams, "tool_press_params");
+1 -6
View File
@@ -70,8 +70,6 @@ pub enum Method {
ToolReload,
#[serde(rename = "tool.click")]
ToolClick,
#[serde(rename = "tool.hover")]
ToolHover,
#[serde(rename = "tool.fill")]
ToolFill,
#[serde(rename = "tool.press")]
@@ -163,7 +161,7 @@ impl Method {
// Transient input — no committed browser action, but still page
// input. It must be stopped by pending user interrupts.
Method::ToolHover | Method::ToolObserve => MethodEffect::TransientInput,
Method::ToolObserve => MethodEffect::TransientInput,
// Passive reads — transparent.
// `record_stop` / `record_await` observe / finish a recording
@@ -261,7 +259,6 @@ mod tests {
fn is_mutating_classifies_read_only_tools_as_non_mutating() {
assert!(!Method::ToolTabList.is_mutating());
assert!(!Method::ToolSnapshot.is_mutating());
assert!(!Method::ToolHover.is_mutating());
assert!(!Method::ToolObserve.is_mutating());
assert!(!Method::ToolGetHtml.is_mutating());
assert!(!Method::ToolScreenshot.is_mutating());
@@ -324,7 +321,6 @@ mod tests {
#[test]
fn effect_classifies_observe_as_transient_input() {
assert_eq!(Method::ToolSnapshot.effect(), MethodEffect::PassiveRead);
assert_eq!(Method::ToolHover.effect(), MethodEffect::TransientInput);
assert_eq!(Method::ToolObserve.effect(), MethodEffect::TransientInput);
assert_eq!(Method::ToolClick.effect(), MethodEffect::BrowserMutation);
assert_eq!(Method::Cancel.effect(), MethodEffect::ControlPlane);
@@ -333,7 +329,6 @@ mod tests {
#[test]
fn interrupt_gate_includes_transient_input() {
assert!(!Method::ToolSnapshot.requires_interrupt_gate());
assert!(Method::ToolHover.requires_interrupt_gate());
assert!(Method::ToolObserve.requires_interrupt_gate());
assert!(Method::ToolClick.requires_interrupt_gate());
assert!(!Method::Cancel.requires_interrupt_gate());
@@ -85,52 +85,6 @@ pub struct ClickResult {
pub dialogs: Vec<JavaScriptDialogInfo>,
}
// ---------------------------------------------------------------------------
// hover
// ---------------------------------------------------------------------------
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct HoverParams {
pub session_id: String,
/// Optional `@e<N>` ref allocated by the last observation.
/// Mutually exclusive with `selector`.
#[serde(
rename = "ref",
alias = "ref_",
default,
skip_serializing_if = "Option::is_none"
)]
pub ref_: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub selector: Option<String>,
/// Target tab. Defaults to the Agent Window's active tab.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub tab_id: Option<i64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub modifiers: Option<Vec<KeyModifier>>,
/// Wait after the mouse move so hover-triggered UI can settle.
#[serde(default, skip_serializing_if = "Option::is_none")]
#[schemars(range(min = 0))]
pub settle_ms: Option<u32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
#[schemars(range(min = 1))]
pub timeout_ms: Option<u32>,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct HoverResult {
pub tab_id: i64,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub used_ref: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub used_selector: Option<String>,
/// Viewport-relative hover coordinates (CSS pixels).
pub x: f64,
pub y: f64,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub dialogs: Vec<JavaScriptDialogInfo>,
}
// ---------------------------------------------------------------------------
// fill
// ---------------------------------------------------------------------------
@@ -307,24 +261,6 @@ mod tests {
assert_eq!(p.ref_.as_deref(), Some("e1"));
}
#[test]
fn hover_params_serialise_ref_field_name() {
let p = HoverParams {
session_id: "abcd".into(),
ref_: Some("@e3".into()),
selector: None,
tab_id: Some(42),
modifiers: Some(vec![KeyModifier::Shift]),
settle_ms: Some(200),
timeout_ms: Some(5_000),
};
let v = serde_json::to_value(&p).unwrap();
assert_eq!(v.get("ref").and_then(|v| v.as_str()), Some("@e3"));
assert!(v.get("ref_").is_none());
let round: HoverParams = serde_json::from_value(v).unwrap();
assert_eq!(round, p);
}
#[test]
fn modifiers_render_as_lowercase_strings() {
let v = serde_json::to_value(KeyModifier::Ctrl).unwrap();
@@ -74,32 +74,6 @@ pub struct ObserveParams {
/// best-effort heuristic based on character count).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_tokens: Option<u32>,
/// Include implementation diagnostics for conditional surface probes.
#[serde(default)]
pub debug_surfaces: bool,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct SurfaceProbePoint {
pub x: f64,
pub y: f64,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct SurfaceProbeDebug {
pub trigger_backend_node_id: i64,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub trigger_point: Option<SurfaceProbePoint>,
pub trigger_action: String,
pub sub_items: Vec<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub confidence: Option<String>,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct ObserveDebug {
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub surface_probes: Vec<SurfaceProbeDebug>,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
@@ -118,8 +92,6 @@ pub struct ObserveResult {
pub truncated: bool,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub dialogs: Vec<JavaScriptDialogInfo>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub debug: Option<ObserveDebug>,
}
// ---------------------------------------------------------------------------
@@ -308,7 +280,6 @@ mod tests {
tab_id: 42,
truncated: false,
dialogs: Vec::new(),
debug: None,
};
let v = serde_json::to_value(&r).unwrap();
assert_eq!(v.get("ref_count").and_then(|v| v.as_u64()), Some(1));
-5
View File
@@ -94,11 +94,6 @@ pub enum Step {
common: StepCommon,
target: TargetDescriptor,
},
Hover {
#[serde(flatten)]
common: StepCommon,
target: TargetDescriptor,
},
Fill {
#[serde(flatten)]
common: StepCommon,
@@ -85,7 +85,7 @@
"notificationTitle": "BrowserSkill:Agent 需要你的帮助"
},
"recordOverlay": {
"recording": "正在录制用户操作",
"recording": "正在录制你的操作",
"finish": "结束"
}
}
+1 -1
View File
@@ -404,7 +404,7 @@ describe("renderVom single-layer page", () => {
],
});
expect(out.text).toContain('@e1 button "Products" [hover first: Shoes | Bags | Accessories]');
expect(out.text).toContain('@e1 button "Products" [hover: Shoes | Bags | Accessories]');
expect(out.refs).toEqual([{ ref: "e1", backendNodeId: 2 }]);
});
+1 -1
View File
@@ -1,4 +1,4 @@
export { applyVomInteractionRecovery, isVomReferenceNode, renderVom } from "./render";
export { renderVom } from "./render";
export type {
ActiveScopeBlock,
BlockingLayer,
+11 -13
View File
@@ -197,7 +197,7 @@ function shouldRender(node: VomNode): boolean {
return cleaned(node.name) !== undefined;
}
export function isVomReferenceNode(node: VomNode): boolean {
function shouldReference(node: VomNode): boolean {
return INTERACTIVE_ROLES.has(normalizedRole(node)) || isUrlBearingStructuralAction(node);
}
@@ -342,7 +342,7 @@ function isAncestorOf(
return false;
}
export function applyVomInteractionRecovery(nodes: VomNode[]): VomNode[] {
function applyRecovery(nodes: VomNode[]): VomNode[] {
const parentMap = buildParentMap(nodes);
const children = buildChildren(nodes);
const recoveredIds = new Set<number>();
@@ -350,7 +350,7 @@ export function applyVomInteractionRecovery(nodes: VomNode[]): VomNode[] {
const recoveredNameById = new Map<number, string>();
for (const node of nodes) {
if (isVomReferenceNode(node)) continue;
if (shouldReference(node)) continue;
const role = recoveredRole(node, children);
if (!role || !passesRecoveryGuards(node, children)) continue;
recoveredIds.add(node.id);
@@ -397,7 +397,7 @@ function buildChildren(nodes: VomNode[]): Map<number | null, VomNode[]> {
function duplicateReferenceNames(nodes: VomNode[]): Set<string> {
const counts = new Map<string, number>();
for (const node of nodes) {
if (!isVomReferenceNode(node)) continue;
if (!shouldReference(node)) continue;
const name = cleaned(node.name);
if (!name) continue;
const key = normalizeContextKey(name);
@@ -427,7 +427,7 @@ function contextLabel(node: VomNode, targetName: string): string | undefined {
}
function weakLabelContext(node: VomNode, targetName: string): string | undefined {
if (isVomReferenceNode(node)) return undefined;
if (shouldReference(node)) return undefined;
return weakContextText(
cleaned(node.name) ?? cleaned(node.text) ?? cleaned(node.nearbyText),
targetName,
@@ -509,7 +509,7 @@ function collectWeakLabelsFromSubtree(
state: RenderState,
labels: string[],
): void {
if (isVomReferenceNode(node)) return;
if (shouldReference(node)) return;
const label = weakLabelContext(node, nodeName);
if (label) {
labels.push(label);
@@ -637,9 +637,7 @@ function renderNodeLine(
if (surface && surface.subItems.length > 0) {
const items = surface.subItems.slice(0, MAX_SURFACE_ITEMS).join(" | ");
const suffix = surface.subItems.length > MAX_SURFACE_ITEMS ? " | …" : "";
const action =
surface.triggerAction === "hover" ? "hover first" : `${surface.triggerAction} first`;
line += ` [${action}: ${items}${suffix}]`;
line += ` [${surface.triggerAction}: ${items}${suffix}]`;
}
return line;
@@ -666,7 +664,7 @@ function descendantTextCoveredByName(node: VomNode, state: RenderState): boolean
const texts: string[] = [];
while (stack.length > 0) {
const child = stack.pop() as VomNode;
if (isVomReferenceNode(child)) return false;
if (shouldReference(child)) return false;
const text = cleaned(child.name) ?? cleaned(child.text) ?? cleaned(child.value);
if (text) texts.push(text);
stack.push(...(state.children.get(child.id) ?? []));
@@ -750,7 +748,7 @@ function renderTree(
continue;
}
const ref = isVomReferenceNode(node) ? `e${state.nextRef}` : undefined;
const ref = shouldReference(node) ? `e${state.nextRef}` : undefined;
const context = ref ? handleContext(node, state) : [];
const surface = state.surfaceMap.get(node.id);
const line = renderNodeLine(node, depth, ref, context, surface);
@@ -915,7 +913,7 @@ function applyActiveRegionPolicy(nodes: VomNode[], scene: VomScene): VomNode[] {
const blockedRoots = new Set<number>();
for (const target of nodes) {
if (!isVomReferenceNode(target)) continue;
if (!shouldReference(target)) continue;
const blocked = candidates.some((candidate) =>
isBlockedByRegion(target, candidate.node, parentMap, candidate.viewportCoverage),
);
@@ -970,7 +968,7 @@ function renderDoubleLayer(scene: VomScene, layer: BlockingLayer, options: VomOp
}
export function renderVom(scene: VomScene, options: VomOptions = {}): VomResult {
const nodes = applyVomInteractionRecovery(scene.nodes);
const nodes = applyRecovery(scene.nodes);
const renderScene = { ...scene, nodes };
const layer = detectBlockingLayer(nodes, scene.viewport);
if (layer) return renderDoubleLayer(renderScene, layer, options);
+7 -11
View File
@@ -67,25 +67,22 @@ Write operations only affect tabs in the **Agent Window** (or tabs you **borrowe
```
bsk navigate <url> --session <id>
bsk observe --session <id> → primary semantic VOM view; reveals hover/focus surfaces
bsk snapshot --session <id> → static aria tree fallback when VOM is insufficient
bsk hover @e3 --session <id> → reveal hover-triggered menus before re-observing/clicking
bsk click @e4 --session <id> → or bsk fill, bsk select, bsk press
bsk observe --session <id> → again after navigation / DOM change
bsk snapshot --session <id> → aria tree with @e1, @e2, … refs
bsk observe --session <id> → semantic VOM view; may reveal hover/focus surfaces
bsk click @e3 --session <id> → or bsk fill, bsk select, bsk press
bsk snapshot --session <id> → again after navigation / DOM change
```
**Refs invalidate after navigation** — always re-snapshot before clicking, filling, or selecting on a new page.
Prefer `@eN` refs from the latest snapshot over raw CSS selectors. Use `--ref` / `--selector` when ambiguous (`bsk click --help`).
When VOM renders `[hover first: …]` on an element, the listed items are not currently clickable refs. Run `bsk hover <that-ref> --session <id>`, then immediately run `bsk snapshot` or `bsk observe` again and click the newly visible menu item ref. Do not click the trigger itself unless the user explicitly wants the trigger action.
## Observation priority
Start with `bsk observe` to understand page structure, text, controls, element refs, and conditional hover/focus surfaces. Use `bsk snapshot` only when you need the stricter static accessibility tree or VOM is insufficient. Only escalate to raw HTML or screenshots when the latest observation cannot answer the question:
Start with `bsk snapshot` to understand page structure, text, controls, and element refs. Use `bsk observe` when semantic VOM output or conditional hover/focus surfaces would materially help. Only escalate to raw HTML or screenshots when the latest observation cannot answer the question:
1. `bsk observe` — primary semantic VOM observation; may run bounded perception probes such as hover-surface discovery
2. `bsk snapshot` — strict static accessibility tree fallback
1. `bsk snapshot` — strict static page understanding and interaction planning
2. `bsk observe` — semantic VOM observation; may run bounded perception probes such as hover-surface discovery
3. `bsk get-html` — when hidden DOM, metadata, or markup details are required
4. `bsk screenshot` — when visual layout, canvas/image content, or styling cannot be inferred from the observation. Use `--ref @eN` (from the latest snapshot/observe) to crop to one element; omit `--ref` for the full visible tab.
@@ -189,7 +186,6 @@ bsk emulate --session <id> --off
| Command | Summary |
|---------|---------|
| `bsk click <ref-or-selector>` | Click element (`--button`, `--click-count`, `--modifiers`) |
| `bsk hover <ref-or-selector>` | Move the mouse to an element and wait for hover UI to settle (`--settle`, `--modifiers`) |
| `bsk fill <ref-or-selector> --value <text>` | Clear and type into input |
| `bsk select <ref-or-selector> --value <v>` | Set `<select>` option(s) by `value` (repeat `--value` for multi-select) |
| `bsk press <key>` | Key/combo (`Enter`, `Ctrl+A`, …; optional `--ref` to focus first) |