mirror of
https://github.com/Tencent/BrowserSkill.git
synced 2026-09-10 13:01:23 +08:00
feat(packages): add dsh-plugin-browserskill — DeepSeek Harness tools for bsk
Add a new monorepo package that registers BrowserSkill (bsk) browser automation as model-visible tools in DeepSeek Harness (dsh): - 11 tools mapping to bsk CLI --json commands: session start/stop/list, navigate, snapshot, observe, click, fill, press, screenshot, emulate - multi-session support: optional session arg with a current-session pointer, session id echoed in every result, configurable concurrency cap (default 5), and dispose-time cleanup of plugin-started sessions - cancellation: exec.signal aborts kill the underlying bsk child process - terminal-style UI cards (command line as the call card, output as the result card); screenshots inline the PNG through the host attachment store when the model route accepts image input, else return the path - config: bskPath / defaultTimeoutMs / maxSessions via Schemastery schema - install guidance when the bsk binary is missing; bsk JSON error envelopes (code/message/hint) surfaced to the model - unit tests mock the bsk runner (no real browser required); the root lint script now also runs the package's typecheck + tests so the existing CI frontend job covers the new package (a dedicated CI job diff is provided in the PR; the pushing token lacks workflow scope)
This commit is contained in:
@@ -0,0 +1 @@
|
||||
lib/
|
||||
@@ -0,0 +1,88 @@
|
||||
# dsh-plugin-browserskill
|
||||
|
||||
A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (dsh) tool plugin that exposes
|
||||
[BrowserSkill](https://github.com/Tencent/BrowserSkill) (`bsk`) browser automation to the model.
|
||||
|
||||
Each tool maps to one `bsk <cmd> --json` invocation: the plugin spawns the bsk CLI, parses its
|
||||
structured JSON output, and returns a canonical typed value. The bsk daemon, browser, and Chrome
|
||||
extension keep owning the actual browser control — this package is a thin, well-typed bridge.
|
||||
|
||||
## Tools
|
||||
|
||||
| Tool | bsk command | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `browser_session_start` | `bsk session start` | Open an Agent Window session; optional initial URL, window size, and device emulation preset. Returns the session id and makes it the current session. |
|
||||
| `browser_session_stop` | `bsk session stop` | Stop a session (the current one by default) and close its Agent Window. |
|
||||
| `browser_session_list` | `bsk session list` | List daemon-side sessions, marking the plugin's current session. |
|
||||
| `browser_navigate` | `bsk navigate` | Navigate the active tab, with `waitUntil` / timeout control. |
|
||||
| `browser_snapshot` | `bsk snapshot` | Indented aria-tree snapshot with `@eN` refs for interaction tools. |
|
||||
| `browser_observe` | `bsk observe` | Semantic VOM observation (read-only) with `@eN` refs. |
|
||||
| `browser_click` | `bsk click` | Click a snapshot ref or CSS selector (button / click-count options). |
|
||||
| `browser_fill` | `bsk fill` | Fill an input / textarea / contenteditable (clears first by default). |
|
||||
| `browser_press` | `bsk press` | Dispatch a key or combo, optionally focusing a target first. |
|
||||
| `browser_screenshot` | `bsk screenshot` | PNG capture of the tab or a ref-cropped element; inlines the image when the deployment supports image input, otherwise returns a file path. |
|
||||
| `browser_emulate` | `bsk emulate` | Apply or clear mobile device emulation on the active tab. |
|
||||
|
||||
## Multi-session model
|
||||
|
||||
One agent conversation can drive several browser sessions at once:
|
||||
|
||||
- `browser_session_start` returns the session id and makes it the **current session**.
|
||||
- Every operation tool accepts an optional `session` argument. When omitted, the call acts on the
|
||||
current session (the one most recently started or used); when given, that session becomes current.
|
||||
- Every tool result echoes the session it actually acted on, so the model never has to guess.
|
||||
- The number of concurrent sessions started through the plugin is capped (`maxSessions`, default 5).
|
||||
- Unloading the plugin stops every session it started and kills in-flight bsk processes.
|
||||
|
||||
## Installation
|
||||
|
||||
The plugin follows the standard dsh bundle layout (`dsh.bundle` manifest + `cordis.patch.yml`):
|
||||
|
||||
```sh
|
||||
dsh plugin --profile <name> add <path-or-spec-of-this-package>
|
||||
dsh --profile <name>
|
||||
```
|
||||
|
||||
Prerequisite: the `bsk` CLI must be installed and on `PATH`, and the BrowserSkill Chrome extension
|
||||
must be connected — see the [BrowserSkill README](https://github.com/Tencent/BrowserSkill). When bsk
|
||||
is missing, tool calls fail with install guidance instead of a bare spawn error.
|
||||
|
||||
## Configuration
|
||||
|
||||
All fields are optional and validated through the plugin's Schemastery `Config`:
|
||||
|
||||
```yaml
|
||||
# cordis.patch.yml override example
|
||||
- insert:
|
||||
- id: browserskill
|
||||
name: dsh-plugin-browserskill
|
||||
config:
|
||||
bskPath: bsk # path to the bsk binary (default: resolve from PATH)
|
||||
defaultTimeoutMs: 120000
|
||||
maxSessions: 5
|
||||
```
|
||||
|
||||
## Behavior notes
|
||||
|
||||
- **Cancellation**: aborting a tool call (`exec.signal`) kills the underlying bsk child process,
|
||||
matching BrowserSkill's cooperative tool-cancellation model.
|
||||
- **UI cards**: calls render as terminal cards (command line as title, output as the completed
|
||||
card). Screenshots additionally attach the image itself when the host mounts an attachment store
|
||||
and the active model route declares image input; otherwise the PNG path is returned.
|
||||
- **Errors**: non-zero bsk exits surface the CLI's JSON error envelope (`code`, `message`, `hint`)
|
||||
so the model gets the daemon's actionable guidance.
|
||||
- **Long-running work** (e.g. `bsk record`) is not backgrounded via `ctx.jobs` yet — tracked as a
|
||||
follow-up.
|
||||
|
||||
## Development
|
||||
|
||||
```sh
|
||||
pnpm install
|
||||
pnpm --filter dsh-plugin-browserskill typecheck
|
||||
pnpm --filter dsh-plugin-browserskill test # unit tests mock bsk; no browser needed
|
||||
pnpm --filter dsh-plugin-browserskill build # tsdown -> lib/
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@@ -0,0 +1,3 @@
|
||||
- insert:
|
||||
- id: browserskill
|
||||
name: dsh-plugin-browserskill
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"name": "dsh-plugin-browserskill",
|
||||
"description": "DeepSeek Harness tool plugin that exposes BrowserSkill (bsk) browser automation to the model",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.mjs",
|
||||
"types": "lib/index.d.mts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/index.d.mts",
|
||||
"default": "./lib/index.mjs"
|
||||
},
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib",
|
||||
"cordis.patch.yml"
|
||||
],
|
||||
"dsh": {
|
||||
"bundle": {
|
||||
"patch": "./cordis.patch.yml"
|
||||
}
|
||||
},
|
||||
"scripts": {
|
||||
"build": "tsdown",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"test": "vitest run"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/cordis": "^4.0.1",
|
||||
"@deepseek-ai/dsh-tools": "^0.1.0-rc.6"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@deepseek-ai/dsh-attachment": {
|
||||
"optional": true
|
||||
},
|
||||
"@deepseek-ai/dsh-llm": {
|
||||
"optional": true
|
||||
}
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/cordis": "^4.0.1",
|
||||
"@deepseek-ai/dsh-agent": "^0.1.0-rc.6",
|
||||
"@deepseek-ai/dsh-attachment": "^0.1.0-rc.6",
|
||||
"@deepseek-ai/dsh-llm": "^0.1.0-rc.6",
|
||||
"@deepseek-ai/dsh-tools": "^0.1.0-rc.6",
|
||||
"@deepseek-ai/schemastery": "^3.18.1",
|
||||
"@types/node": "^22.10.0",
|
||||
"tsdown": "^0.22.14",
|
||||
"typescript": "^5.8.0",
|
||||
"vitest": "^4.1.6"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
/**
|
||||
* Screenshot delivery. The canonical tool value stays plain JSON (a PNG file
|
||||
* path plus pixel metadata); when the host mounts a durable attachment store
|
||||
* AND the calling route declares image input, the PNG bytes are additionally
|
||||
* committed through `ctx.attachments` so the render step can attach the image
|
||||
* itself to the tool result. Any uncertainty (no store, unknown route,
|
||||
* text-only model) falls back to the path-only form.
|
||||
*/
|
||||
|
||||
import type { Context } from "@deepseek-ai/cordis";
|
||||
import type { ImageAttachmentRef } from "@deepseek-ai/dsh-attachment";
|
||||
import type { ToolExecution } from "@deepseek-ai/dsh-tools";
|
||||
|
||||
/** Structural view of the attachment seam (avoids a runtime dependency). */
|
||||
interface AttachmentLike {
|
||||
imageLimits: {
|
||||
mediaTypes: readonly string[];
|
||||
maxImageBytes: number;
|
||||
maxMessageImageBytes: number;
|
||||
};
|
||||
saveImage(input: {
|
||||
data: Uint8Array;
|
||||
mediaType: string;
|
||||
name?: string;
|
||||
}): Promise<ImageAttachmentRef>;
|
||||
}
|
||||
|
||||
interface LlmLike {
|
||||
resolveModelInfo(
|
||||
provider: string,
|
||||
model: string,
|
||||
signal?: AbortSignal,
|
||||
): Promise<{ inputModalities?: readonly string[] }>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Commit screenshot bytes to the host attachment store when the composition
|
||||
* supports durable images on the current model route.
|
||||
* @returns the durable reference, or undefined to stay in path-only mode.
|
||||
*/
|
||||
export async function trySaveScreenshot(
|
||||
ctx: Context,
|
||||
exec: ToolExecution,
|
||||
data: Uint8Array,
|
||||
name: string,
|
||||
): Promise<ImageAttachmentRef | undefined> {
|
||||
const attachments = ctx.get("attachments") as AttachmentLike | undefined;
|
||||
if (attachments === undefined) return undefined;
|
||||
if (!attachments.imageLimits.mediaTypes.includes("image/png")) return undefined;
|
||||
if (
|
||||
data.byteLength >
|
||||
Math.min(attachments.imageLimits.maxImageBytes, attachments.imageLimits.maxMessageImageBytes)
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
if (!(await isImageCapableRoute(ctx, exec))) return undefined;
|
||||
try {
|
||||
return await attachments.saveImage({ data, mediaType: "image/png", name });
|
||||
} catch {
|
||||
// A store hiccup must not fail the tool call; the PNG path still works.
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort check that the calling route's model accepts image input,
|
||||
* mirroring dsh-tool-fs `read_image`. Unknown route / missing llm service
|
||||
* answers false (refuse to attach) so history never gains an image block a
|
||||
* text-only adapter cannot replay.
|
||||
*/
|
||||
async function isImageCapableRoute(ctx: Context, exec: ToolExecution): Promise<boolean> {
|
||||
const llm = ctx.get("llm") as LlmLike | undefined;
|
||||
const provider =
|
||||
exec.agent?.session.requestHeader()?.config.provider ?? exec.agent?.options.provider;
|
||||
const model = exec.agent?.session.requestHeader()?.config.model ?? exec.agent?.options.model;
|
||||
if (llm === undefined || provider === undefined || model === undefined) return false;
|
||||
try {
|
||||
const info = await llm.resolveModelInfo(provider, model, exec.signal);
|
||||
return info.inputModalities?.includes("image") ?? false;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
/**
|
||||
* dsh-plugin-browserskill — a DeepSeek Harness bundle that registers
|
||||
* BrowserSkill (`bsk`) browser automation as model-visible tools.
|
||||
*
|
||||
* Each tool spawns `bsk <cmd> --json`, parses the structured output, and
|
||||
* returns a canonical JSON value. The plugin tracks the sessions it starts so
|
||||
* one agent conversation can drive several browsers at once; unloading the
|
||||
* plugin stops every tracked session and kills any in-flight bsk children.
|
||||
*
|
||||
* @module dsh-plugin-browserskill
|
||||
*/
|
||||
|
||||
import type { Context } from "@deepseek-ai/cordis";
|
||||
import Schema from "@deepseek-ai/schemastery";
|
||||
import { createBskRunner } from "./runner";
|
||||
import { SessionRegistry } from "./sessions";
|
||||
import { type PluginConfig, registerTools } from "./tools";
|
||||
|
||||
export const name = "dsh-plugin-browserskill";
|
||||
export const inject = ["tools"];
|
||||
|
||||
/** Runtime configuration schema (validated and defaulted by Cordis). */
|
||||
export const Config = Schema.object({
|
||||
bskPath: Schema.string()
|
||||
.default("bsk")
|
||||
.description("Path to the bsk CLI binary (defaults to resolving `bsk` from PATH)."),
|
||||
defaultTimeoutMs: Schema.number()
|
||||
.default(120_000)
|
||||
.description("Default per-command timeout in milliseconds."),
|
||||
maxSessions: Schema.number()
|
||||
.default(5)
|
||||
.description("Maximum number of concurrent browser sessions started through this plugin."),
|
||||
});
|
||||
|
||||
export type Config = PluginConfig;
|
||||
|
||||
export function apply(ctx: Context, config: Partial<PluginConfig> = {}): void {
|
||||
const resolved: PluginConfig = {
|
||||
bskPath: config.bskPath ?? "bsk",
|
||||
defaultTimeoutMs: config.defaultTimeoutMs ?? 120_000,
|
||||
maxSessions: config.maxSessions ?? 5,
|
||||
};
|
||||
const runner = createBskRunner(resolved.bskPath);
|
||||
const registry = new SessionRegistry(resolved.maxSessions);
|
||||
|
||||
registerTools({ ctx, runner, registry, config: resolved });
|
||||
|
||||
// Non-blocking install probe: warn early when bsk is missing instead of
|
||||
// failing the first tool call with a bare spawn error.
|
||||
runner.run(["status"], { timeoutMs: 10_000 }).then(
|
||||
() => {},
|
||||
(error: unknown) => {
|
||||
const detail = error instanceof Error ? error.message : String(error);
|
||||
console.warn(
|
||||
`[${name}] bsk probe failed (${detail}); browser tools will report install guidance until the bsk CLI is available`,
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
// Unload cleanup: kill in-flight children, then stop every session this
|
||||
// plugin started (best effort; sessions started outside the plugin are
|
||||
// left running).
|
||||
ctx.effect(() => {
|
||||
return () => {
|
||||
runner.killAll();
|
||||
const stops = registry
|
||||
.list()
|
||||
.map((session) =>
|
||||
runner.run(["session", "stop", session.sessionId], { timeoutMs: 15_000 }).catch(() => {}),
|
||||
);
|
||||
return Promise.all(stops).then(() => {});
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
export type { BskRunner, BskRunResult, SpawnImpl } from "./runner";
|
||||
export { BskError, createBskRunner } from "./runner";
|
||||
export { SessionRegistry } from "./sessions";
|
||||
export type { PluginConfig, ToolDeps } from "./tools";
|
||||
export { registerTools } from "./tools";
|
||||
@@ -0,0 +1,194 @@
|
||||
/**
|
||||
* Process runner for the `bsk` CLI. Every model-facing tool in this plugin
|
||||
* maps to one `bsk <cmd> --json` invocation: spawn the child, capture
|
||||
* stdout/stderr, honor the dsh cancellation signal by killing the child, and
|
||||
* map the CLI's JSON error envelope onto a thrown `BskError`.
|
||||
*/
|
||||
|
||||
import { type ChildProcess, spawn } from "node:child_process";
|
||||
|
||||
/** Shape of the JSON error envelope `bsk --json` prints on failure. */
|
||||
export interface BskErrorBody {
|
||||
code?: string;
|
||||
message?: string;
|
||||
hint?: string;
|
||||
exit_code?: number;
|
||||
data?: unknown;
|
||||
}
|
||||
|
||||
/** A failed `bsk` invocation (non-zero exit, timeout, or spawn failure). */
|
||||
export class BskError extends Error {
|
||||
readonly code?: string;
|
||||
readonly hint?: string;
|
||||
readonly exitCode?: number | null;
|
||||
readonly timedOut: boolean;
|
||||
|
||||
constructor(
|
||||
message: string,
|
||||
options: { code?: string; hint?: string; exitCode?: number | null; timedOut?: boolean } = {},
|
||||
) {
|
||||
super(message);
|
||||
this.name = "BskError";
|
||||
this.code = options.code;
|
||||
this.hint = options.hint;
|
||||
this.exitCode = options.exitCode;
|
||||
this.timedOut = options.timedOut ?? false;
|
||||
}
|
||||
}
|
||||
|
||||
export interface BskRunResult {
|
||||
/** Process exit code, or null when killed by a signal / never started. */
|
||||
code: number | null;
|
||||
stdout: string;
|
||||
stderr: string;
|
||||
timedOut: boolean;
|
||||
aborted: boolean;
|
||||
}
|
||||
|
||||
export interface BskRunOptions {
|
||||
signal?: AbortSignal;
|
||||
timeoutMs?: number;
|
||||
}
|
||||
|
||||
/** Minimal spawn signature so tests can substitute a fake child process. */
|
||||
export type SpawnImpl = (command: string, args: string[]) => ChildProcess;
|
||||
|
||||
export interface BskRunner {
|
||||
/** Run `bsk <args...> --json` and collect its output. */
|
||||
run(args: string[], options?: BskRunOptions): Promise<BskRunResult>;
|
||||
/** Kill every in-flight child (used when the plugin unloads). */
|
||||
killAll(): void;
|
||||
}
|
||||
|
||||
const KILL_GRACE_MS = 2000;
|
||||
|
||||
export function createBskRunner(bskPath: string, spawnImpl: SpawnImpl = spawn): BskRunner {
|
||||
const live = new Set<ChildProcess>();
|
||||
|
||||
function killChild(child: ChildProcess): void {
|
||||
if (child.exitCode !== null || child.signalCode !== null) return;
|
||||
child.kill("SIGTERM");
|
||||
const force = setTimeout(() => {
|
||||
if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL");
|
||||
}, KILL_GRACE_MS);
|
||||
force.unref();
|
||||
}
|
||||
|
||||
return {
|
||||
run(args, options = {}) {
|
||||
return new Promise<BskRunResult>((resolve, reject) => {
|
||||
let child: ChildProcess;
|
||||
try {
|
||||
child = spawnImpl(bskPath, [...args, "--json"]);
|
||||
} catch (error) {
|
||||
reject(error);
|
||||
return;
|
||||
}
|
||||
live.add(child);
|
||||
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
let timedOut = false;
|
||||
let aborted = false;
|
||||
child.stdout?.on("data", (chunk: Buffer | string) => {
|
||||
stdout += chunk;
|
||||
});
|
||||
child.stderr?.on("data", (chunk: Buffer | string) => {
|
||||
stderr += chunk;
|
||||
});
|
||||
|
||||
const timeoutMs = options.timeoutMs;
|
||||
const timer =
|
||||
timeoutMs !== undefined && timeoutMs > 0
|
||||
? setTimeout(() => {
|
||||
timedOut = true;
|
||||
killChild(child);
|
||||
}, timeoutMs)
|
||||
: undefined;
|
||||
timer?.unref();
|
||||
|
||||
const onAbort = () => {
|
||||
aborted = true;
|
||||
killChild(child);
|
||||
};
|
||||
if (options.signal?.aborted) {
|
||||
onAbort();
|
||||
} else {
|
||||
options.signal?.addEventListener("abort", onAbort, { once: true });
|
||||
}
|
||||
|
||||
const settle = () => {
|
||||
if (timer !== undefined) clearTimeout(timer);
|
||||
options.signal?.removeEventListener("abort", onAbort);
|
||||
live.delete(child);
|
||||
};
|
||||
|
||||
child.on("error", (error) => {
|
||||
settle();
|
||||
reject(error);
|
||||
});
|
||||
child.on("close", (code) => {
|
||||
settle();
|
||||
resolve({ code, stdout, stderr, timedOut, aborted });
|
||||
});
|
||||
});
|
||||
},
|
||||
killAll() {
|
||||
for (const child of live) killChild(child);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** True when the spawn failure means the bsk binary itself is missing. */
|
||||
export function isCommandNotFound(error: unknown): boolean {
|
||||
return (
|
||||
typeof error === "object" &&
|
||||
error !== null &&
|
||||
"code" in error &&
|
||||
(error as { code?: unknown }).code === "ENOENT"
|
||||
);
|
||||
}
|
||||
|
||||
/** Install guidance shown when the bsk CLI cannot be spawned. */
|
||||
export function bskInstallMessage(bskPath: string): string {
|
||||
return (
|
||||
`the bsk CLI ("${bskPath}") was not found. BrowserSkill must be installed and on PATH ` +
|
||||
"for browser tools to work — install it from https://github.com/Tencent/BrowserSkill " +
|
||||
"(see the README install script or `cargo install`), then retry."
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Interpret one finished run: throw `BskError` on timeout / non-zero exit
|
||||
* (parsing the CLI's JSON error envelope when present), otherwise parse and
|
||||
* return the stdout JSON payload.
|
||||
*/
|
||||
export function parseBskJson(result: BskRunResult, commandLabel: string): unknown {
|
||||
if (result.timedOut) {
|
||||
throw new BskError(`bsk ${commandLabel} timed out`, { timedOut: true });
|
||||
}
|
||||
const body = result.stdout.trim();
|
||||
if (result.code !== 0) {
|
||||
let parsed: BskErrorBody | undefined;
|
||||
try {
|
||||
parsed = JSON.parse(body) as BskErrorBody;
|
||||
} catch {
|
||||
parsed = undefined;
|
||||
}
|
||||
const message =
|
||||
parsed?.message ?? (result.stderr.trim() || body || `bsk ${commandLabel} failed`);
|
||||
throw new BskError(`bsk ${commandLabel} failed: ${message}`, {
|
||||
code: parsed?.code,
|
||||
hint: parsed?.hint,
|
||||
exitCode: result.code,
|
||||
});
|
||||
}
|
||||
try {
|
||||
return JSON.parse(body);
|
||||
} catch {
|
||||
throw new BskError(
|
||||
`bsk ${commandLabel} did not produce JSON output: ${body.slice(0, 200) || "(empty)"}`,
|
||||
{ exitCode: result.code },
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
/**
|
||||
* Tracks the bsk sessions started through this plugin and the "current"
|
||||
* session pointer used when a tool call omits its optional `session` arg.
|
||||
* The pointer moves to whatever session was most recently started, stopped,
|
||||
* or operated on, so a model can run several browsers side by side without
|
||||
* repeating the id on every call.
|
||||
*/
|
||||
|
||||
export interface TrackedSession {
|
||||
sessionId: string;
|
||||
browserInstanceId?: string;
|
||||
startedAtMs: number;
|
||||
}
|
||||
|
||||
export class SessionRegistry {
|
||||
private readonly sessions = new Map<string, TrackedSession>();
|
||||
private currentId: string | undefined;
|
||||
|
||||
constructor(private readonly maxSessions: number) {}
|
||||
|
||||
/**
|
||||
* Register a freshly started session and make it current.
|
||||
* @throws when the configured concurrency cap is already reached.
|
||||
*/
|
||||
add(session: TrackedSession): void {
|
||||
if (!this.sessions.has(session.sessionId) && this.sessions.size >= this.maxSessions) {
|
||||
throw new Error(
|
||||
`session limit reached (${this.maxSessions} concurrent sessions); ` +
|
||||
"stop one with browser_session_stop before starting another",
|
||||
);
|
||||
}
|
||||
this.sessions.set(session.sessionId, session);
|
||||
this.currentId = session.sessionId;
|
||||
}
|
||||
|
||||
/** Forget a stopped session; falls back to the most recent remaining one. */
|
||||
remove(sessionId: string): void {
|
||||
this.sessions.delete(sessionId);
|
||||
if (this.currentId === sessionId) {
|
||||
const rest = [...this.sessions.values()];
|
||||
this.currentId = rest.length > 0 ? rest[rest.length - 1].sessionId : undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/** Mark a session as most recently used. Unknown ids are adopted. */
|
||||
touch(sessionId: string): void {
|
||||
const existing = this.sessions.get(sessionId);
|
||||
if (existing !== undefined) {
|
||||
// Re-insert to refresh recency order.
|
||||
this.sessions.delete(sessionId);
|
||||
this.sessions.set(sessionId, existing);
|
||||
} else {
|
||||
this.sessions.set(sessionId, { sessionId, startedAtMs: Date.now() });
|
||||
}
|
||||
this.currentId = sessionId;
|
||||
}
|
||||
|
||||
/** The current session id, if any. */
|
||||
current(): string | undefined {
|
||||
return this.currentId;
|
||||
}
|
||||
|
||||
/** Tracked sessions in least- to most-recently-used order. */
|
||||
list(): TrackedSession[] {
|
||||
return [...this.sessions.values()];
|
||||
}
|
||||
|
||||
size(): number {
|
||||
return this.sessions.size;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the session a tool call acts on: an explicit `session` argument
|
||||
* wins (and becomes current); otherwise fall back to the current session.
|
||||
* @throws when neither is available.
|
||||
*/
|
||||
resolve(explicit: string | undefined, toolName: string): string {
|
||||
if (explicit !== undefined && explicit.trim().length > 0) {
|
||||
this.touch(explicit);
|
||||
return explicit;
|
||||
}
|
||||
const current = this.current();
|
||||
if (current === undefined) {
|
||||
throw new Error(
|
||||
`${toolName} needs a session but none is active — start one with browser_session_start ` +
|
||||
"or pass an explicit `session` id",
|
||||
);
|
||||
}
|
||||
this.touch(current);
|
||||
return current;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,853 @@
|
||||
/**
|
||||
* Model-facing tool definitions. Each tool maps to one `bsk <cmd> --json`
|
||||
* invocation: spawn via the runner, parse the JSON envelope, and return a
|
||||
* canonical value whose `output.render` produces the model-facing text.
|
||||
* UI cards reuse the terminal style: the pending card is the command line,
|
||||
* the completed card carries the output.
|
||||
*/
|
||||
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import type { Context } from "@deepseek-ai/cordis";
|
||||
import type { ContentBlock } from "@deepseek-ai/dsh-llm";
|
||||
import { defineTool, type ToolResult, type ToolRunContext } from "@deepseek-ai/dsh-tools";
|
||||
import { trySaveScreenshot } from "./image";
|
||||
import { type BskRunner, bskInstallMessage, isCommandNotFound, parseBskJson } from "./runner";
|
||||
import type { SessionRegistry } from "./sessions";
|
||||
|
||||
/** Plugin configuration resolved from the Schemastery schema in index.ts. */
|
||||
export interface PluginConfig {
|
||||
bskPath: string;
|
||||
defaultTimeoutMs: number;
|
||||
maxSessions: number;
|
||||
}
|
||||
|
||||
export interface ToolDeps {
|
||||
ctx: Context;
|
||||
runner: BskRunner;
|
||||
registry: SessionRegistry;
|
||||
config: PluginConfig;
|
||||
}
|
||||
|
||||
/** Device presets supported by `bsk emulate --device`. */
|
||||
const DEVICE_PRESETS = [
|
||||
"iphone-14",
|
||||
"iphone-14-pro-max",
|
||||
"iphone-se",
|
||||
"pixel-7",
|
||||
"galaxy-s23",
|
||||
"ipad-mini",
|
||||
"galaxy-tab-s8",
|
||||
] as const;
|
||||
|
||||
const SESSION_PARAM = {
|
||||
type: "string",
|
||||
description:
|
||||
"bsk session id to act on. Omit to use the current session (the one most recently started or used).",
|
||||
} as const;
|
||||
|
||||
/** Run one bsk command and return its parsed JSON payload, or throw. */
|
||||
async function runBsk(
|
||||
deps: ToolDeps,
|
||||
exec: ToolRunContext,
|
||||
args: string[],
|
||||
label: string,
|
||||
): Promise<unknown> {
|
||||
let result;
|
||||
try {
|
||||
result = await deps.runner.run(args, {
|
||||
signal: exec.signal,
|
||||
timeoutMs: deps.config.defaultTimeoutMs,
|
||||
});
|
||||
} catch (error) {
|
||||
if (isCommandNotFound(error)) {
|
||||
throw new Error(bskInstallMessage(deps.config.bskPath));
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
if (result.aborted) {
|
||||
const error = new Error("tool call aborted");
|
||||
error.name = "AbortError";
|
||||
throw error;
|
||||
}
|
||||
return parseBskJson(result, label);
|
||||
}
|
||||
|
||||
/** Build the command line shown on the pending terminal card (pure). */
|
||||
function cmdline(deps: ToolDeps, args: string[]): string {
|
||||
return [deps.config.bskPath, ...args].join(" ");
|
||||
}
|
||||
|
||||
/** Shared completed-card presenter: terminal card with the rendered text. */
|
||||
function presentTerminalResult(_args: never, result: ToolResult) {
|
||||
const block = result.content.length === 1 ? result.content[0] : undefined;
|
||||
if (block === undefined || block.type !== "text") return undefined;
|
||||
if (result.isError) return undefined;
|
||||
return { card: "terminal" as const, output: block.text, exitCode: 0 };
|
||||
}
|
||||
|
||||
function abortError(): Error {
|
||||
const error = new Error("tool call aborted");
|
||||
error.name = "AbortError";
|
||||
return error;
|
||||
}
|
||||
|
||||
/** Register the full browser tool suite. */
|
||||
export function registerTools(deps: ToolDeps): void {
|
||||
const { ctx, registry } = deps;
|
||||
|
||||
ctx.tools.register(
|
||||
defineTool({
|
||||
name: "browser_session_start",
|
||||
description:
|
||||
"Start a new browser session: opens an Agent Window in the connected browser and returns " +
|
||||
"its session id. The new session becomes the current session for subsequent browser_* calls. " +
|
||||
"Optionally navigate to an initial URL and/or apply a mobile device emulation preset.",
|
||||
parameters: {
|
||||
url: {
|
||||
type: "string",
|
||||
description: "Initial URL to navigate to after the session starts.",
|
||||
},
|
||||
width: {
|
||||
type: "integer",
|
||||
description: "Agent Window outer width in CSS pixels (100..=7680). Requires height.",
|
||||
},
|
||||
height: {
|
||||
type: "integer",
|
||||
description: "Agent Window outer height in CSS pixels (100..=7680). Requires width.",
|
||||
},
|
||||
noFocus: {
|
||||
type: "boolean",
|
||||
description: "Open the Agent Window in the background without stealing focus.",
|
||||
},
|
||||
browser: {
|
||||
type: "string",
|
||||
description:
|
||||
"Target browser instance id (only needed when multiple browsers are connected).",
|
||||
},
|
||||
device: {
|
||||
type: "string",
|
||||
enum: DEVICE_PRESETS,
|
||||
description: "Mobile device emulation preset applied to the tab after start.",
|
||||
},
|
||||
},
|
||||
output: {
|
||||
schema: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
sessionId: { type: "string", required: true },
|
||||
browserInstanceId: { type: "string", required: true },
|
||||
url: { type: "string" },
|
||||
device: { type: "string" },
|
||||
},
|
||||
},
|
||||
render: (_args, value) => [
|
||||
{
|
||||
type: "text",
|
||||
text:
|
||||
`started browser session ${value.sessionId}` +
|
||||
(value.url !== undefined ? ` and navigated to ${value.url}` : "") +
|
||||
(value.device !== undefined ? ` (device: ${value.device})` : ""),
|
||||
},
|
||||
],
|
||||
},
|
||||
async execute(args, exec) {
|
||||
if (args.url !== undefined && args.url.trim().length === 0) {
|
||||
throw new Error("url must be a non-empty string");
|
||||
}
|
||||
if ((args.width === undefined) !== (args.height === undefined)) {
|
||||
throw new Error("width and height must be given together");
|
||||
}
|
||||
const startArgs = ["session", "start"];
|
||||
if (args.width !== undefined && args.height !== undefined) {
|
||||
startArgs.push("--width", String(args.width), "--height", String(args.height));
|
||||
}
|
||||
if (args.noFocus === true) startArgs.push("--no-focus");
|
||||
if (args.browser !== undefined) startArgs.push("--browser", args.browser);
|
||||
const reply = (await runBsk(deps, exec, startArgs, "session start")) as {
|
||||
session_id: string;
|
||||
browser_instance_id: string;
|
||||
};
|
||||
registry.add({
|
||||
sessionId: reply.session_id,
|
||||
browserInstanceId: reply.browser_instance_id,
|
||||
startedAtMs: Date.now(),
|
||||
});
|
||||
try {
|
||||
if (args.device !== undefined) {
|
||||
await runBsk(
|
||||
deps,
|
||||
exec,
|
||||
["emulate", "--session", reply.session_id, "--device", args.device],
|
||||
"emulate",
|
||||
);
|
||||
}
|
||||
if (args.url !== undefined) {
|
||||
await runBsk(
|
||||
deps,
|
||||
exec,
|
||||
["navigate", "--session", reply.session_id, args.url],
|
||||
"navigate",
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
// A half-initialized session must not leak: stop it before surfacing.
|
||||
await deps.runner
|
||||
.run(["session", "stop", reply.session_id], { timeoutMs: 30_000 })
|
||||
.catch(() => {});
|
||||
registry.remove(reply.session_id);
|
||||
throw error;
|
||||
}
|
||||
return {
|
||||
sessionId: reply.session_id,
|
||||
browserInstanceId: reply.browser_instance_id,
|
||||
...(args.url !== undefined ? { url: args.url } : {}),
|
||||
...(args.device !== undefined ? { device: args.device } : {}),
|
||||
};
|
||||
},
|
||||
presentCall: (args) => ({
|
||||
card: "terminal",
|
||||
title: cmdline(deps, [
|
||||
"session",
|
||||
"start",
|
||||
...(args.device !== undefined ? ["+ emulate", args.device] : []),
|
||||
...(args.url !== undefined ? ["+ navigate", args.url] : []),
|
||||
]),
|
||||
description: "Start a browser session",
|
||||
}),
|
||||
presentResult: presentTerminalResult,
|
||||
}),
|
||||
);
|
||||
|
||||
ctx.tools.register(
|
||||
defineTool({
|
||||
name: "browser_session_stop",
|
||||
description:
|
||||
"Stop a browser session and close its Agent Window. Stops the given session, or the " +
|
||||
"current session when `session` is omitted.",
|
||||
parameters: { session: SESSION_PARAM },
|
||||
output: {
|
||||
schema: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
properties: { stopped: { type: "string", required: true } },
|
||||
},
|
||||
render: (_args, value) => [
|
||||
{ type: "text", text: `stopped browser session ${value.stopped}` },
|
||||
],
|
||||
},
|
||||
async execute(args, exec) {
|
||||
const sessionId = registry.resolve(args.session, "browser_session_stop");
|
||||
await runBsk(deps, exec, ["session", "stop", sessionId], "session stop");
|
||||
registry.remove(sessionId);
|
||||
return { stopped: sessionId };
|
||||
},
|
||||
presentCall: (args) => ({
|
||||
card: "terminal",
|
||||
title: cmdline(deps, ["session", "stop", args.session ?? "(current session)"]),
|
||||
description: "Stop a browser session",
|
||||
}),
|
||||
presentResult: presentTerminalResult,
|
||||
}),
|
||||
);
|
||||
|
||||
ctx.tools.register(
|
||||
defineTool({
|
||||
name: "browser_session_list",
|
||||
description:
|
||||
"List active browser sessions known to the bsk daemon. The session marked `current` is " +
|
||||
"the one browser_* tools act on when no explicit `session` is passed.",
|
||||
parameters: {},
|
||||
output: {
|
||||
schema: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
sessions: {
|
||||
type: "array",
|
||||
required: true,
|
||||
items: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
sessionId: { type: "string", required: true },
|
||||
browserInstanceId: { type: "string", required: true },
|
||||
current: { type: "boolean", required: true },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
render: (_args, value) => [
|
||||
{
|
||||
type: "text",
|
||||
text:
|
||||
value.sessions.length === 0
|
||||
? "no active browser sessions"
|
||||
: value.sessions
|
||||
.map(
|
||||
(s) =>
|
||||
`${s.sessionId} (browser ${s.browserInstanceId})${s.current ? " [current]" : ""}`,
|
||||
)
|
||||
.join("\n"),
|
||||
},
|
||||
],
|
||||
},
|
||||
isConcurrencySafe: () => true,
|
||||
async execute(_args, exec) {
|
||||
const reply = (await runBsk(deps, exec, ["session", "list"], "session list")) as {
|
||||
session_id: string;
|
||||
browser_instance_id: string;
|
||||
}[];
|
||||
const current = registry.current();
|
||||
return {
|
||||
sessions: reply.map((entry) => ({
|
||||
sessionId: entry.session_id,
|
||||
browserInstanceId: entry.browser_instance_id,
|
||||
current: entry.session_id === current,
|
||||
})),
|
||||
};
|
||||
},
|
||||
presentCall: () => ({
|
||||
card: "terminal",
|
||||
title: cmdline(deps, ["session", "list"]),
|
||||
description: "List browser sessions",
|
||||
}),
|
||||
presentResult: presentTerminalResult,
|
||||
}),
|
||||
);
|
||||
|
||||
ctx.tools.register(
|
||||
defineTool({
|
||||
name: "browser_navigate",
|
||||
description:
|
||||
"Navigate the session's active tab to a URL and wait for a page lifecycle phase " +
|
||||
"(default: load). Returns the final URL after redirects.",
|
||||
parameters: {
|
||||
url: { type: "string", required: true, description: "Destination URL." },
|
||||
session: SESSION_PARAM,
|
||||
waitUntil: {
|
||||
type: "string",
|
||||
enum: ["load", "domcontentloaded", "networkidle", "commit"],
|
||||
description: "Lifecycle phase to wait for (default: load).",
|
||||
},
|
||||
timeoutMs: {
|
||||
type: "integer",
|
||||
description: "Navigation wait timeout in milliseconds (default 30000).",
|
||||
},
|
||||
},
|
||||
output: {
|
||||
schema: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
session: { type: "string", required: true },
|
||||
tabId: { type: "integer", required: true },
|
||||
url: { type: "string", required: true },
|
||||
finalUrl: { type: "string" },
|
||||
reached: { type: "string", required: true },
|
||||
},
|
||||
},
|
||||
render: (_args, value) => [
|
||||
{
|
||||
type: "text",
|
||||
text:
|
||||
`[session ${value.session}] navigated to ${value.finalUrl ?? value.url} (reached: ${value.reached})` +
|
||||
(value.reached === "timeout"
|
||||
? " — the wait timed out before the requested phase"
|
||||
: ""),
|
||||
},
|
||||
],
|
||||
},
|
||||
async execute(args, exec) {
|
||||
if (args.url.trim().length === 0) throw new Error("url must be a non-empty string");
|
||||
const sessionId = registry.resolve(args.session, "browser_navigate");
|
||||
const cmdArgs = ["navigate", "--session", sessionId];
|
||||
if (args.waitUntil !== undefined) cmdArgs.push("--wait-until", args.waitUntil);
|
||||
if (args.timeoutMs !== undefined) cmdArgs.push("--timeout", `${args.timeoutMs}ms`);
|
||||
cmdArgs.push(args.url);
|
||||
const reply = (await runBsk(deps, exec, cmdArgs, "navigate")) as {
|
||||
tab_id: number;
|
||||
url: string;
|
||||
final_url?: string;
|
||||
reached: string;
|
||||
};
|
||||
return {
|
||||
session: sessionId,
|
||||
tabId: reply.tab_id,
|
||||
url: reply.url,
|
||||
...(reply.final_url !== undefined ? { finalUrl: reply.final_url } : {}),
|
||||
reached: reply.reached,
|
||||
};
|
||||
},
|
||||
presentCall: (args) => ({
|
||||
card: "terminal",
|
||||
title: cmdline(deps, ["navigate", "--session", args.session ?? "(current)", args.url]),
|
||||
description: "Navigate to a URL",
|
||||
}),
|
||||
presentResult: presentTerminalResult,
|
||||
}),
|
||||
);
|
||||
|
||||
const registerObservationTool = (kind: "snapshot" | "observe") => {
|
||||
const isSnapshot = kind === "snapshot";
|
||||
const name = isSnapshot ? "browser_snapshot" : "browser_observe";
|
||||
const description = isSnapshot
|
||||
? "Capture an indented aria-tree snapshot of the session's active tab. Interactive elements " +
|
||||
"carry @eN refs for browser_click / browser_fill. Prefer browser_observe for a richer semantic view."
|
||||
: "Produce a semantic VOM observation of the session's active tab (roles, states, perception " +
|
||||
"probes) with @eN refs for browser_click / browser_fill. Read-only: never submits input.";
|
||||
ctx.tools.register(
|
||||
defineTool({
|
||||
name,
|
||||
description,
|
||||
parameters: {
|
||||
session: SESSION_PARAM,
|
||||
maxDepth: { type: "integer", description: "Cap on tree depth before truncating." },
|
||||
maxTokens: {
|
||||
type: "integer",
|
||||
description: "Soft cap on rendered tokens (~4 chars/token).",
|
||||
},
|
||||
},
|
||||
output: {
|
||||
schema: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
session: { type: "string", required: true },
|
||||
tabId: { type: "integer", required: true },
|
||||
text: { type: "string", required: true },
|
||||
refCount: { type: "integer", required: true },
|
||||
truncated: { type: "boolean", required: true },
|
||||
},
|
||||
},
|
||||
render: (_args, value) => [
|
||||
{
|
||||
type: "text",
|
||||
text:
|
||||
value.text.length > 0
|
||||
? value.text + (value.truncated ? "\n(truncated — re-run with looser caps)" : "")
|
||||
: "(empty observation — page may still be loading)",
|
||||
},
|
||||
],
|
||||
},
|
||||
isConcurrencySafe: () => true,
|
||||
async execute(args, exec) {
|
||||
const sessionId = registry.resolve(args.session, name);
|
||||
const cmdArgs = [kind, "--session", sessionId];
|
||||
if (args.maxDepth !== undefined) cmdArgs.push("--max-depth", String(args.maxDepth));
|
||||
if (args.maxTokens !== undefined) cmdArgs.push("--max-tokens", String(args.maxTokens));
|
||||
const reply = (await runBsk(deps, exec, cmdArgs, kind)) as {
|
||||
text: string;
|
||||
ref_count: number;
|
||||
tab_id: number;
|
||||
truncated?: boolean;
|
||||
};
|
||||
return {
|
||||
session: sessionId,
|
||||
tabId: reply.tab_id,
|
||||
text: reply.text,
|
||||
refCount: reply.ref_count,
|
||||
truncated: reply.truncated ?? false,
|
||||
};
|
||||
},
|
||||
presentCall: (args) => ({
|
||||
card: "terminal",
|
||||
title: cmdline(deps, [kind, "--session", args.session ?? "(current)"]),
|
||||
description: isSnapshot ? "Capture an aria snapshot" : "Observe the page semantically",
|
||||
}),
|
||||
presentResult: presentTerminalResult,
|
||||
}),
|
||||
);
|
||||
};
|
||||
registerObservationTool("snapshot");
|
||||
registerObservationTool("observe");
|
||||
|
||||
ctx.tools.register(
|
||||
defineTool({
|
||||
name: "browser_click",
|
||||
description:
|
||||
"Click an element in the session's active tab. Target is a snapshot ref (@e3) from the " +
|
||||
"last browser_snapshot/browser_observe, or a CSS selector.",
|
||||
parameters: {
|
||||
target: {
|
||||
type: "string",
|
||||
required: true,
|
||||
description: "Snapshot ref (@e3 / e3) or CSS selector of the element to click.",
|
||||
},
|
||||
session: SESSION_PARAM,
|
||||
button: {
|
||||
type: "string",
|
||||
enum: ["left", "middle", "right"],
|
||||
description: "Mouse button (default: left).",
|
||||
},
|
||||
clickCount: {
|
||||
type: "integer",
|
||||
description: "Number of consecutive presses (double-click = 2).",
|
||||
},
|
||||
},
|
||||
output: {
|
||||
schema: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
session: { type: "string", required: true },
|
||||
tabId: { type: "integer", required: true },
|
||||
x: { type: "number", required: true },
|
||||
y: { type: "number", required: true },
|
||||
},
|
||||
},
|
||||
render: (_args, value) => [
|
||||
{
|
||||
type: "text",
|
||||
text: `[session ${value.session}] clicked at (${value.x}, ${value.y}) on tab ${value.tabId}`,
|
||||
},
|
||||
],
|
||||
},
|
||||
async execute(args, exec) {
|
||||
if (args.target.trim().length === 0) throw new Error("target must be a non-empty string");
|
||||
const sessionId = registry.resolve(args.session, "browser_click");
|
||||
const cmdArgs = ["click", "--session", sessionId];
|
||||
if (args.button !== undefined) cmdArgs.push("--button", args.button);
|
||||
if (args.clickCount !== undefined) cmdArgs.push("--click-count", String(args.clickCount));
|
||||
cmdArgs.push(args.target);
|
||||
const reply = (await runBsk(deps, exec, cmdArgs, "click")) as {
|
||||
tab_id: number;
|
||||
x: number;
|
||||
y: number;
|
||||
};
|
||||
return { session: sessionId, tabId: reply.tab_id, x: reply.x, y: reply.y };
|
||||
},
|
||||
presentCall: (args) => ({
|
||||
card: "terminal",
|
||||
title: cmdline(deps, ["click", args.target, "--session", args.session ?? "(current)"]),
|
||||
description: "Click an element",
|
||||
}),
|
||||
presentResult: presentTerminalResult,
|
||||
}),
|
||||
);
|
||||
|
||||
ctx.tools.register(
|
||||
defineTool({
|
||||
name: "browser_fill",
|
||||
description:
|
||||
"Fill an input / textarea / contenteditable element, clearing it first by default. " +
|
||||
"Target is a snapshot ref (@e3) or a CSS selector.",
|
||||
parameters: {
|
||||
target: {
|
||||
type: "string",
|
||||
required: true,
|
||||
description: "Snapshot ref (@e3 / e3) or CSS selector of the field.",
|
||||
},
|
||||
value: { type: "string", required: true, description: "Text to type into the element." },
|
||||
session: SESSION_PARAM,
|
||||
noClear: {
|
||||
type: "boolean",
|
||||
description: "Skip the default wipe-the-field-first pass (append instead).",
|
||||
},
|
||||
},
|
||||
output: {
|
||||
schema: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
session: { type: "string", required: true },
|
||||
tabId: { type: "integer", required: true },
|
||||
valueLength: { type: "integer", required: true },
|
||||
},
|
||||
},
|
||||
render: (_args, value) => [
|
||||
{
|
||||
type: "text",
|
||||
text: `[session ${value.session}] filled field on tab ${value.tabId} (${value.valueLength} chars)`,
|
||||
},
|
||||
],
|
||||
},
|
||||
async execute(args, exec) {
|
||||
if (args.target.trim().length === 0) throw new Error("target must be a non-empty string");
|
||||
const sessionId = registry.resolve(args.session, "browser_fill");
|
||||
const cmdArgs = ["fill", "--session", sessionId, "--value", args.value];
|
||||
if (args.noClear === true) cmdArgs.push("--no-clear");
|
||||
cmdArgs.push(args.target);
|
||||
const reply = (await runBsk(deps, exec, cmdArgs, "fill")) as {
|
||||
tab_id: number;
|
||||
value_length: number;
|
||||
};
|
||||
return { session: sessionId, tabId: reply.tab_id, valueLength: reply.value_length };
|
||||
},
|
||||
presentCall: (args) => ({
|
||||
card: "terminal",
|
||||
title: cmdline(deps, ["fill", args.target, "--session", args.session ?? "(current)"]),
|
||||
description: `Fill a field with ${args.value.length} chars`,
|
||||
}),
|
||||
presentResult: presentTerminalResult,
|
||||
}),
|
||||
);
|
||||
|
||||
ctx.tools.register(
|
||||
defineTool({
|
||||
name: "browser_press",
|
||||
description:
|
||||
"Dispatch a keyboard key or combo (Enter, Escape, ArrowDown, Ctrl+A, …) to the session's " +
|
||||
"active tab, optionally focusing a target element first.",
|
||||
parameters: {
|
||||
key: {
|
||||
type: "string",
|
||||
required: true,
|
||||
description: "Key spec: a CDP key name (Enter, a, ArrowLeft) or a combo (Ctrl+A).",
|
||||
},
|
||||
session: SESSION_PARAM,
|
||||
target: {
|
||||
type: "string",
|
||||
description: "Snapshot ref (@e3) or CSS selector to focus before pressing.",
|
||||
},
|
||||
holdMs: {
|
||||
type: "integer",
|
||||
description: "Hold the key down for N milliseconds between keyDown and keyUp.",
|
||||
},
|
||||
},
|
||||
output: {
|
||||
schema: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
session: { type: "string", required: true },
|
||||
tabId: { type: "integer", required: true },
|
||||
key: { type: "string", required: true },
|
||||
},
|
||||
},
|
||||
render: (_args, value) => [
|
||||
{
|
||||
type: "text",
|
||||
text: `[session ${value.session}] pressed ${value.key} on tab ${value.tabId}`,
|
||||
},
|
||||
],
|
||||
},
|
||||
async execute(args, exec) {
|
||||
if (args.key.trim().length === 0) throw new Error("key must be a non-empty string");
|
||||
const sessionId = registry.resolve(args.session, "browser_press");
|
||||
const cmdArgs = ["press", "--session", sessionId];
|
||||
if (args.target !== undefined) {
|
||||
if (args.target.trim().length === 0) throw new Error("target must be a non-empty string");
|
||||
// bsk distinguishes ref vs selector via flags; positional detection is CLI-side for
|
||||
// click/fill only, so pass the explicit flag that matches the target's shape.
|
||||
if (/^@?e\d+$/.test(args.target)) {
|
||||
cmdArgs.push("--ref", args.target);
|
||||
} else {
|
||||
cmdArgs.push("--selector", args.target);
|
||||
}
|
||||
}
|
||||
if (args.holdMs !== undefined) cmdArgs.push("--hold-ms", String(args.holdMs));
|
||||
cmdArgs.push(args.key);
|
||||
const reply = (await runBsk(deps, exec, cmdArgs, "press")) as {
|
||||
tab_id: number;
|
||||
key: string;
|
||||
};
|
||||
return { session: sessionId, tabId: reply.tab_id, key: reply.key };
|
||||
},
|
||||
presentCall: (args) => ({
|
||||
card: "terminal",
|
||||
title: cmdline(deps, ["press", args.key, "--session", args.session ?? "(current)"]),
|
||||
description: "Press a key",
|
||||
}),
|
||||
presentResult: presentTerminalResult,
|
||||
}),
|
||||
);
|
||||
|
||||
ctx.tools.register(
|
||||
defineTool({
|
||||
name: "browser_screenshot",
|
||||
description:
|
||||
"Capture a PNG screenshot of the session's active tab, or crop to a snapshot ref element. " +
|
||||
"Returns the image itself when the deployment supports image input, otherwise a file path.",
|
||||
parameters: {
|
||||
session: SESSION_PARAM,
|
||||
ref: {
|
||||
type: "string",
|
||||
description: "Snapshot ref (@e3) from the last snapshot/observe; crops to that element.",
|
||||
},
|
||||
},
|
||||
output: {
|
||||
schema: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
session: { type: "string", required: true },
|
||||
tabId: { type: "integer", required: true },
|
||||
path: { type: "string", required: true },
|
||||
width: { type: "integer", required: true },
|
||||
height: { type: "integer", required: true },
|
||||
byteSize: { type: "integer", required: true },
|
||||
image: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
attachmentId: { type: "string", required: true },
|
||||
mediaType: { type: "string", required: true, const: "image/png" },
|
||||
bytes: { type: "integer", required: true },
|
||||
width: { type: "integer", required: true },
|
||||
height: { type: "integer", required: true },
|
||||
name: { type: "string" },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
render: (_args, value) => {
|
||||
const text =
|
||||
value.image !== undefined
|
||||
? `[session ${value.session}] screenshot of tab ${value.tabId} (${value.width}x${value.height}px)`
|
||||
: `[session ${value.session}] screenshot saved to ${value.path} (${value.width}x${value.height}px, ${value.byteSize} bytes) — this deployment cannot inline images; read the file to view it`;
|
||||
const blocks: ContentBlock[] = [{ type: "text", text }];
|
||||
if (value.image !== undefined) {
|
||||
blocks.push({
|
||||
type: "image",
|
||||
// Structurally an ImageAttachmentRef; branded without a runtime import.
|
||||
attachment: {
|
||||
attachmentId: value.image.attachmentId,
|
||||
mediaType: value.image.mediaType,
|
||||
bytes: value.image.bytes,
|
||||
width: value.image.width,
|
||||
height: value.image.height,
|
||||
...(value.image.name !== undefined ? { name: value.image.name } : {}),
|
||||
} as never,
|
||||
});
|
||||
}
|
||||
return blocks;
|
||||
},
|
||||
},
|
||||
isConcurrencySafe: () => true,
|
||||
async execute(args, exec) {
|
||||
const sessionId = registry.resolve(args.session, "browser_screenshot");
|
||||
const outPath = join(
|
||||
tmpdir(),
|
||||
`bsk-screenshot-${Date.now()}-${Math.random().toString(36).slice(2, 8)}.png`,
|
||||
);
|
||||
const cmdArgs = ["screenshot", "--session", sessionId, "--out", outPath];
|
||||
if (args.ref !== undefined) cmdArgs.push("--ref", args.ref);
|
||||
const reply = (await runBsk(deps, exec, cmdArgs, "screenshot")) as {
|
||||
tab_id: number;
|
||||
width: number;
|
||||
height: number;
|
||||
path: string;
|
||||
byte_size: number;
|
||||
};
|
||||
const data = await readFile(reply.path);
|
||||
if (exec.signal.aborted) throw abortError();
|
||||
const ref = await trySaveScreenshot(deps.ctx, exec, data, `screenshot-${sessionId}.png`);
|
||||
return {
|
||||
session: sessionId,
|
||||
tabId: reply.tab_id,
|
||||
path: reply.path,
|
||||
width: reply.width,
|
||||
height: reply.height,
|
||||
byteSize: reply.byte_size,
|
||||
...(ref !== undefined
|
||||
? {
|
||||
image: {
|
||||
attachmentId: String(ref.attachmentId),
|
||||
mediaType: "image/png" as const,
|
||||
bytes: ref.bytes,
|
||||
width: ref.width,
|
||||
height: ref.height,
|
||||
...(ref.name !== undefined ? { name: ref.name } : {}),
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
},
|
||||
presentCall: (args) => ({
|
||||
card: "terminal",
|
||||
title: cmdline(deps, [
|
||||
"screenshot",
|
||||
"--session",
|
||||
args.session ?? "(current)",
|
||||
...(args.ref !== undefined ? ["--ref", args.ref] : []),
|
||||
]),
|
||||
description: "Capture a screenshot",
|
||||
}),
|
||||
presentResult: presentTerminalResult,
|
||||
}),
|
||||
);
|
||||
|
||||
ctx.tools.register(
|
||||
defineTool({
|
||||
name: "browser_emulate",
|
||||
description:
|
||||
"Apply (or clear) mobile device emulation on the session's active tab: viewport metrics, " +
|
||||
"user agent, and touch. Overrides are per-tab and not inherited by new tabs.",
|
||||
parameters: {
|
||||
session: SESSION_PARAM,
|
||||
device: {
|
||||
type: "string",
|
||||
enum: DEVICE_PRESETS,
|
||||
description: "Built-in device preset.",
|
||||
},
|
||||
width: { type: "integer", description: "Viewport width in CSS pixels (requires height)." },
|
||||
height: { type: "integer", description: "Viewport height in CSS pixels (requires width)." },
|
||||
mobile: { type: "boolean", description: "Emulate a mobile viewport." },
|
||||
off: { type: "boolean", description: "Clear every emulation override on the tab." },
|
||||
},
|
||||
output: {
|
||||
schema: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
session: { type: "string", required: true },
|
||||
tabId: { type: "integer", required: true },
|
||||
cleared: { type: "boolean", required: true },
|
||||
},
|
||||
},
|
||||
render: (_args, value) => [
|
||||
{
|
||||
type: "text",
|
||||
text: value.cleared
|
||||
? `[session ${value.session}] cleared emulation on tab ${value.tabId}`
|
||||
: `[session ${value.session}] applied emulation on tab ${value.tabId}`,
|
||||
},
|
||||
],
|
||||
},
|
||||
async execute(args, exec) {
|
||||
const sessionId = registry.resolve(args.session, "browser_emulate");
|
||||
if (args.off === true) {
|
||||
if (args.device !== undefined || args.width !== undefined || args.height !== undefined) {
|
||||
throw new Error("off is mutually exclusive with device/width/height");
|
||||
}
|
||||
} else if (args.device === undefined && args.width === undefined) {
|
||||
throw new Error("nothing to apply: pass device, width+height, or off");
|
||||
}
|
||||
if ((args.width === undefined) !== (args.height === undefined)) {
|
||||
throw new Error("width and height must be given together");
|
||||
}
|
||||
const cmdArgs = ["emulate", "--session", sessionId];
|
||||
if (args.off === true) {
|
||||
cmdArgs.push("--off");
|
||||
} else {
|
||||
if (args.device !== undefined) cmdArgs.push("--device", args.device);
|
||||
if (args.width !== undefined && args.height !== undefined) {
|
||||
cmdArgs.push("--width", String(args.width), "--height", String(args.height));
|
||||
}
|
||||
if (args.mobile === true) cmdArgs.push("--mobile");
|
||||
}
|
||||
const reply = (await runBsk(deps, exec, cmdArgs, "emulate")) as {
|
||||
tab_id: number;
|
||||
cleared: boolean;
|
||||
};
|
||||
return { session: sessionId, tabId: reply.tab_id, cleared: reply.cleared };
|
||||
},
|
||||
presentCall: (args) => ({
|
||||
card: "terminal",
|
||||
title: cmdline(deps, [
|
||||
"emulate",
|
||||
"--session",
|
||||
args.session ?? "(current)",
|
||||
...(args.off === true ? ["--off"] : []),
|
||||
...(args.device !== undefined ? ["--device", args.device] : []),
|
||||
]),
|
||||
description: "Emulate a device environment",
|
||||
}),
|
||||
presentResult: presentTerminalResult,
|
||||
}),
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
import type { ChildProcess } from "node:child_process";
|
||||
import { EventEmitter } from "node:events";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { type BskRunResult, createBskRunner, isCommandNotFound, parseBskJson } from "../src/runner";
|
||||
|
||||
/** Minimal fake ChildProcess driven by the test. */
|
||||
class FakeChild extends EventEmitter {
|
||||
stdout = new EventEmitter();
|
||||
stderr = new EventEmitter();
|
||||
exitCode: number | null = null;
|
||||
signalCode: string | null = null;
|
||||
killedWith: string[] = [];
|
||||
|
||||
kill(signal: string): boolean {
|
||||
if (this.exitCode !== null || this.signalCode !== null) return false;
|
||||
this.killedWith.push(signal);
|
||||
this.signalCode = signal;
|
||||
queueMicrotask(() => this.emit("close", null));
|
||||
return true;
|
||||
}
|
||||
|
||||
finish(code: number, stdout = "", stderr = ""): void {
|
||||
if (stdout) this.stdout.emit("data", stdout);
|
||||
if (stderr) this.stderr.emit("data", stderr);
|
||||
this.exitCode = code;
|
||||
this.emit("close", code);
|
||||
}
|
||||
}
|
||||
|
||||
function fakeSpawn(children: FakeChild[]) {
|
||||
return () => {
|
||||
const child = children.shift();
|
||||
if (child === undefined) throw new Error("no fake child queued");
|
||||
return child as unknown as ChildProcess;
|
||||
};
|
||||
}
|
||||
|
||||
describe("createBskRunner", () => {
|
||||
it("appends --json and collects stdout", async () => {
|
||||
const child = new FakeChild();
|
||||
const seen: string[][] = [];
|
||||
const runner = createBskRunner("bsk", (cmd: string, args: string[]) => {
|
||||
seen.push([cmd, ...args]);
|
||||
return child as unknown as ChildProcess;
|
||||
});
|
||||
const promise = runner.run(["session", "list"]);
|
||||
child.finish(0, '{"ok":true}');
|
||||
const result = await promise;
|
||||
expect(seen).toEqual([["bsk", "session", "list", "--json"]]);
|
||||
expect(result).toMatchObject({
|
||||
code: 0,
|
||||
stdout: '{"ok":true}',
|
||||
aborted: false,
|
||||
timedOut: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("kills the child when the abort signal fires", async () => {
|
||||
const child = new FakeChild();
|
||||
const runner = createBskRunner("bsk", fakeSpawn([child]));
|
||||
const controller = new AbortController();
|
||||
const promise = runner.run(["snapshot"], { signal: controller.signal });
|
||||
controller.abort();
|
||||
const result = await promise;
|
||||
expect(result.aborted).toBe(true);
|
||||
expect(child.killedWith).toContain("SIGTERM");
|
||||
});
|
||||
|
||||
it("kills the child on timeout", async () => {
|
||||
const child = new FakeChild();
|
||||
const runner = createBskRunner("bsk", fakeSpawn([child]));
|
||||
const result = await runner.run(["snapshot"], { timeoutMs: 5 });
|
||||
expect(result.timedOut).toBe(true);
|
||||
expect(child.killedWith.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("killAll terminates in-flight children", async () => {
|
||||
const child = new FakeChild();
|
||||
const runner = createBskRunner("bsk", fakeSpawn([child]));
|
||||
const promise = runner.run(["session", "list"]);
|
||||
runner.killAll();
|
||||
const result = await promise;
|
||||
expect(child.killedWith).toContain("SIGTERM");
|
||||
expect(result.code).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseBskJson", () => {
|
||||
const base: BskRunResult = { code: 0, stdout: "", stderr: "", timedOut: false, aborted: false };
|
||||
|
||||
it("parses stdout JSON on success", () => {
|
||||
expect(parseBskJson({ ...base, stdout: '{"a":1}' }, "status")).toEqual({ a: 1 });
|
||||
});
|
||||
|
||||
it("maps the JSON error envelope on non-zero exit", () => {
|
||||
const body = JSON.stringify({
|
||||
code: "no_browser",
|
||||
message: "no browser connected",
|
||||
hint: "open Chrome",
|
||||
});
|
||||
try {
|
||||
parseBskJson({ ...base, code: 3, stdout: body }, "session start");
|
||||
expect.unreachable();
|
||||
} catch (error) {
|
||||
expect(error).toMatchObject({
|
||||
name: "BskError",
|
||||
code: "no_browser",
|
||||
hint: "open Chrome",
|
||||
exitCode: 3,
|
||||
});
|
||||
expect((error as Error).message).toContain("no browser connected");
|
||||
}
|
||||
});
|
||||
|
||||
it("falls back to stderr when the error body is not JSON", () => {
|
||||
expect(() => parseBskJson({ ...base, code: 1, stderr: "boom" }, "x")).toThrow(/boom/);
|
||||
});
|
||||
|
||||
it("reports timeouts distinctly", () => {
|
||||
expect(() => parseBskJson({ ...base, timedOut: true }, "x")).toThrow(/timed out/);
|
||||
});
|
||||
|
||||
it("rejects non-JSON success output", () => {
|
||||
expect(() => parseBskJson({ ...base, stdout: "not json" }, "x")).toThrow(
|
||||
/did not produce JSON/,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("isCommandNotFound", () => {
|
||||
it("detects ENOENT spawn failures", () => {
|
||||
expect(
|
||||
isCommandNotFound(Object.assign(new Error("spawn bsk ENOENT"), { code: "ENOENT" })),
|
||||
).toBe(true);
|
||||
expect(isCommandNotFound(new Error("other"))).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,55 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { SessionRegistry } from "../src/sessions";
|
||||
|
||||
describe("SessionRegistry", () => {
|
||||
it("tracks the current session across add and remove", () => {
|
||||
const registry = new SessionRegistry(5);
|
||||
expect(registry.current()).toBeUndefined();
|
||||
registry.add({ sessionId: "s1", startedAtMs: 1 });
|
||||
expect(registry.current()).toBe("s1");
|
||||
registry.add({ sessionId: "s2", startedAtMs: 2 });
|
||||
expect(registry.current()).toBe("s2");
|
||||
registry.remove("s2");
|
||||
expect(registry.current()).toBe("s1");
|
||||
registry.remove("s1");
|
||||
expect(registry.current()).toBeUndefined();
|
||||
});
|
||||
|
||||
it("enforces the concurrency cap", () => {
|
||||
const registry = new SessionRegistry(2);
|
||||
registry.add({ sessionId: "s1", startedAtMs: 1 });
|
||||
registry.add({ sessionId: "s2", startedAtMs: 2 });
|
||||
expect(() => registry.add({ sessionId: "s3", startedAtMs: 3 })).toThrow(/session limit/);
|
||||
registry.remove("s1");
|
||||
registry.add({ sessionId: "s3", startedAtMs: 3 });
|
||||
expect(registry.size()).toBe(2);
|
||||
});
|
||||
|
||||
it("resolve prefers the explicit session and adopts unknown ids", () => {
|
||||
const registry = new SessionRegistry(5);
|
||||
registry.add({ sessionId: "s1", startedAtMs: 1 });
|
||||
expect(registry.resolve("s9", "tool")).toBe("s9");
|
||||
expect(registry.current()).toBe("s9");
|
||||
expect(registry.list().map((s) => s.sessionId)).toContain("s9");
|
||||
});
|
||||
|
||||
it("resolve falls back to the current session", () => {
|
||||
const registry = new SessionRegistry(5);
|
||||
registry.add({ sessionId: "s1", startedAtMs: 1 });
|
||||
expect(registry.resolve(undefined, "tool")).toBe("s1");
|
||||
});
|
||||
|
||||
it("resolve throws actionable guidance when no session exists", () => {
|
||||
const registry = new SessionRegistry(5);
|
||||
expect(() => registry.resolve(undefined, "browser_click")).toThrow(/browser_session_start/);
|
||||
});
|
||||
|
||||
it("touch refreshes recency so fallback picks the most recent survivor", () => {
|
||||
const registry = new SessionRegistry(5);
|
||||
registry.add({ sessionId: "s1", startedAtMs: 1 });
|
||||
registry.add({ sessionId: "s2", startedAtMs: 2 });
|
||||
registry.touch("s1");
|
||||
registry.remove("s1");
|
||||
expect(registry.current()).toBe("s2");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,525 @@
|
||||
import { mkdtempSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import type { ToolDefinition, ToolRunContext } from "@deepseek-ai/dsh-tools";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import type { BskRunOptions, BskRunResult } from "../src/runner";
|
||||
import { SessionRegistry } from "../src/sessions";
|
||||
import { type PluginConfig, registerTools } from "../src/tools";
|
||||
|
||||
const CONFIG: PluginConfig = { bskPath: "bsk", defaultTimeoutMs: 120_000, maxSessions: 5 };
|
||||
|
||||
interface FakeCall {
|
||||
args: string[];
|
||||
options: BskRunOptions;
|
||||
}
|
||||
|
||||
/** Mark a canned answer as a per-call sequence (each call shifts one reply). */
|
||||
function seq(items: unknown[]): { __seq: unknown[] } {
|
||||
return { __seq: items };
|
||||
}
|
||||
|
||||
/** A runner that answers canned JSON per command prefix and records calls. */
|
||||
function fakeRunner(responses: Record<string, unknown>) {
|
||||
const calls: FakeCall[] = [];
|
||||
return {
|
||||
calls,
|
||||
runner: {
|
||||
async run(args: string[], options: BskRunOptions = {}): Promise<BskRunResult> {
|
||||
calls.push({ args, options });
|
||||
if (options.signal?.aborted) {
|
||||
return { code: null, stdout: "", stderr: "", timedOut: false, aborted: true };
|
||||
}
|
||||
for (const [prefix, response] of Object.entries(responses)) {
|
||||
if (args.join(" ").startsWith(prefix)) {
|
||||
if (response instanceof Error) throw response;
|
||||
const reply =
|
||||
typeof response === "object" && response !== null && "__seq" in response
|
||||
? (response as { __seq: unknown[] }).__seq.shift()
|
||||
: response;
|
||||
return {
|
||||
code: 0,
|
||||
stdout: JSON.stringify(reply),
|
||||
stderr: "",
|
||||
timedOut: false,
|
||||
aborted: false,
|
||||
};
|
||||
}
|
||||
}
|
||||
return {
|
||||
code: 2,
|
||||
stdout: JSON.stringify({
|
||||
code: "unexpected",
|
||||
message: `no canned response for: ${args.join(" ")}`,
|
||||
}),
|
||||
stderr: "",
|
||||
timedOut: false,
|
||||
aborted: false,
|
||||
};
|
||||
},
|
||||
killAll() {},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function makeCtx(services: Record<string, unknown> = {}) {
|
||||
const tools = new Map<string, ToolDefinition>();
|
||||
const ctx = {
|
||||
tools: { register: (def: ToolDefinition) => tools.set(def.name, def) },
|
||||
get: (key: string) => services[key],
|
||||
};
|
||||
return { ctx, tools };
|
||||
}
|
||||
|
||||
function makeExec(overrides: Record<string, unknown> = {}): ToolRunContext {
|
||||
return {
|
||||
callId: "test-call",
|
||||
name: "test",
|
||||
signal: new AbortController().signal,
|
||||
...overrides,
|
||||
} as unknown as ToolRunContext;
|
||||
}
|
||||
|
||||
const SNAPSHOT_REPLY = {
|
||||
text: '- button "OK" [ref=@e1]',
|
||||
ref_count: 1,
|
||||
tab_id: 7,
|
||||
truncated: false,
|
||||
};
|
||||
|
||||
function setup(
|
||||
responses: Record<string, unknown>,
|
||||
services: Record<string, unknown> = {},
|
||||
maxSessions = 5,
|
||||
) {
|
||||
const { ctx, tools } = makeCtx(services);
|
||||
const { runner, calls } = fakeRunner(responses);
|
||||
const registry = new SessionRegistry(maxSessions);
|
||||
registerTools({ ctx: ctx as never, runner, registry, config: { ...CONFIG, maxSessions } });
|
||||
return { tools, calls, registry };
|
||||
}
|
||||
|
||||
async function startSession(tools: Map<string, ToolDefinition>, sessionId = "s1") {
|
||||
const tool = tools.get("browser_session_start");
|
||||
if (tool === undefined) throw new Error("browser_session_start not registered");
|
||||
return tool.execute({}, makeExec()) as Promise<{ sessionId: string }>;
|
||||
}
|
||||
|
||||
const START_REPLY = (id: string) => ({ session_id: id, browser_instance_id: "chrome-1" });
|
||||
|
||||
describe("tool registration", () => {
|
||||
it("registers the full browser tool suite", () => {
|
||||
const { tools } = setup({});
|
||||
expect([...tools.keys()].sort()).toEqual([
|
||||
"browser_click",
|
||||
"browser_emulate",
|
||||
"browser_fill",
|
||||
"browser_navigate",
|
||||
"browser_observe",
|
||||
"browser_press",
|
||||
"browser_screenshot",
|
||||
"browser_session_list",
|
||||
"browser_session_start",
|
||||
"browser_session_stop",
|
||||
"browser_snapshot",
|
||||
]);
|
||||
});
|
||||
|
||||
it("validates model args through defineTool (missing required param)", async () => {
|
||||
const { tools } = setup({});
|
||||
const navigate = tools.get("browser_navigate");
|
||||
await expect(navigate?.execute({}, makeExec())).rejects.toThrow(/invalid arguments/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("browser_session_start", () => {
|
||||
it("maps the start reply and tracks the session as current", async () => {
|
||||
const { tools, registry } = setup({ "session start": START_REPLY("s1") });
|
||||
const value = await startSession(tools);
|
||||
expect(value.sessionId).toBe("s1");
|
||||
expect(registry.current()).toBe("s1");
|
||||
});
|
||||
|
||||
it("passes window size and opens the initial URL and device preset", async () => {
|
||||
const { tools, calls } = setup({
|
||||
"session start": START_REPLY("s1"),
|
||||
emulate: { tab_id: 7, cleared: false },
|
||||
navigate: { tab_id: 7, url: "https://example.com", reached: "load" },
|
||||
});
|
||||
const tool = tools.get("browser_session_start");
|
||||
const value = (await tool?.execute(
|
||||
{ width: 1280, height: 800, url: "https://example.com", device: "iphone-14", noFocus: true },
|
||||
makeExec(),
|
||||
)) as { sessionId: string; url: string; device: string };
|
||||
expect(value).toMatchObject({
|
||||
sessionId: "s1",
|
||||
url: "https://example.com",
|
||||
device: "iphone-14",
|
||||
});
|
||||
expect(calls[0].args).toEqual([
|
||||
"session",
|
||||
"start",
|
||||
"--width",
|
||||
"1280",
|
||||
"--height",
|
||||
"800",
|
||||
"--no-focus",
|
||||
]);
|
||||
expect(calls[1].args).toEqual(["emulate", "--session", "s1", "--device", "iphone-14"]);
|
||||
expect(calls[2].args).toEqual(["navigate", "--session", "s1", "https://example.com"]);
|
||||
});
|
||||
|
||||
it("requires width and height together", async () => {
|
||||
const { tools } = setup({ "session start": START_REPLY("s1") });
|
||||
const tool = tools.get("browser_session_start");
|
||||
await expect(tool?.execute({ width: 1280 }, makeExec())).rejects.toThrow(/together/);
|
||||
});
|
||||
|
||||
it("cleans up a half-initialized session when the initial navigate fails", async () => {
|
||||
const { tools, calls, registry } = setup({
|
||||
"session start": START_REPLY("s1"),
|
||||
"session stop": {},
|
||||
});
|
||||
// navigate has no canned response -> non-zero exit -> error.
|
||||
const tool = tools.get("browser_session_start");
|
||||
await expect(tool?.execute({ url: "https://example.com" }, makeExec())).rejects.toThrow(
|
||||
/navigate/,
|
||||
);
|
||||
expect(calls.some((c) => c.args.join(" ") === "session stop s1")).toBe(true);
|
||||
expect(registry.current()).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("multi-session behavior", () => {
|
||||
const responses = {
|
||||
"session start": START_REPLY("s1"),
|
||||
snapshot: SNAPSHOT_REPLY,
|
||||
};
|
||||
|
||||
it("operation tools default to the current session and echo it back", async () => {
|
||||
const { tools, calls } = setup(responses);
|
||||
await startSession(tools);
|
||||
const snapshot = tools.get("browser_snapshot");
|
||||
const value = (await snapshot?.execute({}, makeExec())) as {
|
||||
session: string;
|
||||
refCount: number;
|
||||
};
|
||||
expect(value).toMatchObject({ session: "s1", refCount: 1 });
|
||||
expect(calls[1].args).toEqual(["snapshot", "--session", "s1"]);
|
||||
});
|
||||
|
||||
it("an explicit session wins and becomes the new current session", async () => {
|
||||
const { tools, calls, registry } = setup(responses);
|
||||
await startSession(tools);
|
||||
const snapshot = tools.get("browser_snapshot");
|
||||
const value = (await snapshot?.execute({ session: "other" }, makeExec())) as {
|
||||
session: string;
|
||||
};
|
||||
expect(value.session).toBe("other");
|
||||
expect(calls[1].args).toEqual(["snapshot", "--session", "other"]);
|
||||
expect(registry.current()).toBe("other");
|
||||
});
|
||||
|
||||
it("fails with guidance when no session exists", async () => {
|
||||
const { tools } = setup(responses);
|
||||
const snapshot = tools.get("browser_snapshot");
|
||||
await expect(snapshot?.execute({}, makeExec())).rejects.toThrow(/browser_session_start/);
|
||||
});
|
||||
|
||||
it("enforces the configured session cap", async () => {
|
||||
const { tools } = setup(
|
||||
{ "session start": seq([START_REPLY("s1"), START_REPLY("s2")]) },
|
||||
{},
|
||||
1,
|
||||
);
|
||||
await startSession(tools);
|
||||
const start = tools.get("browser_session_start");
|
||||
await expect(start?.execute({}, makeExec())).rejects.toThrow(/session limit/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("browser_session_stop / list", () => {
|
||||
it("stops the current session and clears the registry", async () => {
|
||||
const { tools, calls, registry } = setup({
|
||||
"session start": START_REPLY("s1"),
|
||||
"session stop": {},
|
||||
});
|
||||
await startSession(tools);
|
||||
const stop = tools.get("browser_session_stop");
|
||||
const value = (await stop?.execute({}, makeExec())) as { stopped: string };
|
||||
expect(value.stopped).toBe("s1");
|
||||
expect(calls[1].args).toEqual(["session", "stop", "s1"]);
|
||||
expect(registry.current()).toBeUndefined();
|
||||
});
|
||||
|
||||
it("lists daemon sessions with the current marker", async () => {
|
||||
const { tools } = setup({
|
||||
"session start": START_REPLY("s1"),
|
||||
"session list": [
|
||||
{ session_id: "s1", browser_instance_id: "chrome-1" },
|
||||
{ session_id: "s2", browser_instance_id: "chrome-1" },
|
||||
],
|
||||
});
|
||||
await startSession(tools);
|
||||
const list = tools.get("browser_session_list");
|
||||
const value = (await list?.execute({}, makeExec())) as {
|
||||
sessions: { sessionId: string; current: boolean }[];
|
||||
};
|
||||
expect(value.sessions).toEqual([
|
||||
{ sessionId: "s1", browserInstanceId: "chrome-1", current: true },
|
||||
{ sessionId: "s2", browserInstanceId: "chrome-1", current: false },
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("interaction tools", () => {
|
||||
const responses = {
|
||||
"session start": START_REPLY("s1"),
|
||||
click: { tab_id: 7, used_ref: "@e1", x: 10, y: 20 },
|
||||
fill: { tab_id: 7, used_ref: "@e2", value_length: 5 },
|
||||
press: { tab_id: 7, key: "Enter", code: "Enter", modifiers: [] },
|
||||
navigate: { tab_id: 7, url: "https://a.test", final_url: "https://b.test", reached: "load" },
|
||||
emulate: { tab_id: 7, cleared: false },
|
||||
};
|
||||
|
||||
it("browser_click maps target, button, and click count", async () => {
|
||||
const { tools, calls } = setup(responses);
|
||||
await startSession(tools);
|
||||
const click = tools.get("browser_click");
|
||||
const value = (await click?.execute(
|
||||
{ target: "@e1", button: "right", clickCount: 2 },
|
||||
makeExec(),
|
||||
)) as { session: string; x: number; y: number };
|
||||
expect(value).toMatchObject({ session: "s1", x: 10, y: 20 });
|
||||
expect(calls[1].args).toEqual([
|
||||
"click",
|
||||
"--session",
|
||||
"s1",
|
||||
"--button",
|
||||
"right",
|
||||
"--click-count",
|
||||
"2",
|
||||
"@e1",
|
||||
]);
|
||||
});
|
||||
|
||||
it("browser_fill passes value and --no-clear", async () => {
|
||||
const { tools, calls } = setup(responses);
|
||||
await startSession(tools);
|
||||
const fill = tools.get("browser_fill");
|
||||
await fill?.execute({ target: "e2", value: "hello", noClear: true }, makeExec());
|
||||
expect(calls[1].args).toEqual([
|
||||
"fill",
|
||||
"--session",
|
||||
"s1",
|
||||
"--value",
|
||||
"hello",
|
||||
"--no-clear",
|
||||
"e2",
|
||||
]);
|
||||
});
|
||||
|
||||
it("browser_press picks --ref for snapshot refs and --selector otherwise", async () => {
|
||||
const { tools, calls } = setup(responses);
|
||||
await startSession(tools);
|
||||
const press = tools.get("browser_press");
|
||||
await press?.execute({ key: "Enter", target: "@e3" }, makeExec());
|
||||
expect(calls[1].args).toEqual(["press", "--session", "s1", "--ref", "@e3", "Enter"]);
|
||||
await press?.execute({ key: "Ctrl+A", target: "#field" }, makeExec());
|
||||
expect(calls[2].args).toEqual(["press", "--session", "s1", "--selector", "#field", "Ctrl+A"]);
|
||||
});
|
||||
|
||||
it("browser_navigate converts waitUntil and timeoutMs", async () => {
|
||||
const { tools, calls } = setup(responses);
|
||||
await startSession(tools);
|
||||
const navigate = tools.get("browser_navigate");
|
||||
const value = (await navigate?.execute(
|
||||
{ url: "https://a.test", waitUntil: "networkidle", timeoutMs: 1500 },
|
||||
makeExec(),
|
||||
)) as { finalUrl: string; reached: string };
|
||||
expect(value).toMatchObject({ finalUrl: "https://b.test", reached: "load" });
|
||||
expect(calls[1].args).toEqual([
|
||||
"navigate",
|
||||
"--session",
|
||||
"s1",
|
||||
"--wait-until",
|
||||
"networkidle",
|
||||
"--timeout",
|
||||
"1500ms",
|
||||
"https://a.test",
|
||||
]);
|
||||
});
|
||||
|
||||
it("browser_emulate rejects off combined with overrides", async () => {
|
||||
const { tools } = setup(responses);
|
||||
await startSession(tools);
|
||||
const emulate = tools.get("browser_emulate");
|
||||
await expect(emulate?.execute({ off: true, device: "iphone-14" }, makeExec())).rejects.toThrow(
|
||||
/mutually exclusive/,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("browser_screenshot", () => {
|
||||
function pngFile(): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), "bsk-test-"));
|
||||
const path = join(dir, "shot.png");
|
||||
// Minimal PNG header bytes are enough: the plugin never decodes the image.
|
||||
writeFileSync(path, Buffer.from([0x89, 0x50, 0x4e, 0x47, 1, 2, 3, 4]));
|
||||
return path;
|
||||
}
|
||||
|
||||
it("returns the PNG path when no attachment store is mounted", async () => {
|
||||
const path = pngFile();
|
||||
const { tools, calls } = setup({
|
||||
"session start": START_REPLY("s1"),
|
||||
screenshot: { tab_id: 7, width: 800, height: 600, format: "png", path, byte_size: 8 },
|
||||
});
|
||||
await startSession(tools);
|
||||
const screenshot = tools.get("browser_screenshot");
|
||||
const value = (await screenshot?.execute({ ref: "@e1" }, makeExec())) as {
|
||||
path: string;
|
||||
image?: unknown;
|
||||
};
|
||||
expect(value.path).toBe(path);
|
||||
expect(value.image).toBeUndefined();
|
||||
expect(calls[1].args.slice(0, 3)).toEqual(["screenshot", "--session", "s1"]);
|
||||
expect(calls[1].args).toContain("--ref");
|
||||
const rendered = screenshot?.output.render({ session: "s1" }, value as never);
|
||||
expect(rendered?.every((block) => block.type === "text")).toBe(true);
|
||||
});
|
||||
|
||||
it("commits the image through the attachment store on an image-capable route", async () => {
|
||||
const path = pngFile();
|
||||
const services = {
|
||||
attachments: {
|
||||
imageLimits: {
|
||||
mediaTypes: ["image/png"],
|
||||
maxImageBytes: 10_000_000,
|
||||
maxMessageImageBytes: 10_000_000,
|
||||
},
|
||||
saveImage: async (input: { data: Uint8Array; mediaType: string; name?: string }) => ({
|
||||
attachmentId: "att-1",
|
||||
mediaType: input.mediaType,
|
||||
bytes: input.data.byteLength,
|
||||
width: 800,
|
||||
height: 600,
|
||||
name: input.name,
|
||||
}),
|
||||
},
|
||||
llm: {
|
||||
resolveModelInfo: async () => ({ inputModalities: ["text", "image"] }),
|
||||
},
|
||||
};
|
||||
const { tools } = setup(
|
||||
{
|
||||
"session start": START_REPLY("s1"),
|
||||
screenshot: { tab_id: 7, width: 800, height: 600, format: "png", path, byte_size: 8 },
|
||||
},
|
||||
services,
|
||||
);
|
||||
await startSession(tools);
|
||||
const exec = makeExec({
|
||||
agent: {
|
||||
session: { requestHeader: () => ({ config: { provider: "deepseek", model: "vl" } }) },
|
||||
options: {},
|
||||
},
|
||||
});
|
||||
const screenshot = tools.get("browser_screenshot");
|
||||
const value = (await screenshot?.execute({}, exec)) as {
|
||||
image?: { attachmentId: string; mediaType: string };
|
||||
};
|
||||
expect(value.image).toMatchObject({ attachmentId: "att-1", mediaType: "image/png" });
|
||||
const rendered = screenshot?.output.render({}, value as never) as { type: string }[];
|
||||
expect(rendered.map((block) => block.type)).toEqual(["text", "image"]);
|
||||
});
|
||||
|
||||
it("stays path-only when the model route is text-only", async () => {
|
||||
const path = pngFile();
|
||||
const services = {
|
||||
attachments: {
|
||||
imageLimits: { mediaTypes: ["image/png"], maxImageBytes: 1e7, maxMessageImageBytes: 1e7 },
|
||||
saveImage: async () => {
|
||||
throw new Error("must not be called");
|
||||
},
|
||||
},
|
||||
llm: { resolveModelInfo: async () => ({ inputModalities: ["text"] }) },
|
||||
};
|
||||
const { tools } = setup(
|
||||
{
|
||||
"session start": START_REPLY("s1"),
|
||||
screenshot: { tab_id: 7, width: 1, height: 1, format: "png", path, byte_size: 8 },
|
||||
},
|
||||
services,
|
||||
);
|
||||
await startSession(tools);
|
||||
const exec = makeExec({
|
||||
agent: {
|
||||
session: { requestHeader: () => ({ config: { provider: "p", model: "m" } }) },
|
||||
options: {},
|
||||
},
|
||||
});
|
||||
const screenshot = tools.get("browser_screenshot");
|
||||
const value = (await screenshot?.execute({}, exec)) as { image?: unknown; path: string };
|
||||
expect(value.image).toBeUndefined();
|
||||
expect(value.path).toBe(path);
|
||||
});
|
||||
});
|
||||
|
||||
describe("error and cancellation semantics", () => {
|
||||
it("surfaces the bsk JSON error envelope with code and hint", async () => {
|
||||
const { tools } = setup({
|
||||
"session start": START_REPLY("s1"),
|
||||
});
|
||||
await startSession(tools);
|
||||
const snapshot = tools.get("browser_snapshot");
|
||||
await expect(snapshot?.execute({}, makeExec())).rejects.toThrow(/no canned response/);
|
||||
});
|
||||
|
||||
it("turns an aborted child into an AbortError", async () => {
|
||||
const { tools } = setup({ "session start": START_REPLY("s1") });
|
||||
await startSession(tools);
|
||||
const snapshot = tools.get("browser_snapshot");
|
||||
const controller = new AbortController();
|
||||
controller.abort();
|
||||
const exec = makeExec({ signal: controller.signal });
|
||||
await expect(snapshot?.execute({}, exec)).rejects.toMatchObject({ name: "AbortError" });
|
||||
});
|
||||
|
||||
it("maps a missing bsk binary onto install guidance", async () => {
|
||||
const { ctx, tools } = makeCtx();
|
||||
const registry = new SessionRegistry(5);
|
||||
const runner = {
|
||||
async run(): Promise<BskRunResult> {
|
||||
throw Object.assign(new Error("spawn bsk ENOENT"), { code: "ENOENT" });
|
||||
},
|
||||
killAll() {},
|
||||
};
|
||||
registerTools({ ctx: ctx as never, runner, registry, config: CONFIG });
|
||||
const list = tools.get("browser_session_list");
|
||||
await expect(list?.execute({}, makeExec())).rejects.toThrow(/BrowserSkill must be installed/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("presentation", () => {
|
||||
it("presents calls as terminal cards with the bsk command line", () => {
|
||||
const { tools } = setup({});
|
||||
const click = tools.get("browser_click");
|
||||
const view = click?.presentCall?.({ target: "@e1" }) as {
|
||||
card: string;
|
||||
title: string;
|
||||
};
|
||||
expect(view.card).toBe("terminal");
|
||||
expect(view.title).toContain("bsk click @e1");
|
||||
});
|
||||
|
||||
it("presents results as terminal cards carrying the rendered text", () => {
|
||||
const { tools } = setup({});
|
||||
const click = tools.get("browser_click");
|
||||
const view = click?.presentResult?.({ target: "@e1" }, {
|
||||
content: [{ type: "text", text: "clicked" }],
|
||||
isError: false,
|
||||
} as never) as { card: string; output: string; exitCode: number };
|
||||
expect(view).toMatchObject({ card: "terminal", output: "clicked", exitCode: 0 });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "Bundler",
|
||||
"noEmit": true,
|
||||
"esModuleInterop": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true,
|
||||
"strict": true,
|
||||
"skipLibCheck": true,
|
||||
"types": ["node"]
|
||||
},
|
||||
"include": ["src/**/*", "tests/**/*", "tsdown.config.ts", "vitest.config.ts"]
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
import { defineConfig } from "tsdown";
|
||||
|
||||
export default defineConfig({
|
||||
entry: ["src/index.ts"],
|
||||
format: ["esm"],
|
||||
dts: true,
|
||||
outDir: "lib",
|
||||
// dsh host packages are provided by the profile the plugin is installed into.
|
||||
external: [/^@deepseek-ai\//],
|
||||
});
|
||||
@@ -0,0 +1,8 @@
|
||||
import { defineConfig } from "vitest/config";
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
environment: "node",
|
||||
include: ["tests/**/*.test.ts"],
|
||||
},
|
||||
});
|
||||
Reference in New Issue
Block a user