feat: add bsk record for semantic user-action traces

Capture user actions in the Agent Window and export a record-only
trace (pages[] + navigate/click/fill/select/press steps with target
descriptors). Adds the record overlay UI aligned with the control
overlay, and wires record start/stop/await through CLI, daemon,
protocol, and extension.
This commit is contained in:
“haonan”
2026-07-22 11:27:30 +08:00
parent ccf3c32140
commit ac028a7a18
47 changed files with 6038 additions and 46 deletions
+35 -2
View File
@@ -46,6 +46,21 @@ Optional: `bsk session start --browser <instance-id-or-label>` when multiple bro
Emergency cleanup: `bsk session stop --all` or the Agent Window overlay **Stop all**.
## Stop when the goal is met
Every task is a **bounded goal**, not open-ended browsing. The goal may come from the user's request, a recorded `trace.json`, or both.
1. **Define success first** — one concrete, observable condition derived from the user's words, `purpose`, or the last meaningful step in a trace (e.g. "form submitted", "item added to cart", "playback started").
2. **Take the shortest path** — snapshot → act → at most one check. Do not wander, re-try unrelated actions, or stack exploratory steps.
3. **Stop as soon as success is reached** — run `bsk session stop <id>` immediately unless the user explicitly asked to keep the session open (e.g. "don't close yet", "keep browsing").
4. **No post-success work** — once the goal is met, do not click, refresh, navigate, re-search, switch tabs, or "double-check" that it worked. Further verification is a new task.
5. **When blocked, pause — do not brute-force** — if the page requires human input (login, captcha, OTP, payment confirmation) or an action fails twice with no progress, call `bsk request-help` instead of retrying blindly. See **Ask the human for help** below.
6. **When unsure** — at most one extra `bsk snapshot`. If success looks met, stop. If not, ask the user; do not keep clicking.
**With a trace:** replay steps in order using `target` role/name/tag and raw `value`/`selection` fields. After the last step (or when its `effect.navigated_to` / success hint is satisfied), apply rules 3–4 immediately. The trace guides execution; it does not extend control beyond the goal.
**Without a trace:** the user's request *is* the success condition. Satisfying it ends the task — same stop rules apply.
## Core interaction loop
Write operations only affect tabs in the **Agent Window** (or tabs you **borrowed** into it).
@@ -191,6 +206,23 @@ acts. The result `outcome` is one of:
`note` carries any text the user typed back. `resolved_targets` reports
which refs/selectors matched a live element.
### Recording — `bsk record`
Capture the user's own actions in the Agent Window to a `trace.json`, for later LLM-driven automation:
```bash
bsk record start --url https://… [--purpose "publish a wiki doc"] [--output trace.json]
# `--url` is required on a fresh session (about:blank cannot host the content script).
# Blocks until the user clicks Finish in the recording panel, then writes ./trace.json and closes the window.
bsk record stop [--output trace.json] # terminal fallback if the browser panel is unavailable
```
- The trace is a **record-only action log** (a `pages[]` dictionary + `navigate`/`click`/`fill`/`select`/`press` steps with `target` descriptors). It records *what the user did*; deciding which inputs are variable is left to the executing agent.
- `--purpose` is optional context metadata; it does **not** change what gets captured.
- There is **no** `bsk replay` — to redo a flow, read the trace and reuse the existing `session` / `snapshot` / `@eN` / `click` / `fill` tools. Follow **Stop when the goal is met**.
- Do **not** record on banking/SSO/password-manager pages; passwords are redacted but traces may still contain sensitive text.
## Error handling
### Exit codes (`echo $?` after `bsk …`)
@@ -221,8 +253,9 @@ Always **`bsk session stop <id>`** in a `finally`-style path so the Agent Window
1. **No token theft** — do not `bsk evaluate` on sensitive sites to read `localStorage`, cookies, or auth headers for exfiltration.
2. **No long borrow** — do not leave a user's personal tab in the Agent Window across unrelated tasks.
3. **No skip stop** — always `bsk session stop <id>`; never assume idle timeout will clean up.
4. **No observe escalation before snapshot** — use `bsk snapshot` first; only use `bsk get-html` or `bsk screenshot` when the snapshot is insufficient. Element screenshots (`--ref @eN`) still require a fresh snapshot ref — never skip snapshot just to grab a visual.
5. **`evaluate` is powerful and risky** — use only when snapshot + click/fill/select cannot suffice; never on credential surfaces.
4. **No post-success control** — once the user’s goal (or last trace step) is met, do not keep operating the page; stop the session unless they asked to keep it open.
5. **No observe escalation before snapshot** — use `bsk snapshot` first; only use `bsk get-html` or `bsk screenshot` when the snapshot is insufficient. Element screenshots (`--ref @eN`) still require a fresh snapshot ref — never skip snapshot just to grab a visual.
6. **`evaluate` is powerful and risky** — use only when snapshot + click/fill/select cannot suffice; never on credential surfaces.
---
+6
View File
@@ -19,6 +19,8 @@ pub mod interaction;
pub mod logs;
pub mod navigate;
pub mod network;
pub mod record;
pub mod record_state;
pub mod render_error;
pub mod screenshot;
pub mod session;
@@ -39,6 +41,7 @@ use crate::cli::install_skill::InstallSkillArgs;
use crate::cli::interaction::{ClickArgs, FillArgs, PressArgs, SelectArgs};
use crate::cli::navigate::{NavigateCommand, NavigateHistoryArgs, ReloadArgs};
use crate::cli::network::NetworkArgs;
use crate::cli::record::RecordCmd;
use crate::cli::screenshot::ScreenshotArgs;
use crate::cli::session::SessionCmd;
use crate::cli::snapshot::SnapshotArgs;
@@ -169,6 +172,9 @@ pub enum Command {
/// Ask the human to complete an in-page step (captcha / login / confirm).
#[command(name = "request-help")]
RequestHelp(RequestHelpArgs),
/// Record user actions into a semantic `trace.json` textbook for LLMs.
Record(RecordCmd),
}
#[derive(Debug, Clone, Args, Default)]
+256
View File
@@ -0,0 +1,256 @@
//! `bsk record start|stop` — capture user actions in the Agent Window.
use std::fs;
use std::path::PathBuf;
use std::time::Duration;
use anyhow::Context;
use bsk_protocol::Method;
use bsk_protocol::tools::{
RecordAwaitParams, RecordAwaitResult, RecordStartParams, RecordStartResult, RecordStopParams,
RecordStopResult, Trace,
};
use clap::{Args, Subcommand};
use crate::cli::TOOL_IPC_TIMEOUT;
use crate::cli::business_rpc;
use crate::cli::ensure_daemon::ensure_daemon;
use crate::cli::error::{CliError, Format};
use crate::cli::record_state;
use crate::cli::session::{start_session, stop_session};
/// Max wait for the user to click 结束 in the browser (24 hours).
const RECORD_AWAIT_TIMEOUT_MS: u32 = 86_400_000;
/// `bsk record …` subcommand tree.
#[derive(Debug, Clone, Args)]
pub struct RecordCmd {
#[command(subcommand)]
pub sub: RecordSub,
}
#[derive(Debug, Clone, Subcommand)]
pub enum RecordSub {
/// Open the Agent Window, record user actions, and block until finished in the browser.
Start(RecordStartArgs),
/// Stop recording from the terminal (fallback), write trace JSON, and close the window.
/// Works even while `record start` is blocked in `record_await` (daemon forwards
/// `tool.record_stop` without the per-session busy gate).
Stop(RecordStopArgs),
}
#[derive(Debug, Clone, Args)]
pub struct RecordStartArgs {
#[arg(long = "tab-id")]
pub tab_id: Option<i64>,
/// Navigate to this http(s) URL before recording (required for a fresh
/// session — Agent Window starts on about:blank, which cannot host the
/// content script).
#[arg(long)]
pub url: Option<String>,
/// Optional goal text stored on the exported trace for LLM context.
#[arg(long)]
pub purpose: Option<String>,
/// Output path for the trace JSON (default `./trace.json`).
#[arg(long, default_value = "trace.json")]
pub output: PathBuf,
}
#[derive(Debug, Clone, Args)]
pub struct RecordStopArgs {
/// Output path for the trace JSON (default `./trace.json`).
#[arg(long, default_value = "trace.json")]
pub output: PathBuf,
}
pub fn dispatch(cmd: RecordCmd, format: Format) -> Result<(), CliError> {
match cmd.sub {
RecordSub::Start(args) => dispatch_start(args, format),
RecordSub::Stop(args) => dispatch_stop(args, format),
}
}
fn dispatch_start(args: RecordStartArgs, format: Format) -> Result<(), CliError> {
if record_state::read().is_ok() {
return Err(CliError::Local(anyhow::anyhow!(
"a recording is already in progress; run `bsk record stop` first"
)));
}
let info = ensure_daemon().context("ensure daemon is running")?;
let session = start_session(info.sock_path.clone(), None)?;
let start_params = RecordStartParams {
session_id: session.session_id.clone(),
tab_id: args.tab_id,
url: args.url,
purpose: args.purpose.clone(),
};
let start_result = business_rpc::call::<RecordStartParams, RecordStartResult>(
info.sock_path.clone(),
"record-start",
Method::ToolRecordStart,
Some(start_params),
TOOL_IPC_TIMEOUT,
);
let start_result = match start_result {
Ok(result) => result,
Err(err) => {
let _ = stop_session(info.sock_path, &session.session_id);
return Err(err);
}
};
if let Err(err) = record_state::write(&session.session_id) {
let _ = stop_session(info.sock_path.clone(), &session.session_id);
return Err(CliError::Local(err));
}
if format == Format::Human {
println!(
"recording on tab={} — click 结束 in the browser when done",
start_result.tab_id
);
}
let await_params = RecordAwaitParams {
session_id: session.session_id.clone(),
timeout_ms: Some(RECORD_AWAIT_TIMEOUT_MS),
};
let await_result = business_rpc::call::<RecordAwaitParams, RecordAwaitResult>(
info.sock_path.clone(),
"record-await",
Method::ToolRecordAwait,
Some(await_params),
record_await_ipc_timeout(RECORD_AWAIT_TIMEOUT_MS),
);
// Keep write/render inside a Result so `?` cannot skip session teardown.
let run_result: Result<(), CliError> = match await_result {
Ok(result) => (|| {
write_trace_file(&args.output, &result.trace)?;
render_finish(&result.trace, &args.output, format)
})(),
Err(err) => Err(err),
};
let session_stop_result = stop_session(info.sock_path, &session.session_id);
record_state::clear();
run_result?;
session_stop_result?;
Ok(())
}
fn dispatch_stop(args: RecordStopArgs, format: Format) -> Result<(), CliError> {
let state = record_state::read().map_err(CliError::Local)?;
let info = ensure_daemon().context("ensure daemon is running")?;
let session_id = state.session_id.clone();
let run_result: Result<(), CliError> = (|| {
let params = RecordStopParams {
session_id: session_id.clone(),
};
let result = business_rpc::call::<RecordStopParams, RecordStopResult>(
info.sock_path.clone(),
"record-stop",
Method::ToolRecordStop,
Some(params),
TOOL_IPC_TIMEOUT,
)?;
write_trace_file(&args.output, &result.trace)?;
render_stop(&result, &args.output, format)
})();
let session_stop_result = stop_session(info.sock_path, &session_id);
record_state::clear();
run_result?;
session_stop_result?;
Ok(())
}
fn record_await_ipc_timeout(timeout_ms: u32) -> Duration {
Duration::from_millis(u64::from(timeout_ms))
.checked_add(Duration::from_secs(15))
.unwrap_or(Duration::from_secs(u64::from(timeout_ms / 1_000) + 15))
}
fn write_trace_file(output: &PathBuf, trace: &Trace) -> Result<(), CliError> {
let json = serde_json::to_string_pretty(trace)
.context("serialize trace JSON")
.map_err(CliError::Local)?;
if let Some(parent) = output.parent() {
if !parent.as_os_str().is_empty() {
fs::create_dir_all(parent)
.with_context(|| format!("create output directory {}", parent.display()))
.map_err(CliError::Local)?;
}
}
fs::write(output, format!("{json}\n"))
.with_context(|| format!("write trace to {}", output.display()))
.map_err(CliError::Local)?;
Ok(())
}
fn render_finish(trace: &Trace, output: &PathBuf, format: Format) -> Result<(), CliError> {
match format {
Format::Json => {
println!(
"{}",
serde_json::to_string(&serde_json::json!({
"output": output,
"trace": trace,
"window_closed": true,
}))
.map_err(|e| CliError::Local(anyhow::anyhow!(e)))?
);
}
Format::Human => {
println!("saved {} steps to {}", trace.steps.len(), output.display());
}
}
Ok(())
}
fn render_stop(
result: &RecordStopResult,
output: &PathBuf,
format: Format,
) -> Result<(), CliError> {
render_finish(&result.trace, output, format)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn default_output_is_trace_json() {
let args = RecordStopArgs {
output: PathBuf::from("trace.json"),
};
assert_eq!(args.output, PathBuf::from("trace.json"));
}
#[test]
fn start_args_default_output_is_trace_json() {
let args = RecordStartArgs {
tab_id: None,
url: None,
purpose: None,
output: PathBuf::from("trace.json"),
};
assert_eq!(args.output, PathBuf::from("trace.json"));
}
#[test]
fn record_await_ipc_timeout_covers_long_wait() {
let got = record_await_ipc_timeout(RECORD_AWAIT_TIMEOUT_MS);
assert!(got >= Duration::from_secs(86_400));
}
}
+117
View File
@@ -0,0 +1,117 @@
//! Persist the session id between `bsk record start` and `bsk record stop`.
use std::fs;
use std::time::{SystemTime, UNIX_EPOCH};
use anyhow::{Context, Result};
use serde::{Deserialize, Serialize};
use crate::daemon::paths;
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct RecordSessionState {
pub session_id: String,
pub started_at: String,
}
pub fn write(session_id: &str) -> Result<()> {
let path = paths::record_session_path()?;
paths::ensure_bsk_home()?;
if path.exists() {
return Err(anyhow::anyhow!(
"a recording is already in progress; run `bsk record stop` first"
));
}
let state = RecordSessionState {
session_id: session_id.to_string(),
started_at: started_at_unix_ms(),
};
let json = serde_json::to_string_pretty(&state).context("serialize record session state")?;
fs::write(&path, format!("{json}\n")).with_context(|| format!("write {}", path.display()))?;
Ok(())
}
pub fn read() -> Result<RecordSessionState> {
let path = paths::record_session_path()?;
if !path.exists() {
return Err(anyhow::anyhow!(
"no recording in progress; run `bsk record start` first"
));
}
let raw = fs::read_to_string(&path).with_context(|| format!("read {}", path.display()))?;
serde_json::from_str(&raw).with_context(|| format!("parse {}", path.display()))
}
pub fn clear() {
if let Ok(path) = paths::record_session_path() {
let _ = fs::remove_file(path);
}
}
fn started_at_unix_ms() -> String {
let ms = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_millis();
format!("{ms}")
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Mutex;
fn env_guard() -> &'static Mutex<()> {
static GUARD: Mutex<()> = Mutex::new(());
&GUARD
}
fn with_temp_home<F: FnOnce()>(f: F) {
let _lock = env_guard().lock().unwrap_or_else(|e| e.into_inner());
let tmp = tempfile::tempdir().unwrap();
unsafe {
std::env::set_var(crate::daemon::paths::BSK_HOME_ENV, tmp.path());
}
f();
clear();
unsafe {
std::env::remove_var(crate::daemon::paths::BSK_HOME_ENV);
}
}
#[test]
fn write_read_round_trips() {
with_temp_home(|| {
write("abcd").unwrap();
let state = read().unwrap();
assert_eq!(state.session_id, "abcd");
assert!(!state.started_at.is_empty());
});
}
#[test]
fn write_rejects_duplicate() {
with_temp_home(|| {
write("abcd").unwrap();
let err = write("efgh").unwrap_err();
assert!(err.to_string().contains("already in progress"));
});
}
#[test]
fn read_errors_when_missing() {
with_temp_home(|| {
let err = read().unwrap_err();
assert!(err.to_string().contains("no recording in progress"));
});
}
#[test]
fn clear_removes_state_file() {
with_temp_home(|| {
write("abcd").unwrap();
clear();
assert!(read().is_err());
});
}
}
+39 -21
View File
@@ -63,11 +63,11 @@ struct StartParams {
}
#[derive(Debug, Deserialize)]
struct StartReply {
session_id: String,
browser_instance_id: String,
pub struct StartReply {
pub session_id: String,
pub browser_instance_id: String,
#[serde(default)]
agent_window_id: Option<i64>,
pub agent_window_id: Option<i64>,
}
#[derive(Debug, Serialize)]
@@ -78,21 +78,21 @@ struct StopParams {
}
#[derive(Debug, Deserialize)]
struct StopReply {
stopped: Vec<String>,
pub struct StopReply {
pub stopped: Vec<String>,
#[serde(default)]
failed: Vec<StopFailure>,
pub failed: Vec<StopFailure>,
#[serde(default)]
returned_tab_ids: Vec<i64>,
pub returned_tab_ids: Vec<i64>,
#[serde(default)]
return_failures: Vec<ReturnFailure>,
pub return_failures: Vec<ReturnFailure>,
}
#[derive(Debug, Deserialize, Serialize)]
struct StopFailure {
session_id: String,
code: ErrorCode,
message: String,
pub struct StopFailure {
pub session_id: String,
pub code: ErrorCode,
pub message: String,
}
#[derive(Debug, Deserialize)]
@@ -113,14 +113,7 @@ pub fn dispatch(cmd: SessionCmd, format: Format) -> Result<(), CliError> {
}
fn run_start(sock: PathBuf, args: SessionStartArgs, format: Format) -> Result<(), CliError> {
let result: Result<StartReply, CliError> = call(
sock,
Method::SessionStart,
Some(StartParams {
browser_instance_id: args.browser,
}),
Duration::from_secs(30),
);
let result = start_session(sock, args.browser);
match result {
Ok(reply) => match format {
Format::Json => {
@@ -143,6 +136,31 @@ fn run_start(sock: PathBuf, args: SessionStartArgs, format: Format) -> Result<()
Ok(())
}
/// Start a session and open the Agent Window. Used by `session start` and `record start`.
pub fn start_session(sock: PathBuf, browser: Option<String>) -> Result<StartReply, CliError> {
call(
sock,
Method::SessionStart,
Some(StartParams {
browser_instance_id: browser,
}),
Duration::from_secs(30),
)
}
/// Stop a single session by id.
pub fn stop_session(sock: PathBuf, session_id: &str) -> Result<StopReply, CliError> {
call(
sock,
Method::SessionStop,
Some(StopParams {
session_id: Some(session_id.to_string()),
all: false,
}),
SESSION_STOP_IPC_TIMEOUT,
)
}
/// Render a `session.start` failure (review I3).
///
/// The previous implementation hand-rolled `eprintln!("error: ...")`
+17 -5
View File
@@ -242,7 +242,10 @@ pub fn full_handler(status: DaemonStatus, state: Arc<DaemonState>) -> RpcHandler
| Method::ToolSelect
| Method::ToolEvaluate
| Method::ToolWaitForNavigation
| Method::ToolRequestHelp => {
| Method::ToolRequestHelp
| Method::ToolRecordStart
| Method::ToolRecordStop
| Method::ToolRecordAwait => {
handle_tool_dispatch(&state, rpc_id, method, params).await
}
Method::ToolWaitMs => handle_wait_ms(&state.abort_registry, rpc_id, params).await,
@@ -319,10 +322,19 @@ async fn handle_tool_dispatch(
}
};
let entry = inflight_guard.entry();
let outcome = state
.tool_queues
.dispatch(&session_id, method, params, timeout, Some(entry))
.await;
// `record_stop` must reach the extension while `record_await` holds the
// serial busy lock — finishing the recording unblocks await.
let outcome = if method == Method::ToolRecordStop {
state
.tool_queues
.dispatch_unlocked(&session_id, method, params, timeout, Some(entry))
.await
} else {
state
.tool_queues
.dispatch(&session_id, method, params, timeout, Some(entry))
.await
};
drop(inflight_guard);
match outcome {
Ok(v) => ResponseBody::Ok(v),
+9
View File
@@ -95,6 +95,11 @@ pub fn sock_path() -> Result<PathBuf> {
Ok(bsk_home()?.join("run").join("daemon.sock"))
}
/// Path to the in-progress recording session state (`record-session.json`).
pub fn record_session_path() -> Result<PathBuf> {
Ok(bsk_home()?.join("record-session.json"))
}
/// Windows named-pipe name. Include the resolved `BSK_HOME` path in the
/// token so test homes and custom installs do not share a predictable
/// per-username pipe.
@@ -163,6 +168,10 @@ mod tests {
assert_eq!(log_path().unwrap(), home.join("daemon.log"));
assert_eq!(update_check_path().unwrap(), home.join("update-check.json"));
assert_eq!(sock_path().unwrap(), home.join("run").join("daemon.sock"));
assert_eq!(
record_session_path().unwrap(),
home.join("record-session.json")
);
});
}
}
+90
View File
@@ -221,6 +221,13 @@ impl ToolQueueRegistry {
guard.get(sid).is_some_and(|entry| entry.accepting)
}
#[cfg(test)]
fn force_busy_for_tests(&self, sid: &SessionId, busy: bool) {
let guard = self.queues.lock().expect("tool queue registry poisoned");
let entry = guard.get(sid).expect("queue must exist");
entry.state.lock().expect("queue state poisoned").busy = busy;
}
/// Spawn the worker task for a session id. Idempotent: spawning the
/// same id twice replaces the previous queue (the previous worker
/// closes when its sender drops).
@@ -320,6 +327,42 @@ impl ToolQueueRegistry {
outcome
}
/// Forward a tool RPC to the extension without taking the
/// per-session busy lock. Used by `tool.record_stop` so a second
/// CLI can finish an in-flight `tool.record_await` (extension
/// dispatch already runs concurrent `void dispatch(msg)` handlers).
pub async fn dispatch_unlocked(
&self,
sid: &SessionId,
method: Method,
params: Value,
timeout: Duration,
inflight: Option<Arc<ToolInflightEntry>>,
) -> Result<Value, DispatchError> {
{
let guard = self.queues.lock().expect("tool queue registry poisoned");
let Some(entry) = guard.get(sid) else {
return Err(DispatchError::SessionNotFound);
};
if !entry.accepting {
return Err(DispatchError::SessionStopping);
}
}
let (respond_tx, _respond_rx) = oneshot::channel();
let job = ToolJob {
method,
params,
timeout,
respond: respond_tx,
inflight,
session_id: sid.clone(),
};
match forward_one(sid, &self.browsers, &self.sessions, &job).await {
Ok(v) => Ok(v),
Err(err) => Err(DispatchError::Rpc(err)),
}
}
/// Stop accepting new jobs for `sid`, enqueue one final control RPC,
/// and wait for it behind any already-queued tools. Used by
/// `session.stop` so extension teardown cannot race an in-flight
@@ -789,3 +832,50 @@ mod tool_job_session_id_tests {
assert_eq!(job.session_id.0, "sess-A");
}
}
#[cfg(test)]
mod dispatch_unlocked_tests {
use super::*;
use crate::daemon::browsers::BrowserRegistry;
use crate::daemon::sessions::{SessionId, SessionRegistry};
use bsk_protocol::Method;
use serde_json::json;
use std::sync::Arc;
#[tokio::test]
async fn unlocked_dispatch_skips_session_busy_gate() {
let browsers = Arc::new(BrowserRegistry::new());
let sessions = Arc::new(SessionRegistry::new());
let queues = ToolQueueRegistry::new(browsers, sessions);
let sid = SessionId("s-record".into());
queues.spawn(sid.clone());
queues.force_busy_for_tests(&sid, true);
let busy = queues
.dispatch(
&sid,
Method::ToolTabList,
json!({ "session_id": "s-record" }),
Duration::from_secs(1),
None,
)
.await;
assert!(matches!(busy, Err(DispatchError::SessionBusy)));
let unlocked = queues
.dispatch_unlocked(
&sid,
Method::ToolRecordStop,
json!({ "session_id": "s-record" }),
Duration::from_secs(1),
None,
)
.await;
assert!(
!matches!(unlocked, Err(DispatchError::SessionBusy)),
"record_stop must not be rejected as SessionBusy; got {unlocked:?}"
);
// No session row → forward_one returns NotFound as Rpc.
assert!(matches!(unlocked, Err(DispatchError::Rpc(_))));
}
}
+1
View File
@@ -97,6 +97,7 @@ fn dispatch(cli: Cli, format: Format) -> Result<(), CliError> {
Command::WaitForNavigation(args) => cli::waits::dispatch_wait_for_navigation(args, format),
Command::WaitMs(args) => cli::waits::dispatch_wait_ms(args, format),
Command::RequestHelp(args) => cli::human_loop::dispatch(args, format),
Command::Record(cmd) => cli::record::dispatch(cmd, format),
}
}
@@ -0,0 +1,21 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "RecordAwaitParams",
"type": "object",
"required": [
"session_id"
],
"properties": {
"session_id": {
"type": "string"
},
"timeout_ms": {
"type": [
"integer",
"null"
],
"format": "uint32",
"minimum": 0.0
}
}
}
@@ -0,0 +1,417 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "RecordAwaitResult",
"type": "object",
"required": [
"trace"
],
"properties": {
"trace": {
"$ref": "#/definitions/Trace"
}
},
"definitions": {
"KeyModifier": {
"description": "Keyboard modifier flags. Multiple flags may be combined; the extension folds them into CDP's bitfield (`alt=1, ctrl=2, meta=4, shift=8`).",
"type": "string",
"enum": [
"alt",
"ctrl",
"meta",
"shift"
]
},
"PageRef": {
"description": "Page context dictionary entry — referenced by steps via `page` id.",
"type": "object",
"required": [
"id",
"url"
],
"properties": {
"id": {
"type": "string"
},
"title": {
"type": [
"string",
"null"
]
},
"url": {
"type": "string"
}
}
},
"SelectedOption": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
"value"
],
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"value": {
"type": "string"
}
}
},
"Step": {
"description": "One recorded user action — discriminated union by `op`.",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"to"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target",
"value"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"redacted": {
"type": [
"boolean",
"null"
]
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"selection",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOption"
}
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"page"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptor"
},
{
"type": "null"
}
]
}
}
}
]
},
"StepEffect": {
"description": "Observed navigation after a step (objective fact only).",
"type": "object",
"required": [
"navigated_to"
],
"properties": {
"navigated_to": {
"description": "Reference into `pages[]` for the destination page.",
"type": "string"
}
}
},
"TargetDescriptor": {
"description": "Stable semantic handle for an interacted element.\n\n`name` and `nearby_label` are **untrusted page text**.",
"type": "object",
"required": [
"tag"
],
"properties": {
"name": {
"type": [
"string",
"null"
]
},
"name_attr": {
"type": [
"string",
"null"
]
},
"nearby_label": {
"type": [
"string",
"null"
]
},
"placeholder": {
"type": [
"string",
"null"
]
},
"role": {
"type": [
"string",
"null"
]
},
"tag": {
"type": "string"
}
}
},
"Trace": {
"description": "Persisted user-action trace exported by `tool.record_stop` / `await`.",
"type": "object",
"required": [
"entry",
"pages",
"recorded_at",
"steps"
],
"properties": {
"entry": {
"$ref": "#/definitions/TraceEntry"
},
"pages": {
"type": "array",
"items": {
"$ref": "#/definitions/PageRef"
}
},
"purpose": {
"description": "Optional user-provided goal from `--purpose` (metadata only).",
"type": [
"string",
"null"
]
},
"recorded_at": {
"description": "RFC 3339 timestamp when recording stopped.",
"type": "string"
},
"started_at": {
"type": [
"string",
"null"
]
},
"steps": {
"type": "array",
"items": {
"$ref": "#/definitions/Step"
}
}
}
},
"TraceEntry": {
"description": "Recording entry point — first URL the flow starts from.",
"type": "object",
"required": [
"start_url"
],
"properties": {
"start_url": {
"type": "string"
}
}
}
}
}
@@ -0,0 +1,32 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "RecordStartParams",
"type": "object",
"required": [
"session_id"
],
"properties": {
"purpose": {
"type": [
"string",
"null"
]
},
"session_id": {
"type": "string"
},
"tab_id": {
"type": [
"integer",
"null"
],
"format": "int64"
},
"url": {
"type": [
"string",
"null"
]
}
}
}
@@ -0,0 +1,18 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "RecordStartResult",
"type": "object",
"required": [
"recording",
"tab_id"
],
"properties": {
"recording": {
"type": "boolean"
},
"tab_id": {
"type": "integer",
"format": "int64"
}
}
}
@@ -0,0 +1,13 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "RecordStopParams",
"type": "object",
"required": [
"session_id"
],
"properties": {
"session_id": {
"type": "string"
}
}
}
@@ -0,0 +1,417 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "RecordStopResult",
"type": "object",
"required": [
"trace"
],
"properties": {
"trace": {
"$ref": "#/definitions/Trace"
}
},
"definitions": {
"KeyModifier": {
"description": "Keyboard modifier flags. Multiple flags may be combined; the extension folds them into CDP's bitfield (`alt=1, ctrl=2, meta=4, shift=8`).",
"type": "string",
"enum": [
"alt",
"ctrl",
"meta",
"shift"
]
},
"PageRef": {
"description": "Page context dictionary entry — referenced by steps via `page` id.",
"type": "object",
"required": [
"id",
"url"
],
"properties": {
"id": {
"type": "string"
},
"title": {
"type": [
"string",
"null"
]
},
"url": {
"type": "string"
}
}
},
"SelectedOption": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
"value"
],
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"value": {
"type": "string"
}
}
},
"Step": {
"description": "One recorded user action — discriminated union by `op`.",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"to"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target",
"value"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"redacted": {
"type": [
"boolean",
"null"
]
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"selection",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOption"
}
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"page"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptor"
},
{
"type": "null"
}
]
}
}
}
]
},
"StepEffect": {
"description": "Observed navigation after a step (objective fact only).",
"type": "object",
"required": [
"navigated_to"
],
"properties": {
"navigated_to": {
"description": "Reference into `pages[]` for the destination page.",
"type": "string"
}
}
},
"TargetDescriptor": {
"description": "Stable semantic handle for an interacted element.\n\n`name` and `nearby_label` are **untrusted page text**.",
"type": "object",
"required": [
"tag"
],
"properties": {
"name": {
"type": [
"string",
"null"
]
},
"name_attr": {
"type": [
"string",
"null"
]
},
"nearby_label": {
"type": [
"string",
"null"
]
},
"placeholder": {
"type": [
"string",
"null"
]
},
"role": {
"type": [
"string",
"null"
]
},
"tag": {
"type": "string"
}
}
},
"Trace": {
"description": "Persisted user-action trace exported by `tool.record_stop` / `await`.",
"type": "object",
"required": [
"entry",
"pages",
"recorded_at",
"steps"
],
"properties": {
"entry": {
"$ref": "#/definitions/TraceEntry"
},
"pages": {
"type": "array",
"items": {
"$ref": "#/definitions/PageRef"
}
},
"purpose": {
"description": "Optional user-provided goal from `--purpose` (metadata only).",
"type": [
"string",
"null"
]
},
"recorded_at": {
"description": "RFC 3339 timestamp when recording stopped.",
"type": "string"
},
"started_at": {
"type": [
"string",
"null"
]
},
"steps": {
"type": "array",
"items": {
"$ref": "#/definitions/Step"
}
}
}
},
"TraceEntry": {
"description": "Recording entry point — first URL the flow starts from.",
"type": "object",
"required": [
"start_url"
],
"properties": {
"start_url": {
"type": "string"
}
}
}
}
}
+406
View File
@@ -0,0 +1,406 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Trace",
"description": "Persisted user-action trace exported by `tool.record_stop` / `await`.",
"type": "object",
"required": [
"entry",
"pages",
"recorded_at",
"steps"
],
"properties": {
"entry": {
"$ref": "#/definitions/TraceEntry"
},
"pages": {
"type": "array",
"items": {
"$ref": "#/definitions/PageRef"
}
},
"purpose": {
"description": "Optional user-provided goal from `--purpose` (metadata only).",
"type": [
"string",
"null"
]
},
"recorded_at": {
"description": "RFC 3339 timestamp when recording stopped.",
"type": "string"
},
"started_at": {
"type": [
"string",
"null"
]
},
"steps": {
"type": "array",
"items": {
"$ref": "#/definitions/Step"
}
}
},
"definitions": {
"KeyModifier": {
"description": "Keyboard modifier flags. Multiple flags may be combined; the extension folds them into CDP's bitfield (`alt=1, ctrl=2, meta=4, shift=8`).",
"type": "string",
"enum": [
"alt",
"ctrl",
"meta",
"shift"
]
},
"PageRef": {
"description": "Page context dictionary entry — referenced by steps via `page` id.",
"type": "object",
"required": [
"id",
"url"
],
"properties": {
"id": {
"type": "string"
},
"title": {
"type": [
"string",
"null"
]
},
"url": {
"type": "string"
}
}
},
"SelectedOption": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
"value"
],
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"value": {
"type": "string"
}
}
},
"Step": {
"description": "One recorded user action — discriminated union by `op`.",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"to"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target",
"value"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"redacted": {
"type": [
"boolean",
"null"
]
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"selection",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOption"
}
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"page"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptor"
},
{
"type": "null"
}
]
}
}
}
]
},
"StepEffect": {
"description": "Observed navigation after a step (objective fact only).",
"type": "object",
"required": [
"navigated_to"
],
"properties": {
"navigated_to": {
"description": "Reference into `pages[]` for the destination page.",
"type": "string"
}
}
},
"TargetDescriptor": {
"description": "Stable semantic handle for an interacted element.\n\n`name` and `nearby_label` are **untrusted page text**.",
"type": "object",
"required": [
"tag"
],
"properties": {
"name": {
"type": [
"string",
"null"
]
},
"name_attr": {
"type": [
"string",
"null"
]
},
"nearby_label": {
"type": [
"string",
"null"
]
},
"placeholder": {
"type": [
"string",
"null"
]
},
"role": {
"type": [
"string",
"null"
]
},
"tag": {
"type": "string"
}
}
},
"TraceEntry": {
"description": "Recording entry point — first URL the flow starts from.",
"type": "object",
"required": [
"start_url"
],
"properties": {
"start_url": {
"type": "string"
}
}
}
}
}
+328
View File
@@ -0,0 +1,328 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Step",
"description": "One recorded user action — discriminated union by `op`.",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"to"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target",
"value"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"redacted": {
"type": [
"boolean",
"null"
]
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"selection",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOption"
}
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"page"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptor"
},
{
"type": "null"
}
]
}
}
}
],
"definitions": {
"KeyModifier": {
"description": "Keyboard modifier flags. Multiple flags may be combined; the extension folds them into CDP's bitfield (`alt=1, ctrl=2, meta=4, shift=8`).",
"type": "string",
"enum": [
"alt",
"ctrl",
"meta",
"shift"
]
},
"SelectedOption": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
"value"
],
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"value": {
"type": "string"
}
}
},
"StepEffect": {
"description": "Observed navigation after a step (objective fact only).",
"type": "object",
"required": [
"navigated_to"
],
"properties": {
"navigated_to": {
"description": "Reference into `pages[]` for the destination page.",
"type": "string"
}
}
},
"TargetDescriptor": {
"description": "Stable semantic handle for an interacted element.\n\n`name` and `nearby_label` are **untrusted page text**.",
"type": "object",
"required": [
"tag"
],
"properties": {
"name": {
"type": [
"string",
"null"
]
},
"name_attr": {
"type": [
"string",
"null"
]
},
"nearby_label": {
"type": [
"string",
"null"
]
},
"placeholder": {
"type": [
"string",
"null"
]
},
"role": {
"type": [
"string",
"null"
]
},
"tag": {
"type": "string"
}
}
}
}
}
@@ -100,4 +100,13 @@ fn main() {
dump!(WaitMsResult, "tool_wait_ms_result");
dump!(RequestHelpParams, "tool_request_help_params");
dump!(RequestHelpResult, "tool_request_help_result");
dump!(Trace, "trace");
dump!(Step, "trace_step");
dump!(RecordStartParams, "tool_record_start_params");
dump!(RecordStartResult, "tool_record_start_result");
dump!(RecordStopParams, "tool_record_stop_params");
dump!(RecordStopResult, "tool_record_stop_result");
dump!(RecordAwaitParams, "tool_record_await_params");
dump!(RecordAwaitResult, "tool_record_await_result");
}
+24 -2
View File
@@ -74,6 +74,12 @@ pub enum Method {
ToolWaitMs,
#[serde(rename = "tool.request_help")]
ToolRequestHelp,
#[serde(rename = "tool.record_start")]
ToolRecordStart,
#[serde(rename = "tool.record_stop")]
ToolRecordStop,
#[serde(rename = "tool.record_await")]
ToolRecordAwait,
#[serde(rename = "cancel")]
Cancel,
@@ -121,9 +127,16 @@ impl Method {
| Method::ToolFill
| Method::ToolPress
| Method::ToolSelect
| Method::ToolEvaluate => true,
| Method::ToolEvaluate
// May navigate via optional `url` and changes Agent Window
// chrome; gate behind pending-interrupt like other writes.
| Method::ToolRecordStart => true,
// Read-only tool calls — transparent.
//
// `record_stop` / `record_await` observe / finish a recording
// without driving new automation gestures, so they stay
// ungated (teardown after interrupt must still work).
Method::ToolTabList
| Method::ToolSnapshot
| Method::ToolGetHtml
@@ -132,7 +145,9 @@ impl Method {
| Method::ToolNetwork
| Method::ToolWaitForNavigation
| Method::ToolWaitMs
| Method::ToolRequestHelp => false,
| Method::ToolRequestHelp
| Method::ToolRecordStop
| Method::ToolRecordAwait => false,
// Session lifecycle — not gated.
Method::SessionStart
@@ -218,6 +233,13 @@ mod tests {
assert!(Method::ToolPress.is_mutating());
assert!(Method::ToolSelect.is_mutating());
assert!(Method::ToolEvaluate.is_mutating());
assert!(Method::ToolRecordStart.is_mutating());
}
#[test]
fn is_mutating_classifies_record_stop_await_as_non_mutating() {
assert!(!Method::ToolRecordStop.is_mutating());
assert!(!Method::ToolRecordAwait.is_mutating());
}
#[test]
+3 -1
View File
@@ -1,4 +1,4 @@
//! Typed params/results for the 21 `tool.*` RPC methods (§7).
//! Typed params/results for the `tool.*` RPC methods (§7).
pub mod console;
pub mod dialog;
@@ -7,6 +7,7 @@ pub mod interaction;
pub mod navigation;
pub mod network;
pub mod observation;
pub mod record;
pub mod script;
pub mod session;
pub mod tabs;
@@ -19,6 +20,7 @@ pub use interaction::*;
pub use navigation::*;
pub use network::*;
pub use observation::*;
pub use record::*;
pub use script::*;
pub use session::*;
pub use tabs::*;
+458
View File
@@ -0,0 +1,458 @@
//! Semantic user-action recording (`tool.record_start` / `stop` / `await`).
//!
//! Trace v2 is a **record-only** log of user actions: what was clicked, filled,
//! selected, and where navigation occurred. No LLM runs during recording, so
//! variable-vs-constant classification is **not** stored — executing agents
//! infer that at run time from raw values and control names.
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
use super::interaction::KeyModifier;
// ---------------------------------------------------------------------------
// Target
// ---------------------------------------------------------------------------
/// Stable semantic handle for an interacted element.
///
/// `name` and `nearby_label` are **untrusted page text**.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TargetDescriptor {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub role: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub name: Option<String>,
pub tag: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub name_attr: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub placeholder: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub nearby_label: Option<String>,
}
// ---------------------------------------------------------------------------
// Trace envelope
// ---------------------------------------------------------------------------
/// Recording entry point — first URL the flow starts from.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TraceEntry {
pub start_url: String,
}
/// Page context dictionary entry — referenced by steps via `page` id.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct PageRef {
pub id: String,
pub url: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub title: Option<String>,
}
/// One selected option (`select` op).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct SelectedOption {
pub value: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub label: Option<String>,
}
/// Observed navigation after a step (objective fact only).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct StepEffect {
/// Reference into `pages[]` for the destination page.
pub navigated_to: String,
}
/// Fields shared by every step variant (flattened in JSON).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct StepCommon {
pub id: u32,
/// Reference into `pages[]`.
pub page: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub effect: Option<StepEffect>,
}
// ---------------------------------------------------------------------------
// Step op-specific payloads
// ---------------------------------------------------------------------------
/// One recorded user action — discriminated union by `op`.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
#[serde(tag = "op", rename_all = "snake_case")]
pub enum Step {
Navigate {
#[serde(flatten)]
common: StepCommon,
to: String,
},
Click {
#[serde(flatten)]
common: StepCommon,
target: TargetDescriptor,
},
Fill {
#[serde(flatten)]
common: StepCommon,
target: TargetDescriptor,
value: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
redacted: Option<bool>,
},
Select {
#[serde(flatten)]
common: StepCommon,
target: TargetDescriptor,
selection: Vec<SelectedOption>,
},
Press {
#[serde(flatten)]
common: StepCommon,
key: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
modifiers: Option<Vec<KeyModifier>>,
#[serde(default, skip_serializing_if = "Option::is_none")]
target: Option<TargetDescriptor>,
},
}
// ---------------------------------------------------------------------------
// Trace root
// ---------------------------------------------------------------------------
/// Persisted user-action trace exported by `tool.record_stop` / `await`.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct Trace {
/// RFC 3339 timestamp when recording stopped.
pub recorded_at: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub started_at: Option<String>,
/// Optional user-provided goal from `--purpose` (metadata only).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub purpose: Option<String>,
pub entry: TraceEntry,
pub pages: Vec<PageRef>,
pub steps: Vec<Step>,
}
// ---------------------------------------------------------------------------
// RPC params / results
// ---------------------------------------------------------------------------
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct RecordStartParams {
pub session_id: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub tab_id: Option<i64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub url: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub purpose: Option<String>,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct RecordStartResult {
pub tab_id: i64,
pub recording: bool,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct RecordStopParams {
pub session_id: String,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct RecordStopResult {
pub trace: Trace,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct RecordAwaitParams {
pub session_id: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub timeout_ms: Option<u32>,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct RecordAwaitResult {
pub trace: Trace,
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
fn sample_common(id: u32) -> StepCommon {
StepCommon {
id,
page: "p1".into(),
effect: None,
}
}
fn sample_target() -> TargetDescriptor {
TargetDescriptor {
role: Some("button".into()),
name: Some("发布".into()),
tag: "button".into(),
name_attr: None,
placeholder: None,
nearby_label: None,
}
}
fn sample_trace() -> Trace {
Trace {
recorded_at: "2026-07-17T09:01:10Z".into(),
started_at: Some("2026-07-17T09:00:00Z".into()),
purpose: Some("发布一篇文章".into()),
entry: TraceEntry {
start_url: "https://x.com/editor".into(),
},
pages: vec![
PageRef {
id: "p1".into(),
url: "https://x.com/editor".into(),
title: Some("写文章".into()),
},
PageRef {
id: "p2".into(),
url: "https://x.com/p/99".into(),
title: None,
},
],
steps: vec![
Step::Fill {
common: sample_common(1),
target: TargetDescriptor {
role: Some("textbox".into()),
name: Some("标题".into()),
tag: "input".into(),
name_attr: None,
placeholder: None,
nearby_label: None,
},
value: "我的第一篇文章".into(),
redacted: None,
},
Step::Select {
common: sample_common(3),
target: TargetDescriptor {
role: Some("combobox".into()),
name: Some("分类".into()),
tag: "select".into(),
name_attr: None,
placeholder: None,
nearby_label: None,
},
selection: vec![SelectedOption {
value: "tech".into(),
label: Some("技术分享".into()),
}],
},
Step::Click {
common: StepCommon {
effect: Some(StepEffect {
navigated_to: "p2".into(),
}),
..sample_common(4)
},
target: sample_target(),
},
],
}
}
#[test]
fn step_click_round_trips() {
let step = Step::Click {
common: sample_common(1),
target: sample_target(),
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v.get("op").and_then(|v| v.as_str()), Some("click"));
assert_eq!(v.get("page").and_then(|v| v.as_str()), Some("p1"));
assert!(v.get("key").is_none());
let round: Step = serde_json::from_value(v).unwrap();
assert_eq!(round, step);
}
#[test]
fn step_fill_with_raw_value_round_trips() {
let step = Step::Fill {
common: sample_common(2),
target: TargetDescriptor {
role: Some("textbox".into()),
name: Some("服务名称".into()),
tag: "input".into(),
name_attr: Some("serviceName".into()),
placeholder: None,
nearby_label: None,
},
value: "my-svc".into(),
redacted: None,
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["op"], "fill");
assert_eq!(v["value"], "my-svc");
let round: Step = serde_json::from_value(v).unwrap();
assert_eq!(round, step);
}
#[test]
fn step_fill_password_is_redacted() {
let step = Step::Fill {
common: sample_common(1),
target: TargetDescriptor {
role: Some("textbox".into()),
name: Some("密码".into()),
tag: "input".into(),
name_attr: None,
placeholder: None,
nearby_label: None,
},
value: "***".into(),
redacted: Some(true),
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["value"], "***");
assert_eq!(v["redacted"], true);
}
#[test]
fn step_select_uses_object_array() {
let step = Step::Select {
common: sample_common(3),
target: TargetDescriptor {
role: Some("combobox".into()),
name: Some("分类".into()),
tag: "select".into(),
name_attr: None,
placeholder: None,
nearby_label: None,
},
selection: vec![SelectedOption {
value: "tech".into(),
label: Some("技术分享".into()),
}],
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["selection"][0]["value"], "tech");
assert_eq!(v["selection"][0]["label"], "技术分享");
let round: Step = serde_json::from_value(v).unwrap();
assert_eq!(round, step);
}
#[test]
fn step_navigate_round_trips() {
let step = Step::Navigate {
common: sample_common(1),
to: "https://example.com".into(),
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["op"], "navigate");
assert_eq!(v["to"], "https://example.com");
let round: Step = serde_json::from_value(v).unwrap();
assert_eq!(round, step);
}
#[test]
fn step_press_with_modifiers_round_trips() {
let step = Step::Press {
common: StepCommon {
effect: Some(StepEffect {
navigated_to: "p2".into(),
}),
..sample_common(2)
},
key: "Enter".into(),
modifiers: Some(vec![KeyModifier::Ctrl, KeyModifier::Shift]),
target: Some(sample_target()),
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["key"], "Enter");
assert_eq!(v["modifiers"], json!(["ctrl", "shift"]));
assert_eq!(v["effect"]["navigated_to"], "p2");
let round: Step = serde_json::from_value(v).unwrap();
assert_eq!(round, step);
}
#[test]
fn trace_record_only_round_trips() {
let trace = sample_trace();
let v = serde_json::to_value(&trace).unwrap();
assert!(v.get("version").is_none());
assert_eq!(
v.get("purpose").and_then(|v| v.as_str()),
Some("发布一篇文章")
);
assert!(v.get("parameters").is_none());
assert!(v.get("site").is_none());
assert!(v.get("goal").is_none());
assert_eq!(
v.get("entry")
.and_then(|e| e.get("start_url"))
.and_then(|u| u.as_str()),
Some("https://x.com/editor")
);
let round: Trace = serde_json::from_value(v).unwrap();
assert_eq!(round, trace);
}
#[test]
fn discriminated_union_click_ignores_extra_key_field() {
let bad = json!({
"op": "click",
"id": 1,
"page": "p1",
"target": { "tag": "button", "name": "OK" },
"key": "Enter"
});
let step: Step = serde_json::from_value(bad).unwrap();
assert!(matches!(step, Step::Click { .. }));
let v = serde_json::to_value(&step).unwrap();
assert!(v.get("key").is_none());
}
#[test]
fn extension_trace_deserializes() {
let v = json!({
"recorded_at": "2026-07-21T08:00:00Z",
"started_at": "2026-07-21T07:59:00Z",
"purpose": "demo",
"entry": { "start_url": "https://example.com/editor" },
"pages": [
{ "id": "p1", "url": "https://example.com/editor" },
{ "id": "p2", "url": "https://example.com/p/99" }
],
"steps": [
{
"op": "fill",
"id": 1,
"page": "p1",
"target": { "tag": "input", "role": "textbox", "name": "标题" },
"value": "hello"
},
{
"op": "click",
"id": 2,
"page": "p1",
"target": { "tag": "button", "role": "button", "name": "发布" },
"effect": { "navigated_to": "p2" }
}
]
});
let trace: Trace = serde_json::from_value(v).unwrap();
assert_eq!(trace.entry.start_url, "https://example.com/editor");
assert_eq!(trace.pages.len(), 2);
assert_eq!(trace.steps.len(), 2);
}
}