mirror of
https://github.com/Tencent/BrowserSkill.git
synced 2026-09-10 13:01:23 +08:00
feat(vom): canvas design and fixtures
This commit is contained in:
@@ -0,0 +1,119 @@
|
||||
# Visual observation design fixtures
|
||||
|
||||
These are synthetic, site-independent inputs for the visual observation architecture in
|
||||
[the design](../../../../docs/design/visual-observation.md). They contain no original page
|
||||
data, account identifiers, external fonts or production selectors. They are not a production
|
||||
discovery implementation.
|
||||
|
||||
Use [the acceptance checklist](../../../../docs/design/visual-observation-acceptance.md)
|
||||
for each PR's steps and success criteria. No separate report is required.
|
||||
Historical probe success does not certify future production modules.
|
||||
|
||||
## Browser contract probe
|
||||
|
||||
Requires Node.js with built-in WebSocket support (Node 22+) and an installed Chrome. No extra
|
||||
packages, user profile, extension, or login are required.
|
||||
|
||||
From the repository root:
|
||||
|
||||
```sh
|
||||
node apps/extension/test-fixtures/visual-observation/probe.mjs /tmp/bsk-visual-evidence
|
||||
```
|
||||
|
||||
On platforms other than macOS, set `CHROME_PATH` to the installed Chrome executable. The probe
|
||||
starts its own headless Chrome with an ephemeral profile and a loopback fixture server. It
|
||||
removes the profile and stops Chrome/server on completion or failure. `report.json`, raw CDP
|
||||
replies and PNGs are written to the explicitly supplied directory; do not commit them.
|
||||
|
||||
The probe verifies CSS exposure versus visibility, snapshot units, DPR raster dimensions,
|
||||
CSS zoom, frame content origins, child scrolling, nested same-target geometry, a confirmed
|
||||
OOPIF target, and navigation/document.open identity behavior. It records page-scale emulation
|
||||
separately; that is **not** browser UI zoom. It does not certify production VOM behavior, arbitrary
|
||||
occlusion, exact clip-path geometry, or all browser versions.
|
||||
|
||||
## Minimal end-to-end reproduction
|
||||
|
||||
```sh
|
||||
python3 -m http.server 4177 --bind 127.0.0.1 \
|
||||
--directory apps/extension/test-fixtures/visual-observation
|
||||
bsk session start --width 1440 --height 1100
|
||||
bsk navigate http://127.0.0.1:4177/table.html --session <session>
|
||||
bsk observe --session <session>
|
||||
bsk screenshot --session <session> --out /tmp/bsk-table.png
|
||||
bsk session stop <session>
|
||||
```
|
||||
|
||||
Always stop the session and the local server. Use a foreground Agent Window and confirm the
|
||||
painted grid is visible before interpreting the observation. `load`, AX toolbar appearance,
|
||||
and background screenshots alone do not prove a canvas has drawn.
|
||||
|
||||
`table.html` embeds `grid.html` through `localhost` from a `127.0.0.1` parent. The grid has DOM
|
||||
toolbar controls and one unlabeled Canvas with three synthetic columns and six painted rows.
|
||||
|
||||
Main baseline: `Page action`, `Insert record`, and `Filter` are referenceable; the painted
|
||||
`Canvas column A/B/C` and `Canvas row 1..6` have no textual observation representation or visual
|
||||
ref, despite an untruncated result. This is a **known gap**, not a golden expectation that future
|
||||
implementations must preserve.
|
||||
|
||||
Future acceptance: retain the DOM controls and add a screenshot-only visual ref for the visible
|
||||
grid, with correct frame projection. Do not invent AX cell text. Validate the corresponding
|
||||
screenshot, then move/remove/resize and re-observe according to the freshness contract.
|
||||
|
||||
## Discovery policy cases
|
||||
|
||||
At a 1440×1100 window and without scrolling:
|
||||
|
||||
| Canvas ID | Expected candidate in the new design | Reason |
|
||||
|---|---|---|
|
||||
| plain | yes | No name required |
|
||||
| named | yes | Name does not establish complete pixel semantics |
|
||||
| aria-hidden | yes | Accessibility exclusion is not visual exclusion |
|
||||
| inert | yes | Inertness is not visual hiding |
|
||||
| pointer-none | yes | Screenshot observation does not require pointer interaction |
|
||||
| visibility-override | yes | Child restores computed visibility |
|
||||
| opacity-parent | no | Ancestor makes subtree transparent |
|
||||
| display-none | no | No layout box |
|
||||
| visibility-hidden | no | Computed visibility suppresses this element |
|
||||
| zero | no | Empty geometry |
|
||||
| hidden-overridden | yes | CSS display overrides normal hidden presentation |
|
||||
| partial-clip | yes, 60×30 crop | Supported overflow intersection |
|
||||
| full-clip | no | Fully outside ancestor clip |
|
||||
| axis-clip | yes, x clipped only | overflow-x:clip and overflow-y:visible remain independent |
|
||||
| border-clip | yes, clipped to client box | Ancestor border box is not its scrollport |
|
||||
| border-canvas | yes, border-box region | Observation and live screenshot resolver must use the same box kind |
|
||||
| fallback | yes | AX fallback is not full pixel coverage |
|
||||
| stack-a / stack-b | one output entry after selection | Same document/parent/exact crop; not proof of a logical object |
|
||||
|
||||
`checkVisibility()` is used in the probe only for CSS suppression evidence; it does not establish
|
||||
viewport intersection, ancestor overflow clipping, or absence of occlusion. `frame.html` supplies
|
||||
known local coordinates, bordered owners, nested frames and scrollable content for independent
|
||||
geometry verification. The fixture IDs must never enter production discovery code.
|
||||
|
||||
## Renderer operation counts
|
||||
|
||||
After installing the repository's normal dependencies:
|
||||
|
||||
```sh
|
||||
node apps/extension/test-fixtures/visual-observation/measure-render.mjs /tmp/bsk-render-evidence
|
||||
```
|
||||
|
||||
This instruments a temporary copy of the current renderer, runs the installed Vitest, and writes
|
||||
the source hash and loop counters to `render-counts.json`. It checks instrumentation sites and
|
||||
fails explicitly when source structure changes; update the diagnostic deliberately then. It never
|
||||
modifies production sources and never asserts that today's quadratic costs must persist.
|
||||
|
||||
The wide fixture contains N headings followed by N equally named buttons in a common context.
|
||||
The deep fixture contains 2,000/4,000 transparent wrappers. Counters cover render-stack visits,
|
||||
ordered DOM context candidates, and same-container sibling visits. They are not a complete CPU
|
||||
profile or an assertion about the complexity of all VOM stages.
|
||||
|
||||
Use these existing-renderer counts only as a before/after regression baseline. Improving the
|
||||
historical algorithm is not a prerequisite for Canvas work. New Canvas stages need their own
|
||||
workload models and measurements; do not copy this renderer's algorithms or use these counts
|
||||
as evidence that visual discovery, clipping, selection, or screenshot execution is efficient.
|
||||
|
||||
## Review boundary
|
||||
|
||||
Commit only these synthetic fixtures, diagnostic tools, and reviewed design/evidence summaries.
|
||||
Keep local browser profiles, original internal-site screenshots/HTML/observations, generated
|
||||
captures, temporary source instrumentation and build outputs outside the changeset.
|
||||
@@ -0,0 +1,25 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<meta charset="utf-8">
|
||||
<title>Frame geometry contract</title>
|
||||
<style>
|
||||
body { margin: 0; width: 600px; height: 900px; }
|
||||
canvas { position: absolute; left: 17px; top: 23px; width: 120px; height: 40px; }
|
||||
iframe { position: absolute; left: 30px; top: 90px; width: 180px; height: 100px; border: 7px solid #555; }
|
||||
</style>
|
||||
<canvas id="frame-canvas" width="120" height="40"></canvas>
|
||||
<script>
|
||||
const context = document.querySelector("canvas").getContext("2d");
|
||||
context.fillStyle = "#15803d";
|
||||
context.fillRect(0, 0, 120, 40);
|
||||
context.fillStyle = "white";
|
||||
context.fillText("FRAME PIXELS", 4, 24);
|
||||
if (new URLSearchParams(location.search).has("nested")) {
|
||||
const child = document.createElement("iframe");
|
||||
child.title = "nested same-target frame";
|
||||
child.src = "/frame.html";
|
||||
document.body.append(child);
|
||||
}
|
||||
window.fixtureReady = true;
|
||||
</script>
|
||||
</html>
|
||||
@@ -0,0 +1,26 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<meta charset="utf-8">
|
||||
<title>Canvas grid</title>
|
||||
<style>
|
||||
body { margin: 0; font: 16px sans-serif; }
|
||||
nav { height: 40px; }
|
||||
canvas { display: block; width: 780px; height: 360px; }
|
||||
</style>
|
||||
<nav><button type="button">Insert record</button> <button type="button">Filter</button></nav>
|
||||
<canvas width="1560" height="720"></canvas>
|
||||
<script>
|
||||
const context = document.querySelector("canvas").getContext("2d");
|
||||
context.scale(2, 2);
|
||||
context.fillStyle = "white";
|
||||
context.fillRect(0, 0, 780, 360);
|
||||
context.strokeStyle = "#777";
|
||||
for (let x = 0; x <= 780; x += 260) { context.beginPath(); context.moveTo(x, 0); context.lineTo(x, 350); context.stroke(); }
|
||||
for (let y = 0; y <= 350; y += 50) { context.beginPath(); context.moveTo(0, y); context.lineTo(780, y); context.stroke(); }
|
||||
context.fillStyle = "#111";
|
||||
context.font = "16px sans-serif";
|
||||
["Canvas column A", "Canvas column B", "Canvas column C"].forEach((label, i) => context.fillText(label, i * 260 + 12, 30));
|
||||
for (let row = 1; row <= 6; row++) context.fillText(`Canvas row ${row}`, 12, row * 50 + 30);
|
||||
window.fixtureReady = true;
|
||||
</script>
|
||||
</html>
|
||||
@@ -0,0 +1,56 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<meta charset="utf-8">
|
||||
<title>Visual observation contract fixture</title>
|
||||
<style>
|
||||
body { margin: 0; min-height: 1800px; font: 14px sans-serif; }
|
||||
#cases { display: grid; grid-template-columns: repeat(4, 180px); gap: 12px; padding: 20px; }
|
||||
.case { width: 160px; height: 70px; }
|
||||
canvas { width: 120px; height: 40px; }
|
||||
.clip { width: 60px; height: 30px; overflow: hidden; }
|
||||
.axis-clip { width: 60px; height: 20px; overflow-x: clip; overflow-y: visible; }
|
||||
#same, #cross { position: absolute; top: 600px; width: 300px; height: 220px; border: 10px solid #555; }
|
||||
#same { left: 40px; }
|
||||
#cross { left: 400px; }
|
||||
</style>
|
||||
<h1>Canvas contract</h1>
|
||||
<button type="button" id="control">Ordinary DOM control</button>
|
||||
<div id="cases">
|
||||
<div class="case"><canvas id="plain"></canvas></div>
|
||||
<div class="case"><canvas id="named" aria-label="Named chart"></canvas></div>
|
||||
<div class="case" aria-hidden="true"><canvas id="aria-hidden"></canvas></div>
|
||||
<div class="case" inert><canvas id="inert"></canvas></div>
|
||||
<div class="case"><canvas id="pointer-none" style="pointer-events:none"></canvas></div>
|
||||
<div class="case" style="visibility:hidden"><canvas id="visibility-override" style="visibility:visible"></canvas></div>
|
||||
<div class="case" style="opacity:0"><canvas id="opacity-parent"></canvas></div>
|
||||
<div class="case"><canvas id="display-none" style="display:none"></canvas></div>
|
||||
<div class="case"><canvas id="visibility-hidden" style="visibility:hidden"></canvas></div>
|
||||
<div class="case"><canvas id="zero" style="width:0"></canvas></div>
|
||||
<div class="case"><canvas id="hidden-overridden" hidden style="display:block"></canvas></div>
|
||||
<div class="case"><div class="clip"><canvas id="partial-clip"></canvas></div></div>
|
||||
<div class="case"><div class="clip"><canvas id="full-clip" style="margin-left:80px"></canvas></div></div>
|
||||
<div class="case"><canvas id="fallback">Accessible fallback is not pixel content</canvas></div>
|
||||
<div class="case"><div class="axis-clip"><canvas id="axis-clip"></canvas></div></div>
|
||||
<div class="case"><div id="border-clip-owner" style="width:60px;height:30px;border:5px solid black;padding:4px;overflow:hidden"><canvas id="border-clip"></canvas></div></div>
|
||||
<div class="case"><canvas id="border-canvas" style="border:5px solid black"></canvas></div>
|
||||
<div class="case" style="position:relative"><canvas id="stack-a"></canvas><canvas id="stack-b" style="position:absolute;left:0;top:0"></canvas></div>
|
||||
</div>
|
||||
<iframe id="same" title="same target"></iframe>
|
||||
<iframe id="cross" title="cross site"></iframe>
|
||||
<script>
|
||||
for (const canvas of document.querySelectorAll("canvas")) {
|
||||
canvas.width = 120;
|
||||
canvas.height = 40;
|
||||
const context = canvas.getContext("2d");
|
||||
context.fillStyle = "#1d4ed8";
|
||||
context.fillRect(0, 0, 120, 40);
|
||||
context.fillStyle = "white";
|
||||
context.fillText("PIXEL TABLE 42", 5, 24);
|
||||
}
|
||||
if (!new URLSearchParams(location.search).has("noFrames")) {
|
||||
document.querySelector("#same").src = "/frame.html?nested=1";
|
||||
document.querySelector("#cross").src = `${location.protocol}//localhost:${location.port}/frame.html`;
|
||||
}
|
||||
window.fixtureReady = true;
|
||||
</script>
|
||||
</html>
|
||||
@@ -0,0 +1,86 @@
|
||||
// Diagnostic, not a golden test requiring the current quadratic cost to persist.
|
||||
// Instrument a temporary copy; never modify the production renderer.
|
||||
import assert from "node:assert/strict";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { copyFile, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { createRequire } from "node:module";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../../../..");
|
||||
const output = process.argv[2];
|
||||
if (!output || !path.isAbsolute(output))
|
||||
throw new Error("Pass an absolute evidence output directory");
|
||||
const require = createRequire(path.join(root, "apps/extension/package.json"));
|
||||
const vitestDirectory = path.dirname(require.resolve("vitest/package.json"));
|
||||
const temporary = await mkdtemp(path.join(os.tmpdir(), "bsk-render-measure-"));
|
||||
await mkdir(output, { recursive: true });
|
||||
try {
|
||||
const source = await readFile(path.join(root, "packages/vom/src/render.ts"), "utf8");
|
||||
let instrumented = source;
|
||||
for (const [needle, field] of [
|
||||
["const { node, depth } = stack.pop() as { node: VomNode; depth: number };", "renderVisits"],
|
||||
["for (const candidate of orderedCandidates) {", "contextCandidates"],
|
||||
["for (const sibling of siblings) {", "siblingVisits"],
|
||||
]) {
|
||||
assert.equal(instrumented.split(needle).length, 2, `Update measurement site: ${field}`);
|
||||
instrumented = instrumented.replace(needle, `${needle} designCounts.${field}++;`);
|
||||
}
|
||||
instrumented = `export const designCounts = {renderVisits:0,contextCandidates:0,siblingVisits:0};\n${instrumented}`;
|
||||
await writeFile(path.join(temporary, "render.ts"), instrumented);
|
||||
for (const file of ["layers.ts", "types.ts"])
|
||||
await copyFile(path.join(root, "packages/vom/src", file), path.join(temporary, file));
|
||||
const reportPath = path.join(output, "render-counts.json");
|
||||
await writeFile(
|
||||
path.join(temporary, "measure.test.ts"),
|
||||
`
|
||||
import {writeFileSync} from 'node:fs';
|
||||
import {test,expect} from ${JSON.stringify(path.join(vitestDirectory, "dist/index.js"))};
|
||||
import {renderVom,designCounts} from './render';
|
||||
test('measure operation counts',()=>{
|
||||
const rows=[];
|
||||
const make=(id,extra={})=>({id,parentId:1,backendNodeId:id,tag:'div',role:'generic',rect:{x:0,y:0,w:100,h:20},paintOrder:0,position:'static',pointerEvents:'auto',contextScopeId:'root',...extra});
|
||||
const run=(shape,n,nodes)=>{
|
||||
Object.assign(designCounts,{renderVisits:0,contextCandidates:0,siblingVisits:0});
|
||||
const result=renderVom({viewport:{width:1440,height:1000},nodes},{maxTokens:Infinity});
|
||||
rows.push({shape,n,counts:{...designCounts},refs:result.refs.length});
|
||||
};
|
||||
for(const n of [100,200,400]){
|
||||
const nodes=[make(1,{parentId:null,role:'RootWebArea',name:'Doc'})];
|
||||
for(let i=0;i<n;i++)nodes.push(make(i+2,{role:'heading',name:'Region '+i,domParentId:1,domAncestorIds:[1]}));
|
||||
for(let i=0;i<n;i++)nodes.push(make(n+i+2,{tag:'button',role:'button',name:'View',domParentId:1,domAncestorIds:[1]}));
|
||||
run('wide',n,nodes);
|
||||
}
|
||||
for(const n of [2000,4000]){
|
||||
const nodes=[make(1,{parentId:null,role:'RootWebArea',name:'Doc'})];
|
||||
for(let i=2;i<=n;i++)nodes.push(make(i,{parentId:i-1}));
|
||||
nodes.push(make(n+1,{parentId:n,role:'button',tag:'button',name:'Unique control'}));
|
||||
run('deep',n,nodes);
|
||||
}
|
||||
expect(rows.at(-1).refs).toBe(1);
|
||||
writeFileSync(${JSON.stringify(reportPath)},JSON.stringify({sourceSha256:${JSON.stringify(createHash("sha256").update(source).digest("hex"))},rows},null,2));
|
||||
});
|
||||
`,
|
||||
);
|
||||
const config = path.join(temporary, "vitest.config.mjs");
|
||||
await writeFile(
|
||||
config,
|
||||
`export default {test:{environment:'node',include:[${JSON.stringify(path.join(temporary, "measure.test.ts"))}]}};`,
|
||||
);
|
||||
const result = spawnSync(
|
||||
process.execPath,
|
||||
[path.join(vitestDirectory, "vitest.mjs"), "run", "--config", config],
|
||||
{ cwd: root, encoding: "utf8", timeout: 60000 },
|
||||
);
|
||||
await writeFile(
|
||||
path.join(output, "render-measure.log"),
|
||||
`${result.stdout ?? ""}\n${result.stderr ?? ""}`,
|
||||
);
|
||||
if (result.error) throw result.error;
|
||||
assert.equal(result.status, 0, `Measurement failed; see ${output}/render-measure.log`);
|
||||
process.stdout.write(`${reportPath}\n`);
|
||||
} finally {
|
||||
await rm(temporary, { recursive: true, force: true });
|
||||
}
|
||||
@@ -0,0 +1,533 @@
|
||||
// Standalone browser contract probe. No extension, credentials or user profile.
|
||||
// Raw replies and screenshots are written only to the caller-supplied directory.
|
||||
import assert from "node:assert/strict";
|
||||
import { spawn } from "node:child_process";
|
||||
import { once } from "node:events";
|
||||
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import http from "node:http";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const fixtureDirectory = path.dirname(fileURLToPath(import.meta.url));
|
||||
const output = process.argv[2];
|
||||
if (!output || !path.isAbsolute(output))
|
||||
throw new Error("Pass an absolute evidence output directory");
|
||||
const chromePath =
|
||||
process.env.CHROME_PATH ?? "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome";
|
||||
const pause = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
|
||||
const styles = [
|
||||
"position",
|
||||
"pointer-events",
|
||||
"cursor",
|
||||
"visibility",
|
||||
"opacity",
|
||||
"overflow-x",
|
||||
"overflow-y",
|
||||
"display",
|
||||
];
|
||||
const checks = [];
|
||||
function check(name, condition) {
|
||||
checks.push({ name, passed: Boolean(condition) });
|
||||
assert.ok(condition, name);
|
||||
}
|
||||
|
||||
class Cdp {
|
||||
nextId = 0;
|
||||
pending = new Map();
|
||||
events = [];
|
||||
counts = {};
|
||||
async connect(url) {
|
||||
this.socket = new WebSocket(url);
|
||||
this.socket.addEventListener("message", ({ data }) => {
|
||||
const reply = JSON.parse(data);
|
||||
if (!reply.id) {
|
||||
this.events.push(reply);
|
||||
return;
|
||||
}
|
||||
const task = this.pending.get(reply.id);
|
||||
if (!task) return;
|
||||
this.pending.delete(reply.id);
|
||||
clearTimeout(task.timer);
|
||||
if (reply.error) task.reject(new Error(JSON.stringify(reply.error)));
|
||||
else task.resolve(reply.result);
|
||||
});
|
||||
await new Promise((resolve, reject) => {
|
||||
this.socket.addEventListener("open", resolve, { once: true });
|
||||
this.socket.addEventListener("error", reject, { once: true });
|
||||
});
|
||||
}
|
||||
send(method, params = {}, sessionId) {
|
||||
const id = ++this.nextId;
|
||||
this.counts[method] = (this.counts[method] ?? 0) + 1;
|
||||
return new Promise((resolve, reject) => {
|
||||
const timer = setTimeout(() => {
|
||||
this.pending.delete(id);
|
||||
reject(new Error(`CDP timeout: ${method}`));
|
||||
}, 10000);
|
||||
this.pending.set(id, { resolve, reject, timer });
|
||||
this.socket.send(JSON.stringify({ id, method, params, ...(sessionId ? { sessionId } : {}) }));
|
||||
});
|
||||
}
|
||||
close() {
|
||||
for (const task of this.pending.values()) {
|
||||
clearTimeout(task.timer);
|
||||
task.reject(new Error("CDP closed"));
|
||||
}
|
||||
this.pending.clear();
|
||||
this.socket?.close();
|
||||
}
|
||||
}
|
||||
|
||||
function decodedCanvases(snapshot) {
|
||||
return snapshot.documents.flatMap((document) => {
|
||||
const layoutByNode = new Map(document.layout.nodeIndex.map((node, index) => [node, index]));
|
||||
return document.nodes.nodeName.flatMap((name, index) => {
|
||||
if (snapshot.strings[name] !== "CANVAS") return [];
|
||||
const attrs = document.nodes.attributes[index];
|
||||
const attributes = {};
|
||||
for (let i = 0; i < attrs.length; i += 2)
|
||||
attributes[snapshot.strings[attrs[i]]] = snapshot.strings[attrs[i + 1]];
|
||||
const layoutIndex = layoutByNode.get(index) ?? -1;
|
||||
return [
|
||||
{
|
||||
id: attributes.id,
|
||||
backendNodeId: document.nodes.backendNodeId[index],
|
||||
frameId: snapshot.strings[document.frameId],
|
||||
attributes,
|
||||
bounds: layoutIndex < 0 ? null : document.layout.bounds[layoutIndex],
|
||||
styles:
|
||||
layoutIndex < 0
|
||||
? null
|
||||
: Object.fromEntries(
|
||||
styles.map((style, i) => [
|
||||
style,
|
||||
snapshot.strings[document.layout.styles[layoutIndex][i]],
|
||||
]),
|
||||
),
|
||||
},
|
||||
];
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
await mkdir(output, { recursive: true });
|
||||
const profile = await mkdtemp(path.join(os.tmpdir(), "bsk-visual-probe-"));
|
||||
const server = http.createServer(async (request, response) => {
|
||||
const name = new URL(request.url, "http://fixture").pathname;
|
||||
if (!["/index.html", "/frame.html"].includes(name)) {
|
||||
response.writeHead(404).end();
|
||||
return;
|
||||
}
|
||||
try {
|
||||
response.setHeader("Content-Type", "text/html; charset=utf-8");
|
||||
response.end(await readFile(path.join(fixtureDirectory, name.slice(1))));
|
||||
} catch {
|
||||
response.writeHead(500).end();
|
||||
}
|
||||
});
|
||||
const cdp = new Cdp();
|
||||
let chrome;
|
||||
let report = {};
|
||||
try {
|
||||
server.listen(0, "127.0.0.1");
|
||||
await once(server, "listening");
|
||||
const port = server.address().port;
|
||||
chrome = spawn(
|
||||
chromePath,
|
||||
[
|
||||
"--headless=new",
|
||||
`--user-data-dir=${profile}`,
|
||||
"--remote-debugging-port=0",
|
||||
"--no-first-run",
|
||||
"--no-default-browser-check",
|
||||
"--site-per-process",
|
||||
"--window-size=1440,1100",
|
||||
"about:blank",
|
||||
],
|
||||
{ stdio: "ignore" },
|
||||
);
|
||||
chrome.on("error", (error) => {
|
||||
report.launchError = error.message;
|
||||
});
|
||||
let activePort;
|
||||
for (let attempt = 0; attempt < 100; attempt++) {
|
||||
try {
|
||||
activePort = (await readFile(path.join(profile, "DevToolsActivePort"), "utf8"))
|
||||
.trim()
|
||||
.split("\n");
|
||||
break;
|
||||
} catch {
|
||||
await pause(100);
|
||||
}
|
||||
}
|
||||
if (!activePort)
|
||||
throw new Error(report.launchError ?? "Chrome did not expose DevToolsActivePort");
|
||||
await cdp.connect(`ws://127.0.0.1:${activePort[0]}${activePort[1]}`);
|
||||
report.browser = await cdp.send("Browser.getVersion");
|
||||
const { targetId } = await cdp.send("Target.createTarget", { url: "about:blank" });
|
||||
const { sessionId: root } = await cdp.send("Target.attachToTarget", { targetId, flatten: true });
|
||||
const send = (method, params = {}) => cdp.send(method, params, root);
|
||||
const evaluate = async (expression, session = root, contextId) => {
|
||||
const result = await cdp.send(
|
||||
"Runtime.evaluate",
|
||||
{ expression, returnByValue: true, awaitPromise: true, ...(contextId ? { contextId } : {}) },
|
||||
session,
|
||||
);
|
||||
if (result.exceptionDetails) throw new Error(JSON.stringify(result.exceptionDetails));
|
||||
return result.result.value;
|
||||
};
|
||||
await send("Page.enable");
|
||||
await send("Runtime.enable");
|
||||
async function navigate(url) {
|
||||
await send("Page.navigate", { url });
|
||||
for (let i = 0; i < 100; i++) {
|
||||
if (await evaluate("document.readyState === 'complete' && window.fixtureReady === true"))
|
||||
return;
|
||||
await pause(50);
|
||||
}
|
||||
throw new Error("Fixture did not become ready");
|
||||
}
|
||||
const url = `http://127.0.0.1:${port}/index.html`;
|
||||
await navigate(`${url}?noFrames=1`);
|
||||
async function capture(label) {
|
||||
const metrics = await send("Page.getLayoutMetrics");
|
||||
const snapshot = await send("DOMSnapshot.captureSnapshot", {
|
||||
computedStyles: styles,
|
||||
includePaintOrder: true,
|
||||
includeDOMRects: true,
|
||||
});
|
||||
const ax = await send("Accessibility.getFullAXTree");
|
||||
const runtime = await evaluate(`Array.from(document.querySelectorAll('canvas'), el => {
|
||||
const r=el.getBoundingClientRect(), s=getComputedStyle(el);
|
||||
return {id:el.id,rect:{x:r.x,y:r.y,w:r.width,h:r.height},visibility:s.visibility,display:s.display,
|
||||
checkVisibility:el.checkVisibility({opacityProperty:true,visibilityProperty:true}),
|
||||
dpr:devicePixelRatio,scrollX,scrollY}; })`);
|
||||
const result = { metrics, snapshot, ax, runtime, canvases: decodedCanvases(snapshot) };
|
||||
await writeFile(path.join(output, `${label}.json`), JSON.stringify(result, null, 2));
|
||||
return result;
|
||||
}
|
||||
const baseline = await capture("root-baseline");
|
||||
const rootImage = await send("Page.captureScreenshot", { format: "png" });
|
||||
await writeFile(path.join(output, "root.png"), Buffer.from(rootImage.data, "base64"));
|
||||
const runtime = Object.fromEntries(baseline.runtime.map((node) => [node.id, node]));
|
||||
for (const id of [
|
||||
"plain",
|
||||
"named",
|
||||
"aria-hidden",
|
||||
"inert",
|
||||
"pointer-none",
|
||||
"visibility-override",
|
||||
"hidden-overridden",
|
||||
"fallback",
|
||||
])
|
||||
check(`CSS visible: ${id}`, runtime[id].checkVisibility);
|
||||
for (const id of ["opacity-parent", "display-none", "visibility-hidden"])
|
||||
check(`CSS suppressed: ${id}`, !runtime[id].checkVisibility);
|
||||
check("painted text is absent from AX", !JSON.stringify(baseline.ax).includes("PIXEL TABLE 42"));
|
||||
report.css = baseline.runtime;
|
||||
const bordered = baseline.canvases.find((node) => node.id === "border-canvas");
|
||||
report.canvasBox = await send("DOM.getBoxModel", { backendNodeId: bordered.backendNodeId });
|
||||
check(
|
||||
"Canvas snapshot uses border box",
|
||||
bordered.bounds[2] === 130 &&
|
||||
report.canvasBox.model.border[2] - report.canvasBox.model.border[0] === 130 &&
|
||||
report.canvasBox.model.content[2] - report.canvasBox.model.content[0] === 120,
|
||||
);
|
||||
const rootDocument = baseline.snapshot.documents[0];
|
||||
const ownerNode = rootDocument.nodes.attributes.findIndex((attrs) =>
|
||||
attrs.some(
|
||||
(value, index) => index % 2 === 1 && baseline.snapshot.strings[value] === "border-clip-owner",
|
||||
),
|
||||
);
|
||||
const ownerLayout = rootDocument.layout.nodeIndex.indexOf(ownerNode);
|
||||
report.clipClientBox = {
|
||||
snapshot: rootDocument.layout.clientRects[ownerLayout],
|
||||
runtime: await evaluate(
|
||||
"(() => {const e=document.getElementById('border-clip-owner');return [e.clientLeft,e.clientTop,e.clientWidth,e.clientHeight]})()",
|
||||
),
|
||||
};
|
||||
check(
|
||||
"snapshot client rect contains local client offsets and size",
|
||||
JSON.stringify(report.clipClientBox.snapshot) ===
|
||||
JSON.stringify(report.clipClientBox.runtime) &&
|
||||
JSON.stringify(report.clipClientBox.runtime) === "[5,5,68,38]",
|
||||
);
|
||||
report.clips = await evaluate(`['partial-clip','full-clip','axis-clip'].map(id=>{
|
||||
const el=document.getElementById(id),p=el.parentElement,s=getComputedStyle(p);
|
||||
const a=el.getBoundingClientRect(),b=p.getBoundingClientRect();
|
||||
return {id,overflowX:s.overflowX,overflowY:s.overflowY,
|
||||
child:{x:a.x,y:a.y,w:a.width,h:a.height},parent:{x:b.x,y:b.y,w:b.width,h:b.height}};
|
||||
})`);
|
||||
const [partial, full, axis] = report.clips;
|
||||
check(
|
||||
"partial overflow crop dimensions",
|
||||
partial.parent.w === 60 &&
|
||||
partial.parent.h === 30 &&
|
||||
partial.child.x === partial.parent.x &&
|
||||
partial.child.y === partial.parent.y,
|
||||
);
|
||||
check(
|
||||
"fully clipped candidate lies beyond ancestor",
|
||||
full.child.x >= full.parent.x + full.parent.w,
|
||||
);
|
||||
check(
|
||||
"single-axis clip remains independent in computed style",
|
||||
axis.overflowX === "clip" && axis.overflowY === "visible" && axis.child.h > axis.parent.h,
|
||||
);
|
||||
const samples = [];
|
||||
for (let i = 0; i < 12; i++)
|
||||
samples.push(
|
||||
await evaluate(
|
||||
"JSON.stringify(document.querySelector('#plain').getBoundingClientRect().toJSON())",
|
||||
),
|
||||
);
|
||||
check("static rect stable over 12 reads", new Set(samples).size === 1);
|
||||
await evaluate("document.querySelector('#plain').style.transform='translateX(0.5px)'");
|
||||
report.halfPixelMovement = await evaluate(
|
||||
"document.querySelector('#plain').getBoundingClientRect().x",
|
||||
);
|
||||
await evaluate("document.querySelector('#plain').style.transform=''");
|
||||
report.raster = [];
|
||||
for (const dpr of [1, 1.5, 2]) {
|
||||
await send("Emulation.setDeviceMetricsOverride", {
|
||||
width: 1000,
|
||||
height: 900,
|
||||
deviceScaleFactor: dpr,
|
||||
mobile: false,
|
||||
});
|
||||
const state = await capture(`dpr-${dpr}`);
|
||||
const image = await send("Page.captureScreenshot", {
|
||||
format: "png",
|
||||
clip: { x: 20, y: 100, width: 120, height: 40, scale: 1 },
|
||||
});
|
||||
const bytes = Buffer.from(image.data, "base64");
|
||||
const dimensions = { width: bytes.readUInt32BE(16), height: bytes.readUInt32BE(20) };
|
||||
report.raster.push({
|
||||
dpr,
|
||||
dimensions,
|
||||
metrics: state.metrics,
|
||||
plain: state.canvases.find((node) => node.id === "plain"),
|
||||
});
|
||||
check(
|
||||
`screenshot raster DPR ${dpr}`,
|
||||
dimensions.width === 120 * dpr && dimensions.height === 40 * dpr,
|
||||
);
|
||||
}
|
||||
await send("Emulation.clearDeviceMetricsOverride");
|
||||
await evaluate("document.body.style.zoom='1.25'");
|
||||
const cssZoom = await capture("css-zoom-1.25");
|
||||
check(
|
||||
"CSS zoom is reflected in snapshot geometry",
|
||||
cssZoom.canvases.find((n) => n.id === "plain").bounds[2] === 150,
|
||||
);
|
||||
report.cssZoom = {
|
||||
runtime: cssZoom.runtime[0],
|
||||
snapshot: cssZoom.canvases.find((n) => n.id === "plain"),
|
||||
};
|
||||
await evaluate("document.body.style.zoom=''");
|
||||
await send("Emulation.setPageScaleFactor", { pageScaleFactor: 1.5 });
|
||||
report.pageScale = (await capture("page-scale-1.5")).metrics;
|
||||
await send("Emulation.setPageScaleFactor", { pageScaleFactor: 1 });
|
||||
await evaluate("scrollTo(0,100)");
|
||||
const scrolled = await capture("scroll-100");
|
||||
check(
|
||||
"snapshot document bounds stable under scroll",
|
||||
JSON.stringify(scrolled.canvases.find((n) => n.id === "plain").bounds) ===
|
||||
JSON.stringify(baseline.canvases.find((n) => n.id === "plain").bounds),
|
||||
);
|
||||
check(
|
||||
"runtime viewport y follows scroll",
|
||||
scrolled.runtime[0].rect.y === baseline.runtime[0].rect.y - 100,
|
||||
);
|
||||
await navigate(url);
|
||||
await pause(200);
|
||||
const targets = await cdp.send("Target.getTargets");
|
||||
const iframeTarget = targets.targetInfos.find(
|
||||
(target) => target.type === "iframe" && target.url.startsWith(`http://localhost:${port}/`),
|
||||
);
|
||||
check("cross-site frame is an actual OOPIF target", iframeTarget);
|
||||
const { sessionId: child } = await cdp.send("Target.attachToTarget", {
|
||||
targetId: iframeTarget.targetId,
|
||||
flatten: true,
|
||||
});
|
||||
report.frames = {};
|
||||
for (const [name, session] of [
|
||||
["root", root],
|
||||
["oopif", child],
|
||||
]) {
|
||||
const tree = await cdp.send("Page.getFrameTree", {}, session);
|
||||
const snapshot = await cdp.send(
|
||||
"DOMSnapshot.captureSnapshot",
|
||||
{ computedStyles: styles, includeDOMRects: true },
|
||||
session,
|
||||
);
|
||||
const frameRows = [];
|
||||
const stack = [tree.frameTree];
|
||||
while (stack.length) {
|
||||
const node = stack.pop();
|
||||
const world = await cdp.send(
|
||||
"Page.createIsolatedWorld",
|
||||
{ frameId: node.frame.id, worldName: "bsk-design-probe" },
|
||||
session,
|
||||
);
|
||||
const local = await evaluate(
|
||||
"({url:location.pathname,innerWidth,innerHeight,scrollX,scrollY,canvas:document.querySelector('canvas').getBoundingClientRect().toJSON()})",
|
||||
session,
|
||||
world.executionContextId,
|
||||
);
|
||||
let owner;
|
||||
if (node.frame.parentId) {
|
||||
// All owners in this fixture live in the root target; child content can be OOPIF.
|
||||
const address = await send("DOM.getFrameOwner", { frameId: node.frame.id });
|
||||
owner = await send("DOM.getBoxModel", { backendNodeId: address.backendNodeId });
|
||||
}
|
||||
frameRows.push({ frame: node.frame, local, owner });
|
||||
stack.push(...(node.childFrames ?? []));
|
||||
}
|
||||
report.frames[name] = { tree, snapshot, canvases: decodedCanvases(snapshot), frameRows };
|
||||
}
|
||||
check(
|
||||
"same-target snapshot includes nested documents",
|
||||
report.frames.root.snapshot.documents.length === 3,
|
||||
);
|
||||
check("oopif snapshot isolated from root", report.frames.oopif.snapshot.documents.length === 1);
|
||||
const sameRow = report.frames.root.frameRows.find(
|
||||
(row) => row.owner && row.local.innerWidth === 300,
|
||||
);
|
||||
const nestedRow = report.frames.root.frameRows.find(
|
||||
(row) => row.owner && row.local.innerWidth === 180,
|
||||
);
|
||||
check(
|
||||
"same frame content origin includes border",
|
||||
sameRow.owner.model.content[0] === 50 && sameRow.owner.model.content[1] === 610,
|
||||
);
|
||||
check(
|
||||
"nested owner quad is already target-root relative",
|
||||
nestedRow.owner.model.content[0] === 87 && nestedRow.owner.model.content[1] === 707,
|
||||
);
|
||||
const crossRow = report.frames.oopif.frameRows[0];
|
||||
check(
|
||||
"OOPIF owner content origin",
|
||||
crossRow.owner.model.content[0] === 410 && crossRow.owner.model.content[1] === 610,
|
||||
);
|
||||
const sameFrame = sameRow.frame.id;
|
||||
const world = await send("Page.createIsolatedWorld", {
|
||||
frameId: sameFrame,
|
||||
worldName: "bsk-design-probe",
|
||||
});
|
||||
await evaluate("scrollTo(0,10)", root, world.executionContextId);
|
||||
const frameScroll = await send("DOMSnapshot.captureSnapshot", {
|
||||
computedStyles: styles,
|
||||
includeDOMRects: true,
|
||||
});
|
||||
report.frameScroll = {
|
||||
documents: frameScroll.documents.map((d) => ({
|
||||
frameId: frameScroll.strings[d.frameId],
|
||||
scrollOffsetY: d.scrollOffsetY,
|
||||
})),
|
||||
canvases: decodedCanvases(frameScroll),
|
||||
local: await evaluate(
|
||||
"document.querySelector('canvas').getBoundingClientRect().toJSON()",
|
||||
root,
|
||||
world.executionContextId,
|
||||
),
|
||||
};
|
||||
check("frame scroll is local", report.frameScroll.local.y === 13);
|
||||
check(
|
||||
"frame snapshot stays document-relative after child scroll",
|
||||
report.frameScroll.canvases.find((n) => n.frameId === sameFrame).bounds[1] === 23,
|
||||
);
|
||||
await evaluate("scrollTo(0,0)", root, world.executionContextId);
|
||||
await evaluate(
|
||||
"document.querySelector('#same').style.transformOrigin='0 0';document.querySelector('#same').style.transform='scale(1.25)'",
|
||||
);
|
||||
const sameOwner = await send("DOM.getFrameOwner", { frameId: sameFrame });
|
||||
report.transformedOwner = await send("DOM.getBoxModel", {
|
||||
backendNodeId: sameOwner.backendNodeId,
|
||||
});
|
||||
check(
|
||||
"scaled frame owner content width",
|
||||
report.transformedOwner.model.content[2] - report.transformedOwner.model.content[0] === 375,
|
||||
);
|
||||
await evaluate("document.querySelector('#same').style.transform=''");
|
||||
const screenshot = await send("Page.captureScreenshot", { format: "png" });
|
||||
await writeFile(path.join(output, "frames.png"), Buffer.from(screenshot.data, "base64"));
|
||||
const before = (await send("Page.getFrameTree")).frameTree.frame;
|
||||
const oldDocument = (await send("DOM.getDocument")).root.backendNodeId;
|
||||
await evaluate("history.replaceState(null,'','#same-document')");
|
||||
const sameDocument = (await send("Page.getFrameTree")).frameTree.frame;
|
||||
const sameDocumentBackend = (await send("DOM.getDocument")).root.backendNodeId;
|
||||
await navigate(`${url}?noFrames=1&navigation=2`);
|
||||
const after = (await send("Page.getFrameTree")).frameTree.frame;
|
||||
const newDocument = (await send("DOM.getDocument")).root.backendNodeId;
|
||||
report.identity = { before, sameDocument, after, oldDocument, sameDocumentBackend, newDocument };
|
||||
check("same-document navigation preserves loader", before.loaderId === sameDocument.loaderId);
|
||||
check("cross-document navigation changes loader", before.loaderId !== after.loaderId);
|
||||
check("frame id survives document replacement", before.id === after.id);
|
||||
check(
|
||||
"document backend identity changes on navigation",
|
||||
oldDocument !== newDocument && oldDocument === sameDocumentBackend,
|
||||
);
|
||||
const documentTree = await send("DOM.getDocument");
|
||||
report.identity.htmlBeforeOpen = documentTree.root.children.find(
|
||||
(node) => node.nodeType === 1,
|
||||
).backendNodeId;
|
||||
const anchorNode = await send("DOM.querySelector", {
|
||||
nodeId: documentTree.root.nodeId,
|
||||
selector: "canvas",
|
||||
});
|
||||
const anchor = await send("DOM.describeNode", { nodeId: anchorNode.nodeId });
|
||||
await evaluate(
|
||||
"document.open(); document.write('<!doctype html><title>Replacement</title><canvas></canvas>'); document.close()",
|
||||
);
|
||||
report.identity.documentOpen = (await send("Page.getFrameTree")).frameTree.frame;
|
||||
const afterOpenTree = await send("DOM.getDocument");
|
||||
report.identity.documentOpenBackend = afterOpenTree.root.backendNodeId;
|
||||
report.identity.htmlAfterOpen = afterOpenTree.root.children.find(
|
||||
(node) => node.nodeType === 1,
|
||||
).backendNodeId;
|
||||
check(
|
||||
"document.open replaces documentElement identity",
|
||||
report.identity.htmlBeforeOpen !== report.identity.htmlAfterOpen,
|
||||
);
|
||||
try {
|
||||
const object = await send("DOM.resolveNode", { backendNodeId: anchor.node.backendNodeId });
|
||||
const connected = await send("Runtime.callFunctionOn", {
|
||||
objectId: object.object.objectId,
|
||||
functionDeclaration: "function(){return this.isConnected}",
|
||||
returnByValue: true,
|
||||
});
|
||||
report.identity.oldAnchorConnected = connected.result.value;
|
||||
} catch {
|
||||
report.identity.oldAnchorConnected = false;
|
||||
}
|
||||
check(
|
||||
"document.open invalidates old anchor even if document identifiers survive",
|
||||
report.identity.oldAnchorConnected === false,
|
||||
);
|
||||
report.counts = cdp.counts;
|
||||
report.checks = checks;
|
||||
process.stdout.write(
|
||||
`${JSON.stringify({ browser: report.browser.product, checks: checks.length, output })}\n`,
|
||||
);
|
||||
} catch (error) {
|
||||
report.error = error.stack;
|
||||
report.checks = checks;
|
||||
process.exitCode = 1;
|
||||
process.stderr.write(`${error.stack}\n`);
|
||||
} finally {
|
||||
await writeFile(path.join(output, "report.json"), JSON.stringify(report, null, 2));
|
||||
cdp.close();
|
||||
if (chrome && chrome.exitCode === null) {
|
||||
const exited = once(chrome, "exit");
|
||||
chrome.kill("SIGTERM");
|
||||
await Promise.race([exited, pause(2000)]);
|
||||
if (chrome.exitCode === null) {
|
||||
chrome.kill("SIGKILL");
|
||||
await exited;
|
||||
}
|
||||
}
|
||||
await new Promise((resolve) => server.close(resolve));
|
||||
await rm(profile, { recursive: true, force: true });
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<meta charset="utf-8">
|
||||
<title>Embedded Canvas table reproduction</title>
|
||||
<style>
|
||||
body { font: 16px sans-serif; }
|
||||
iframe { display: block; width: 800px; height: 440px; border: 8px solid #ccc; }
|
||||
</style>
|
||||
<h1>Embedded table</h1>
|
||||
<button type="button">Page action</button>
|
||||
<iframe title="Table application"></iframe>
|
||||
<script>
|
||||
document.querySelector("iframe").src = `http://localhost:${location.port}/grid.html`;
|
||||
</script>
|
||||
</html>
|
||||
Reference in New Issue
Block a user