mirror of
https://github.com/kenhuangus/wikiskill
synced 2026-09-22 23:34:25 +08:00
Add interactive Web UI (Flask + SSE workbench) with full design doc; fix rollback PURPOSE.md bug; add test-split evaluation with paired bootstrap; fix RunManager reentrant-lock deadlock
This commit is contained in:
DistributedApps.AI
@@ -53,11 +53,35 @@ this work:
|
||||
- **Paired bootstrap significance testing** (App. C) in `metrics.py`.
|
||||
- Early stop when `R_best == 1.0`.
|
||||
|
||||
## Quickstart
|
||||
## Web UI
|
||||
|
||||
An interactive workbench for running and inspecting the evolution loop:
|
||||
|
||||
```bash
|
||||
pip install -e ".[ui]" # installs Flask
|
||||
wikiskill serve # http://127.0.0.1:5000
|
||||
# or: python -m wikiskill.cli serve --port 5000
|
||||
```
|
||||
|
||||
Tabs (see `ui_design.md` for the full design):
|
||||
- **Run** — configure backend (offline Mock or any OpenAI-compatible endpoint),
|
||||
iterations, ReAct turns; start/cancel; live SSE console including the Skill
|
||||
Proposer's ReAct `read_file` steps.
|
||||
- **Dashboard** — R_best curve, accepted/rejected validation-gating bars, history table.
|
||||
- **Evaluation** — final skills vs no-skill baseline on the test split with paired
|
||||
bootstrap p-value (paper §4 + App. C); re-evaluate any workspace.
|
||||
- **Workspace** — browse the three-layer workspace (`raw/`, `wiki/`, `skills/`) and
|
||||
view any artifact (JSON pretty-printed, traces, diffs).
|
||||
- **Knowledge** — evolved skills (SKILL.md + PURPOSE.md), wiki pattern pages, index,
|
||||
evolution log, skill-impact audit trail.
|
||||
- **How it works** — the pipeline, Algorithm 1 pseudocode, and glossary mapped to the
|
||||
paper.
|
||||
|
||||
## Quickstart (CLI)
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev]"
|
||||
pytest # 14 tests incl. end-to-end loop
|
||||
pytest # 30 tests incl. UI + end-to-end loop
|
||||
python -m wikiskill.demo # offline demo with the deterministic MockLLM
|
||||
```
|
||||
|
||||
@@ -76,20 +100,20 @@ API key is read from `--api-key`, `WIKISKILL_API_KEY`, or `OPENAI_API_KEY`.
|
||||
Any OpenAI-compatible `/chat/completions` endpoint works (including local vLLM, as in
|
||||
the paper). `tasks.jsonl` lines: `{"id", "x", "y", "meta?"}`.
|
||||
|
||||
## Layout
|
||||
|
||||
Layout
|
||||
```
|
||||
src/wikiskill/
|
||||
├── workspace.py # three layers + patch engine + scoped read_file
|
||||
├── skills.py # skills layer, apply_proposal, skills-only snapshot/rollback
|
||||
├── agents.py # InferenceAgent, WikiMaintainer, ReAct SkillProposer, stratified_sample
|
||||
├── orchestrator.py # Algorithm 1 with gating/rollback + audit trail
|
||||
├── orchestrator.py # Algorithm 1 with gating/rollback + audit trail + test evaluation
|
||||
├── prompts.py # agent system prompts (adapted from paper Appendix E)
|
||||
├── llm.py # LLM protocol, OpenAI-compatible client, MockLLM, extract_json
|
||||
├── metrics.py # accuracy + paired bootstrap (1,000 iterations)
|
||||
├── datasets.py # dataclasses + synthetic demo dataset
|
||||
├── demo.py / cli.py
|
||||
└── tests/… # unit + end-to-end tests
|
||||
├── webapp/ # Flask UI (runner + api + SPA: see ui_design.md)
|
||||
├── demo.py / cli.py # `wikiskill demo | evolve | serve`
|
||||
└── tests/… # unit + end-to-end + UI tests
|
||||
```
|
||||
|
||||
## Notes & limitations (mirroring the paper)
|
||||
|
||||
@@ -11,17 +11,16 @@ wikiskill/
|
||||
│ ├── __init__.py
|
||||
│ ├── llm.py # LLM provider abstraction: MockLLM, OpenAICompatLLM
|
||||
│ ├── workspace.py # Three-layer workspace: raw/, wiki/, skills/ + patch engine
|
||||
│ ├── skills.py # Skills layer: SKILL.md + PURPOSE.md, snapshot/rollback
|
||||
│ ├── prompts.py # System prompts (Inference / Wiki Maintainer / Skill Proposer)
|
||||
│ ├── agents.py # InferenceAgent, WikiMaintainer, SkillProposer (ReAct)
|
||||
│ ├── orchestrator.py # Algorithm 1: evolution loop, gating, rollback, logging
|
||||
│ ├── orchestrator.py # Algorithm 1: loop, gating, rollback, test evaluation (§4)
|
||||
│ ├── metrics.py # accuracy metric + paired bootstrap significance test
|
||||
│ ├── datasets.py # Task/Trace dataclasses + demo dataset
|
||||
│ ├── webapp/ # Web UI (see ui_design.md): runner + Flask api + SPA
|
||||
│ ├── demo.py # End-to-end demo run with mock LLM
|
||||
│ └── cli.py # `wikiskill evolve` / `wikiskill demo` entry point
|
||||
└── tests/
|
||||
├── test_workspace.py
|
||||
├── test_gating.py
|
||||
└── test_e2e.py
|
||||
│ └── cli.py # `wikiskill demo | evolve | serve` entry point
|
||||
└── tests/ # unit, end-to-end, and UI tests
|
||||
```
|
||||
|
||||
## 2. Core data model (`datasets.py`)
|
||||
|
||||
@@ -91,3 +91,9 @@ experience into the wiki so that subsequent skill updates build on it.
|
||||
workspace with all three layers populated, and accepts/rolls back proposals correctly.
|
||||
3. Code mirrors paper notation (S_k, W_k, P_k, T_sample, R_best) and cites Algorithm 1 steps
|
||||
in docstrings.
|
||||
4. **Web UI (ui_design.md)**: `wikiskill serve` boots an interactive workbench that can
|
||||
start/cancel evolution runs, stream progress live, browse the three-layer workspace,
|
||||
inspect skills/wiki artifacts, and report test-split evaluation with paired bootstrap.
|
||||
UI covered by Flask test-client tests plus a live-server smoke test.
|
||||
5. **Post-run evaluation**: every run reports test-split accuracy for evolved skills vs
|
||||
no-skill baseline with paired-bootstrap significance (paper §4 + App. C).
|
||||
|
||||
@@ -11,6 +11,7 @@ dependencies = []
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = ["pytest>=7"]
|
||||
ui = ["flask>=3.0"]
|
||||
|
||||
[project.scripts]
|
||||
wikiskill = "wikiskill.cli:main"
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
flask>=3.0
|
||||
@@ -0,0 +1,87 @@
|
||||
"""Live smoke test: boots the real wikiskill serve stack and exercises it over HTTP."""
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
PORT = 8765
|
||||
BASE = f"http://127.0.0.1:{PORT}"
|
||||
ok = True
|
||||
|
||||
|
||||
def check(name, cond, extra=""):
|
||||
global ok
|
||||
print(f"[{'PASS' if cond else 'FAIL'}] {name} {extra}")
|
||||
ok = ok and bool(cond)
|
||||
|
||||
|
||||
def get(path, timeout=20):
|
||||
with urllib.request.urlopen(BASE + path, timeout=timeout) as r:
|
||||
return r.status, r.read().decode("utf-8")
|
||||
|
||||
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "wikiskill.cli", "serve", "--host", "127.0.0.1",
|
||||
"--port", str(PORT)],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
|
||||
|
||||
try:
|
||||
# wait for server
|
||||
for _ in range(100):
|
||||
try:
|
||||
st, body = get("/api/health", timeout=5)
|
||||
break
|
||||
except Exception:
|
||||
time.sleep(0.3)
|
||||
else:
|
||||
raise RuntimeError("server did not start")
|
||||
|
||||
check("health", st == 200 and '"ok":true' in body.replace(" ", ""))
|
||||
|
||||
st, body = get("/")
|
||||
check("index served", st == 200 and 'WikiSkill' in body and 'run-form' in body)
|
||||
|
||||
req = urllib.request.Request(BASE + "/api/run",
|
||||
data=json.dumps({"llm_backend": "mock",
|
||||
"iterations": 2,
|
||||
"workspace_root": "smoke_run"}).encode(),
|
||||
headers={"Content-Type": "application/json"})
|
||||
with urllib.request.urlopen(req, timeout=10) as r:
|
||||
st, body = r.status, r.read().decode()
|
||||
check("run started", st == 202 and '"run_id"' in body)
|
||||
|
||||
done = False
|
||||
for _ in range(200):
|
||||
st, body = get("/api/status")
|
||||
state = json.loads(body)["state"]
|
||||
if state and state["status"] in ("done", "error"):
|
||||
done = state["status"] == "done"
|
||||
break
|
||||
time.sleep(0.2)
|
||||
check("run completed (mock, live server)", done)
|
||||
|
||||
st, body = get("/api/workspace?root=smoke_run")
|
||||
data = json.loads(body)
|
||||
check("workspace browsable", st == 200 and "wiki/index.md" in data["files"],
|
||||
f"({sorted(data['files'])[:3]})")
|
||||
|
||||
req = urllib.request.Request(BASE + "/api/evaluate",
|
||||
data=json.dumps({"llm_backend": "mock",
|
||||
"workspace_root": "smoke_run"}).encode(),
|
||||
headers={"Content-Type": "application/json"})
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
ev = json.loads(r.read().decode())["evaluation"]
|
||||
check("evaluation end-to-end", ev["significant"] is True and
|
||||
ev["skilled_accuracy"] == 1.0,
|
||||
f"skilled={ev['skilled_accuracy']}")
|
||||
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
print("SMOKE TEST", "PASSED" if ok else "FAILED")
|
||||
sys.exit(0 if ok else 1)
|
||||
+19
-3
@@ -112,9 +112,18 @@ class SkillProposer:
|
||||
"""§3.2.3: P_k = M_P(W'_k, S_{k-1}, T_train,k). Multi-turn ReAct agent that
|
||||
reads the wiki index, skill-impact tracker, and traces on demand via read_file."""
|
||||
|
||||
def __init__(self, llm: LLM, ws: Workspace, skills: SkillsLayer, max_turns: int = 20):
|
||||
def __init__(self, llm: LLM, ws: Workspace, skills: SkillsLayer, max_turns: int = 20,
|
||||
on_step=None):
|
||||
self.llm, self.ws, self.skills = llm, ws, skills
|
||||
self.max_turns = max_turns
|
||||
self.on_step = on_step # callable(step: dict) for UI transparency
|
||||
|
||||
def _step(self, **payload):
|
||||
if self.on_step is not None:
|
||||
try:
|
||||
self.on_step({"proposer_step": True, **payload})
|
||||
except Exception: # pragma: no cover - never break evolution on UI hooks
|
||||
pass
|
||||
|
||||
def propose(self, outcome_summary: str, iteration: int) -> Optional[dict]:
|
||||
index = self.ws.wiki_index() or "(empty wiki)"
|
||||
@@ -126,19 +135,26 @@ class SkillProposer:
|
||||
"(raw/iter_.../...) as needed, then finish with your atomic proposal JSON.")
|
||||
messages = [{"role": "system", "content": SKILL_PROPOSER_SYSTEM},
|
||||
{"role": "user", "content": user}]
|
||||
for _ in range(self.max_turns):
|
||||
for turn in range(1, self.max_turns + 1):
|
||||
text = self.llm.chat(messages, temperature=0.2)
|
||||
parsed = extract_json(text)
|
||||
if parsed and parsed.get("action") in ("create", "patch"):
|
||||
self._step(turn=turn, kind="proposal", skill=parsed.get("skill"),
|
||||
action=parsed.get("action"))
|
||||
return parsed
|
||||
fm = re.search(r'"path"\s*:\s*"([^"]+)"', text)
|
||||
if fm:
|
||||
obs = self.ws.read_file(fm.group(1))
|
||||
path = fm.group(1)
|
||||
obs = self.ws.read_file(path)
|
||||
self._step(turn=turn, kind="read", path=path,
|
||||
found=not obs.startswith("ERROR"),
|
||||
snippet=obs[:200])
|
||||
messages.append({"role": "assistant", "content": text})
|
||||
messages.append({"role": "user",
|
||||
"content": f"read_file observation:\n{obs}"})
|
||||
continue
|
||||
messages.append({"role": "assistant", "content": text})
|
||||
self._step(turn=turn, kind="no_tool", snippet=text[:200])
|
||||
messages.append({"role": "user", "content":
|
||||
"Reply with a read_file tool call or the finish JSON."})
|
||||
return None
|
||||
|
||||
@@ -25,6 +25,11 @@ def main(argv=None) -> int:
|
||||
e.add_argument("--base-url", default=None)
|
||||
e.add_argument("--api-key", default=None)
|
||||
e.add_argument("--iterations", type=int, default=8)
|
||||
|
||||
s = sub.add_parser("serve", help="run the WikiSkill web UI (Flask)")
|
||||
s.add_argument("--host", default="127.0.0.1")
|
||||
s.add_argument("--port", type=int, default=5000)
|
||||
s.add_argument("--debug", action="store_true")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
if args.cmd == "demo":
|
||||
@@ -32,6 +37,17 @@ def main(argv=None) -> int:
|
||||
run_demo(args.root, args.iterations)
|
||||
return 0
|
||||
|
||||
if args.cmd == "serve":
|
||||
try:
|
||||
from .webapp import create_app
|
||||
except ImportError as exc:
|
||||
print("The UI requires Flask. Install it with: "
|
||||
"pip install -e '.[ui]' (or: pip install flask)", file=sys.stderr)
|
||||
return 2
|
||||
app = create_app()
|
||||
app.run(host=args.host, port=args.port, debug=args.debug, threaded=True)
|
||||
return 0
|
||||
|
||||
from .datasets import Dataset, Task, ToolSpec, load_jsonl
|
||||
from .llm import OpenAICompatLLM
|
||||
from .orchestrator import EvolutionOrchestrator
|
||||
|
||||
@@ -12,16 +12,21 @@ from .workspace import Workspace
|
||||
|
||||
def run_demo(root: str = "wikiskill_demo_workspace", K: int = 3) -> dict:
|
||||
ws = Workspace(root)
|
||||
skills_logger = []
|
||||
orch = EvolutionOrchestrator(
|
||||
llm=MockLLM(), dataset=make_demo_dataset(), ws=ws, tools=_demo_tools(), K=K)
|
||||
result = orch.run()
|
||||
eval_res = orch.evaluate_test()
|
||||
print("\nTest-set evaluation (final skills vs no-skill baseline):")
|
||||
print(f" skilled accuracy : {eval_res['skilled_accuracy']:.4f}")
|
||||
print(f" baseline accuracy: {eval_res['baseline_accuracy']:.4f}")
|
||||
print(f" bootstrap p : {eval_res['bootstrap_p_value']:.4f} "
|
||||
f"({'significant' if eval_res['significant'] else 'not significant'})")
|
||||
print("\nWorkspace layout:")
|
||||
for dirpath, _, files in os.walk(root):
|
||||
rel = os.path.relpath(dirpath, root)
|
||||
for fn in sorted(files):
|
||||
print(f" {os.path.join(rel, fn)}")
|
||||
return result
|
||||
return {**result, "evaluation": eval_res}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -31,7 +31,9 @@ class EvolutionOrchestrator:
|
||||
|
||||
def __init__(self, llm: LLM, dataset: Dataset, ws: Workspace, tools: list,
|
||||
metric: Callable[[List[bool]], float] = None, K: int = 8,
|
||||
max_react_turns: int = 20, log: Optional[Callable[[str], None]] = None):
|
||||
max_react_turns: int = 20, log: Optional[Callable[[str], None]] = None,
|
||||
proposer_on_step=None,
|
||||
cancel_event=None):
|
||||
from .metrics import accuracy
|
||||
self.llm = llm
|
||||
self.dataset = dataset
|
||||
@@ -41,17 +43,20 @@ class EvolutionOrchestrator:
|
||||
self.skills = SkillsLayer(ws)
|
||||
self.inference = InferenceAgent(llm, tools, ws, self.skills)
|
||||
self.maintainer = WikiMaintainer(llm, ws)
|
||||
self.proposer = SkillProposer(llm, ws, self.skills, max_turns=max_react_turns)
|
||||
self.proposer = SkillProposer(llm, ws, self.skills, max_turns=max_react_turns,
|
||||
on_step=proposer_on_step)
|
||||
self.log = log or (lambda s: print(s, flush=True))
|
||||
self.cancel_event = cancel_event # optional threading.Event for mid-run stop
|
||||
self.history: List[dict] = []
|
||||
self.r_best = -1.0
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
def _rollout(self, tasks, iteration: int, split: str) -> List[Trajectory]:
|
||||
def _rollout(self, tasks, iteration, split: str) -> List[Trajectory]:
|
||||
trajs = [self.inference.run(t) for t in tasks]
|
||||
for t in trajs:
|
||||
self.ws.add_raw_trace(iteration, t.task_id,
|
||||
{**t.to_dict(), "split": split, "iteration": iteration})
|
||||
{**t.to_dict(), "split": split, "iteration":
|
||||
str(iteration)})
|
||||
return trajs
|
||||
|
||||
def _score(self, trajs: List[Trajectory]) -> float:
|
||||
@@ -76,6 +81,9 @@ class EvolutionOrchestrator:
|
||||
def run(self) -> dict:
|
||||
self._baseline_validation()
|
||||
for k in range(1, self.K + 1):
|
||||
if self.cancel_event is not None and self.cancel_event.is_set():
|
||||
self.log("Run cancelled by user.")
|
||||
break
|
||||
if self.r_best >= 1.0:
|
||||
self.log("R_best reached 1.0; early stop.")
|
||||
break
|
||||
@@ -150,3 +158,47 @@ class EvolutionOrchestrator:
|
||||
self.log(f"\nEvolution complete. R_best={self.r_best:.4f}, skills={final_skills}")
|
||||
return result
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
def evaluate_test(self) -> dict:
|
||||
"""Post-evolution evaluation on the test split (paper §4).
|
||||
|
||||
Compares the final evolved skills against a no-skill baseline on D_test and
|
||||
reports both accuracies plus a paired bootstrap significance p-value
|
||||
(paper Appendix C, 1,000 iterations).
|
||||
"""
|
||||
from .metrics import paired_bootstrap
|
||||
# Eval traces are auxiliary (not part of the immutable evolution history);
|
||||
# clear any previous eval traces so re-evaluation works on the same workspace.
|
||||
for tag in ("eval_skilled", "eval_baseline"):
|
||||
d = os.path.join(self.ws.root, "raw", tag)
|
||||
if os.path.isdir(d):
|
||||
for fn in os.listdir(d):
|
||||
os.remove(os.path.join(d, fn))
|
||||
os.rmdir(d)
|
||||
|
||||
skilled_trajs = self._rollout(self.dataset.test, "eval_skilled", "test")
|
||||
skilled = [t.correct for t in skilled_trajs]
|
||||
skilled_acc = self.metric(skilled)
|
||||
|
||||
snap = self.skills.snapshot()
|
||||
try:
|
||||
for name in self.skills.list_skills():
|
||||
d = os.path.join(self.ws.root, "skills", name)
|
||||
for fn in os.listdir(d):
|
||||
os.remove(os.path.join(d, fn))
|
||||
os.rmdir(d)
|
||||
base_trajs = self._rollout(self.dataset.test, "eval_baseline", "test")
|
||||
finally:
|
||||
self.skills.restore(snap)
|
||||
base = [t.correct for t in base_trajs]
|
||||
base_acc = self.metric(base)
|
||||
|
||||
p = paired_bootstrap(skilled, base, n=1000, seed=42)
|
||||
self.log(f"Test evaluation: skilled={skilled_acc:.4f} "
|
||||
f"baseline={base_acc:.4f} bootstrap_p={p:.4f}")
|
||||
return {"skilled_accuracy": skilled_acc,
|
||||
"baseline_accuracy": base_acc,
|
||||
"bootstrap_p_value": p,
|
||||
"significant": p < 0.05,
|
||||
"n_test": len(skilled)}
|
||||
|
||||
|
||||
+14
-5
@@ -76,15 +76,24 @@ class SkillsLayer:
|
||||
raise ValueError(f"unknown proposal action: {action!r}")
|
||||
|
||||
# ---------- gating rollback (§3.2.4): skills only, wiki untouched ----------
|
||||
def snapshot(self) -> Dict[str, str]:
|
||||
return {name: self.skill_content(name) or "" for name in self.list_skills()}
|
||||
def snapshot(self) -> Dict[str, dict]:
|
||||
"""Snapshot the active skill state (SKILL.md + PURPOSE.md) so a rejected
|
||||
proposal can be rolled back exactly to S_{k-1}. The wiki is never included."""
|
||||
return {
|
||||
name: {"SKILL.md": self.skill_content(name) or "",
|
||||
"PURPOSE.md": self.purpose(name) or ""}
|
||||
for name in self.list_skills()
|
||||
}
|
||||
|
||||
def restore(self, snap: Dict[str, str]) -> None:
|
||||
def restore(self, snap: Dict[str, dict]) -> None:
|
||||
for name in self.list_skills():
|
||||
if name not in snap:
|
||||
d = os.path.join(self.ws.root, "skills", name)
|
||||
for fn in os.listdir(d):
|
||||
os.remove(os.path.join(d, fn))
|
||||
os.rmdir(d)
|
||||
for name, content in snap.items():
|
||||
self.ws.write_file(os.path.join("skills", name, "SKILL.md"), content)
|
||||
for name, files in snap.items():
|
||||
self.ws.write_file(os.path.join("skills", name, "SKILL.md"),
|
||||
files["SKILL.md"])
|
||||
self.ws.write_file(os.path.join("skills", name, "PURPOSE.md"),
|
||||
files["PURPOSE.md"])
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
"""WikiSkill web UI package (ui_design.md)."""
|
||||
from .api import create_app
|
||||
|
||||
__all__ = ["create_app"]
|
||||
@@ -0,0 +1,183 @@
|
||||
"""Flask application factory + REST/SSE endpoints for the WikiSkill UI."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
from flask import Blueprint, Flask, Response, jsonify, render_template, request
|
||||
|
||||
from .. import __version__
|
||||
from .runner import RunManager
|
||||
|
||||
bp = Blueprint("wikiskill_ui", __name__)
|
||||
|
||||
_manager: RunManager = None # module-level singleton
|
||||
|
||||
|
||||
def get_manager() -> RunManager:
|
||||
global _manager
|
||||
if _manager is None:
|
||||
_manager = RunManager()
|
||||
return _manager
|
||||
|
||||
|
||||
def create_app(test_config=None) -> Flask:
|
||||
app = Flask(__name__)
|
||||
app.config.from_mapping(SECRET_KEY="wikiskill-local-dev")
|
||||
if test_config:
|
||||
app.config.update(test_config)
|
||||
app.register_blueprint(bp)
|
||||
return app
|
||||
|
||||
|
||||
def _safe_root(root: str) -> str:
|
||||
"""Reject absolute paths and path traversal; normalize to a relative name."""
|
||||
root = (root or "").strip().replace("\\", "/").strip("/")
|
||||
if not root:
|
||||
raise ValueError("workspace root must not be empty")
|
||||
if os.path.isabs(root):
|
||||
raise ValueError("workspace root must be a relative path")
|
||||
if any(part == ".." for part in root.split("/")):
|
||||
raise ValueError("workspace root must not contain '..'")
|
||||
return root
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@bp.get("/")
|
||||
def index():
|
||||
return render_template("index.html")
|
||||
|
||||
|
||||
@bp.get("/api/health")
|
||||
def health():
|
||||
return jsonify({"ok": True, "version": __version__})
|
||||
|
||||
|
||||
@bp.post("/api/run")
|
||||
def start_run():
|
||||
config = request.get_json(silent=True) or {}
|
||||
backend = config.get("llm_backend", "mock")
|
||||
if backend not in ("mock", "openai"):
|
||||
return jsonify({"error": "llm_backend must be 'mock' or 'openai'"}), 400
|
||||
if backend == "openai" and not config.get("model"):
|
||||
return jsonify({"error": "'model' is required when llm_backend=openai"}), 400
|
||||
root = config.get("workspace_root")
|
||||
if root:
|
||||
try:
|
||||
_safe_root(root)
|
||||
except ValueError as e:
|
||||
return jsonify({"error": str(e)}), 400
|
||||
try:
|
||||
info = get_manager().start(config)
|
||||
except PermissionError as e:
|
||||
return jsonify({"error": str(e)}), 409
|
||||
return jsonify(info), 202
|
||||
|
||||
|
||||
@bp.get("/api/status")
|
||||
def status():
|
||||
return jsonify(get_manager().status())
|
||||
|
||||
|
||||
@bp.get("/api/stream")
|
||||
def stream():
|
||||
def gen():
|
||||
yield from get_manager().stream()
|
||||
return Response(gen(), mimetype="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache",
|
||||
"X-Accel-Buffering": "no"})
|
||||
|
||||
|
||||
@bp.post("/api/cancel")
|
||||
def cancel():
|
||||
return jsonify({"cancelled": get_manager().cancel()})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
_TREE_SKIP = {".pytest_cache", "__pycache__", ".git"}
|
||||
_TEXT_EXT = {".md", ".json", ".txt", ".py", ".toml", ".diff", ".log"}
|
||||
|
||||
|
||||
@bp.get("/api/workspace")
|
||||
def workspace():
|
||||
try:
|
||||
root = _safe_root(request.args.get("root", ""))
|
||||
except ValueError as e:
|
||||
return jsonify({"error": str(e)}), 400
|
||||
if not os.path.isdir(root):
|
||||
return jsonify({"error": f"workspace '{root}' does not exist",
|
||||
"root": root}), 404
|
||||
return jsonify({"root": root, "tree": _build_tree(root),
|
||||
"files": _read_text_files(root)})
|
||||
|
||||
|
||||
def _build_tree(root: str) -> dict:
|
||||
def build(path: str) -> dict:
|
||||
name = os.path.basename(path) or root
|
||||
rel = os.path.relpath(path, root).replace("\\", "/")
|
||||
node = {"name": name, "path": rel, "type": "dir", "children": []}
|
||||
for entry in sorted(os.listdir(path)):
|
||||
if entry in _TREE_SKIP:
|
||||
continue
|
||||
full = os.path.join(path, entry)
|
||||
if os.path.isdir(full):
|
||||
node["children"].append(build(full))
|
||||
else:
|
||||
node["children"].append({
|
||||
"name": entry,
|
||||
"path": os.path.relpath(full, root).replace("\\", "/"),
|
||||
"type": "file"})
|
||||
return node
|
||||
return build(root)
|
||||
|
||||
|
||||
def _read_text_files(root: str) -> dict:
|
||||
"""Collect text-file contents (bounded) for the UI viewer."""
|
||||
out = {}
|
||||
for dirpath, dirnames, filenames in os.walk(root):
|
||||
dirnames[:] = [d for d in dirnames if d not in _TREE_SKIP]
|
||||
for fn in filenames:
|
||||
if os.path.splitext(fn)[1] not in _TEXT_EXT:
|
||||
continue
|
||||
full = os.path.join(dirpath, fn)
|
||||
rel = os.path.relpath(full, root).replace("\\", "/")
|
||||
try:
|
||||
with open(full, "r", encoding="utf-8") as f:
|
||||
out[rel] = f.read(200_000)
|
||||
except (OSError, UnicodeDecodeError):
|
||||
out[rel] = "(unreadable/binary)"
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@bp.post("/api/evaluate")
|
||||
def evaluate():
|
||||
"""Re-evaluate final skills vs baseline on the test split (paper §4)."""
|
||||
config = request.get_json(silent=True) or {}
|
||||
backend = config.get("llm_backend", "mock")
|
||||
if backend not in ("mock", "openai"):
|
||||
return jsonify({"error": "llm_backend must be 'mock' or 'openai'"}), 400
|
||||
try:
|
||||
root = _safe_root(config.get("workspace_root", ""))
|
||||
except ValueError as e:
|
||||
return jsonify({"error": str(e)}), 400
|
||||
if not os.path.isdir(root):
|
||||
return jsonify({"error": f"workspace '{root}' does not exist"}), 404
|
||||
|
||||
from ..datasets import _demo_tools, make_demo_dataset
|
||||
from ..llm import MockLLM, OpenAICompatLLM
|
||||
from ..orchestrator import EvolutionOrchestrator
|
||||
from ..workspace import Workspace
|
||||
|
||||
if backend == "openai":
|
||||
llm = OpenAICompatLLM(model=config.get("model", ""),
|
||||
base_url=config.get("base_url") or None,
|
||||
api_key=config.get("api_key") or None)
|
||||
else:
|
||||
llm = MockLLM()
|
||||
orch = EvolutionOrchestrator(llm=llm, dataset=make_demo_dataset(),
|
||||
ws=Workspace(root), tools=_demo_tools(),
|
||||
K=int(config.get("iterations", 8)),
|
||||
log=lambda msg: None)
|
||||
res = orch.evaluate_test()
|
||||
return jsonify({"evaluation": res, "workspace_root": root})
|
||||
@@ -0,0 +1,166 @@
|
||||
"""RunManager: executes the WikiSkill evolution loop in a background thread and
|
||||
streams progress to the browser via Server-Sent Events (ui_design.md §6–§7)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
import json
|
||||
import os
|
||||
import queue
|
||||
import threading
|
||||
import traceback as _tb
|
||||
import uuid
|
||||
from typing import Optional
|
||||
|
||||
from ..datasets import _demo_tools, make_demo_dataset
|
||||
from ..llm import MockLLM, OpenAICompatLLM
|
||||
from ..orchestrator import EvolutionOrchestrator
|
||||
from ..workspace import Workspace
|
||||
|
||||
|
||||
def _is_fresh_root(path: str) -> bool:
|
||||
"""True if the workspace root has no prior evolution history (raw/iter_*)."""
|
||||
raw = os.path.join(path, "raw")
|
||||
if not os.path.isdir(raw):
|
||||
return True
|
||||
return not any(d.startswith("iter_") for d in os.listdir(raw))
|
||||
|
||||
|
||||
class RunManager:
|
||||
"""Single-flight run manager. `start` spawns a daemon thread; `stream` yields SSE."""
|
||||
|
||||
def __init__(self):
|
||||
self._lock = threading.Lock()
|
||||
self._state: Optional[dict] = None
|
||||
self._events: "queue.Queue[dict]" = queue.Queue()
|
||||
self._cancel_event = threading.Event()
|
||||
|
||||
# ---- public API ----------------------------------------------------
|
||||
@property
|
||||
def active(self) -> bool:
|
||||
with self._lock:
|
||||
return bool(self._state and self._state.get("status")
|
||||
in ("starting", "running", "evaluating"))
|
||||
|
||||
def status(self) -> dict:
|
||||
# Read without holding the lock while computing `active` — the property
|
||||
# also acquires the (non-reentrant) lock, which would deadlock.
|
||||
state = self._state
|
||||
return {"active": bool(state and state.get("status")
|
||||
in ("starting", "running", "evaluating")),
|
||||
"state": state}
|
||||
|
||||
def start(self, config: dict) -> dict:
|
||||
with self._lock:
|
||||
if self._state and self._state.get("status") in (
|
||||
"starting", "running", "evaluating"):
|
||||
raise PermissionError("another evolution run is already active")
|
||||
run_id = uuid.uuid4().hex[:12]
|
||||
self._state = {"run_id": run_id, "config": dict(config), "status":
|
||||
"starting", "events": [], "result": None, "error": None}
|
||||
self._cancel_event = threading.Event()
|
||||
threading.Thread(target=self._execute, args=(run_id, dict(config)),
|
||||
daemon=True).start()
|
||||
return {"run_id": run_id, "active": True}
|
||||
|
||||
def cancel(self) -> bool:
|
||||
self._cancel_event.set()
|
||||
return True
|
||||
|
||||
def _reset_for_tests(self) -> None:
|
||||
"""Test-only: drop state and queue so a fresh run can start."""
|
||||
with self._lock:
|
||||
self._state = None
|
||||
self._cancel_event = threading.Event()
|
||||
while True:
|
||||
try:
|
||||
self._events.get_nowait()
|
||||
except queue.Empty:
|
||||
break
|
||||
|
||||
# ---- streaming -----------------------------------------------------
|
||||
def _emit(self, **payload) -> None:
|
||||
self._events.put(dict(payload))
|
||||
with self._lock:
|
||||
if self._state is not None:
|
||||
self._state["events"].append(dict(payload))
|
||||
self._state["events"] = self._state["events"][-500:]
|
||||
|
||||
def stream(self):
|
||||
"""SSE generator: emits events; closes after done/error."""
|
||||
while True:
|
||||
try:
|
||||
payload = self._events.get(timeout=15)
|
||||
except queue.Empty:
|
||||
yield ":\n\n" # heartbeat
|
||||
continue
|
||||
data = json.dumps(payload)
|
||||
yield f"data: {data}\n\n"
|
||||
if payload.get("type") in ("done", "error"):
|
||||
break
|
||||
|
||||
# ---- execution -----------------------------------------------------
|
||||
def _build_llm(self, config: dict):
|
||||
backend = config.get("llm_backend", "mock")
|
||||
if backend == "openai":
|
||||
return OpenAICompatLLM(
|
||||
model=config.get("model", ""),
|
||||
base_url=config.get("base_url") or None,
|
||||
api_key=config.get("api_key") or None,
|
||||
temperature=float(config.get("temperature", 0.7)))
|
||||
return MockLLM()
|
||||
|
||||
def _root_for(self, config: dict) -> str:
|
||||
root = config.get("workspace_root") or ""
|
||||
if root:
|
||||
return str(root).strip().strip("/").replace("\\", "/").strip("/")
|
||||
stamp = _dt.datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
return f"runs/run_{stamp}"
|
||||
|
||||
def _execute(self, run_id: str, config: dict) -> None:
|
||||
try:
|
||||
self._emit(type="status", phase="starting",
|
||||
message="Preparing run...")
|
||||
root = self._root_for(config)
|
||||
if not _is_fresh_root(root):
|
||||
raise ValueError(
|
||||
f"workspace root '{root}' already contains evolution history; "
|
||||
"choose a fresh root (the raw layer is immutable)")
|
||||
llm = self._build_llm(config)
|
||||
ws = Workspace(root)
|
||||
orch = EvolutionOrchestrator(
|
||||
llm=llm, dataset=make_demo_dataset(), ws=ws, tools=_demo_tools(),
|
||||
K=int(config.get("iterations", 8)),
|
||||
max_react_turns=int(config.get("max_react_turns", 20)),
|
||||
cancel_event=self._cancel_event,
|
||||
log=lambda msg: self._emit(type="log", message=msg),
|
||||
proposer_on_step=lambda ev: self._emit(type="react", **ev))
|
||||
self._emit(type="status", phase="running", message=f"root={root}")
|
||||
result = orch.run()
|
||||
self._emit(type="status", phase="evaluating",
|
||||
message="Evaluating evolved skills on the test split...")
|
||||
evaluation = orch.evaluate_test()
|
||||
with self._lock:
|
||||
if self._state is not None:
|
||||
self._state["status"] = "done"
|
||||
self._state["result"] = {"evolution": result,
|
||||
"evaluation": evaluation,
|
||||
"workspace_root": root}
|
||||
self._emit(type="done", result=self._state["result"] if self._state else
|
||||
{"evolution": result, "evaluation": evaluation,
|
||||
"workspace_root": root})
|
||||
except Exception as exc: # noqa: BLE001 - surface any failure to the UI
|
||||
err = f"{type(exc).__name__}: {exc}"
|
||||
print(err, flush=True)
|
||||
print(_tb.format_exc(), flush=True)
|
||||
with self._lock:
|
||||
if self._state is not None:
|
||||
self._state["status"] = "error"
|
||||
self._state["error"] = err + "\n" + _tb.format_exc()
|
||||
self._emit(type="error", error=str(exc))
|
||||
self._state = None
|
||||
self._cancel_event = threading.Event()
|
||||
while True:
|
||||
try:
|
||||
self._events.get_nowait()
|
||||
except queue.Empty:
|
||||
break
|
||||
@@ -0,0 +1,437 @@
|
||||
"use strict";
|
||||
/* WikiSkill UI client — vanilla JS, no dependencies (ui_design.md §6–§7). */
|
||||
|
||||
const $ = (id) => document.getElementById(id);
|
||||
const S = {
|
||||
ws: null, // workspace data {root, tree, files}
|
||||
lastResult: null, // {evolution, evaluation, workspace_root}
|
||||
activeRoot: null,
|
||||
};
|
||||
|
||||
/* ---------- utils ---------- */
|
||||
function esc(s) {
|
||||
return String(s ?? "").replace(/[&<>"']/g, (c) => ({
|
||||
"&": "&", "<": "<", ">": ">", '"': """, "'": "'",
|
||||
}[c]));
|
||||
}
|
||||
async function api(path, opts = {}) {
|
||||
const res = await fetch(path, {
|
||||
headers: { "Content-Type": "application/json" },
|
||||
method: opts.method || (opts.body ? "POST" : "GET"),
|
||||
body: opts.body ? JSON.stringify(opts.body) : undefined,
|
||||
});
|
||||
let data = null;
|
||||
try { data = await res.json(); } catch (_) { /* empty */ }
|
||||
if (!res.ok) {
|
||||
const msg = (data && data.error) || `HTTP ${res.status}`;
|
||||
throw new Error(msg);
|
||||
}
|
||||
return data;
|
||||
}
|
||||
function fmtPct(v) { return `${(v * 100).toFixed(1)}%`; }
|
||||
|
||||
/* ---------- tabs ---------- */
|
||||
function switchTab(name) {
|
||||
document.querySelectorAll(".tab").forEach((b) =>
|
||||
b.classList.toggle("is-active", b.dataset.tab === name));
|
||||
document.querySelectorAll(".tab-panel").forEach((p) =>
|
||||
p.classList.toggle("is-active", p.id === `tab-${name}`));
|
||||
}
|
||||
document.querySelectorAll(".tab").forEach((b) =>
|
||||
b.addEventListener("click", () => switchTab(b.dataset.tab)));
|
||||
|
||||
/* ---------- status pill ---------- */
|
||||
function setStatus(status, label) {
|
||||
const pill = $("status-pill");
|
||||
pill.textContent = label || status;
|
||||
const cls = { idle: "pill-idle", starting: "pill-running", running: "pill-running",
|
||||
evaluating: "pill-evaluating", done: "pill-done", error: "pill-error" };
|
||||
pill.className = `pill ${cls[status] || "pill-idle"}`;
|
||||
$("btn-cancel").classList.toggle("hidden",
|
||||
!["starting", "running", "evaluating"].includes(status));
|
||||
$("btn-start").disabled = ["starting", "running", "evaluating"].includes(status);
|
||||
}
|
||||
|
||||
/* ---------- console ---------- */
|
||||
function logLine(text, cls) {
|
||||
const con = $("console");
|
||||
const div = document.createElement("span");
|
||||
div.className = `ln ${cls || "ln-tail"}`;
|
||||
div.textContent = text;
|
||||
con.appendChild(div);
|
||||
con.scrollTop = con.scrollHeight;
|
||||
}
|
||||
function renderEvent(ev) {
|
||||
if (ev.type === "log") logLine(ev.message, "ln-tail");
|
||||
else if (ev.type === "status") {
|
||||
logLine(`—— ${ev.message || ev.phase} ——`, "ln-cmd");
|
||||
if (ev.phase) setStatus(ev.phase === "evaluating" ? "evaluating" :
|
||||
["starting", "running"].includes(ev.phase) ? "running" : ev.phase);
|
||||
} else if (ev.type === "react") {
|
||||
if (ev.kind === "read") {
|
||||
logLine(` [proposer] read ${ev.path} ${ev.found ? "" : "(MISSING)"}`, "ln-react");
|
||||
} else if (ev.kind === "proposal") {
|
||||
logLine(` [proposer] → proposal ${ev.action} '${ev.skill}'`, "ln-react");
|
||||
} else {
|
||||
logLine(` [proposer] ${esc(ev.kind)}`, "ln-react");
|
||||
}
|
||||
} else if (ev.type === "done") {
|
||||
S.lastResult = ev.result;
|
||||
setStatus("done", "done");
|
||||
if (ev.result) {
|
||||
showDashboard(ev.result.evolution);
|
||||
showEval(ev.result.evaluation);
|
||||
S.activeRoot = ev.result.workspace_root;
|
||||
$("log-root-label").textContent = `→ ${S.activeRoot}`;
|
||||
}
|
||||
logLine("✔ run finished", "ln-accept");
|
||||
} else if (ev.type === "error") {
|
||||
logLine(`✖ error: ${ev.error}`, "ln-reject");
|
||||
setStatus("error", "error");
|
||||
}
|
||||
}
|
||||
|
||||
/* ---------- run form ---------- */
|
||||
$("cfg-backend").addEventListener("change", () => {
|
||||
$("openai-fields").classList.toggle("hidden", $("cfg-backend").value !== "openai");
|
||||
});
|
||||
$("run-form").addEventListener("submit", async (e) => {
|
||||
/* ---------- SSE ---------- */
|
||||
let esSource = null;
|
||||
function connectStream() {
|
||||
if (esSource) esSource.close();
|
||||
setStatus("running", "running…");
|
||||
esSource = new EventSource("/api/stream");
|
||||
esSource.onmessage = (m) => {
|
||||
let ev;
|
||||
try { ev = JSON.parse(m.data); } catch (_) { return; }
|
||||
renderEvent(ev);
|
||||
if (ev.type === "done" || ev.type === "error") esSource.close();
|
||||
};
|
||||
esSource.onerror = () => { if (esSource) esSource.close(); };
|
||||
}
|
||||
|
||||
/* ---------- dashboard ---------- */
|
||||
function showDashboard(evo) {
|
||||
$("dash-emptystate").classList.add("hidden");
|
||||
$("dash-charts").classList.remove("hidden");
|
||||
const history = evo.history || [];
|
||||
drawRbest(history);
|
||||
drawGate(history);
|
||||
const tbody = document.querySelector("#dash-table tbody");
|
||||
tbody.innerHTML = "";
|
||||
if (!history.length) {
|
||||
tbody.innerHTML = '<tr><td colspan="5" class="muted">No proposals evaluated.</td></tr>';
|
||||
return;
|
||||
}
|
||||
for (const h of history) {
|
||||
const tr = document.createElement("tr");
|
||||
const outcome = h.accepted ? "accepted" : (h.proposal ? "rejected" : "none");
|
||||
tr.innerHTML = `<td>${h.iteration}</td><td>${esc(h.proposal || "—")}</td>
|
||||
<td>${esc(h.action || "—")}</td>
|
||||
<td>${h.val_score == null ? "—" : h.val_score.toFixed(4)}</td>
|
||||
<td><span class="badge ${outcome}">${outcome}</span></td>`;
|
||||
tbody.appendChild(tr);
|
||||
}
|
||||
}
|
||||
function drawRbest(history) {
|
||||
const svg = $("chart-rbest");
|
||||
// Reconstruct R_best trajectory: starts at baseline (0 by convention here, or the
|
||||
// first accepted score), then holds between accepted proposals.
|
||||
const accepted = history.filter((h) => h.accepted && h.val_score != null);
|
||||
const start = accepted.length ? firstVal(history) : 0;
|
||||
if (!accepted.length) {
|
||||
svg.innerHTML = `<text x="12" y="70" fill="#999">no accepted proposals (R_best = baseline)</text>`;
|
||||
return;
|
||||
}
|
||||
const pts = [{ lab: "base", v: start }];
|
||||
let cur = start;
|
||||
for (const h of history) {
|
||||
if (h.accepted) cur = h.val_score;
|
||||
pts.push({ lab: `k${h.iteration}`, v: cur });
|
||||
}
|
||||
renderLine(svg, pts);
|
||||
}
|
||||
function firstVal(history) {
|
||||
for (const h of history) if (h.accepted) return h.val_score;
|
||||
return 0;
|
||||
}
|
||||
function renderLine(svg, pts) {
|
||||
const W = 400, H = 180, pad = { l: 34, r: 10, t: 12, b: 24 };
|
||||
const minV = Math.min(...pts.map((p) => p.v), 0);
|
||||
const maxV = Math.max(...pts.map((p) => p.v), 1);
|
||||
const span = Math.max(0.2, maxV - minV);
|
||||
const x = (i) => pad.l + (i / Math.max(1, pts.length - 1)) * (W - pad.l - pad.r);
|
||||
const y = (v) => H - pad.b - ((v - minV) / span) * (H - pad.t - pad.b);
|
||||
let html = "";
|
||||
for (let g = 0; g <= 4; g++) {
|
||||
const gv = minV + (span * g) / 4;
|
||||
const gy = y(gv);
|
||||
html += `<line x1="${pad.l}" y1="${gy}" x2="${W - pad.r}" y2="${gy}" stroke="#efefea"/>`;
|
||||
html += `<text x="2" y="${gy + 3}" fill="#999">${gv.toFixed(2)}</text>`;
|
||||
}
|
||||
const px = pts.map((p, i) => `${x(i).toFixed(1)},${y(p.v).toFixed(1)}`).join(" ");
|
||||
html += `<polyline points="${px}" fill="none" stroke="#3b5bdb" stroke-width="2"/>`;
|
||||
pts.forEach((p, i) => {
|
||||
html += `<circle cx="${x(i)}" cy="${y(p.v)}" r="3" fill="#3b5bdb">`;
|
||||
html += `<title>${p.lab}: ${p.v.toFixed(3)}</title></circle>`;
|
||||
});
|
||||
html += `<text x="${pad.l}" y="${H - 6}" fill="#999">baseline</text>`;
|
||||
svg.innerHTML = html;
|
||||
}
|
||||
function drawGate(history) {
|
||||
const svg = $("chart-gate");
|
||||
const W = 400, H = 180, pad = { l: 34, r: 10, t: 12, b: 24 };
|
||||
const vals = history.filter((h) => h.val_score != null);
|
||||
const maxV = Math.max(1, ...vals.map((v) => v.val_score));
|
||||
let html = "";
|
||||
for (let g = 0; g <= 4; g++) {
|
||||
const gy = H - pad.b - (g / 4) * (H - pad.t - pad.b);
|
||||
html += `<line x1="${pad.l}" y1="${gy}" x2="${W - pad.r}" y2="${gy}" stroke="#efefea"/>`;
|
||||
}
|
||||
if (!vals.length) {
|
||||
html += `<text x="${pad.l}" y="60" fill="#999">no proposals evaluated yet</text>`;
|
||||
svg.innerHTML = html; return;
|
||||
}
|
||||
const bw = Math.min(46, (W - pad.l - pad.r) / vals.length - 8);
|
||||
vals.forEach((h, i) => {
|
||||
const bh = (h.val_score / maxV) * (H - pad.t - pad.b);
|
||||
const bx = pad.l + i * ((W - pad.l - pad.r) / vals.length) + 4;
|
||||
const by = H - pad.b - bh;
|
||||
const color = h.accepted ? "#099268" : "#e03131";
|
||||
html += `<rect x="${bx}" y="${by}" width="${bw}" height="${Math.max(2, bh)}" fill="${color}">`;
|
||||
html += `<title>iter ${h.iteration}: ${h.val_score.toFixed(3)} ${h.accepted ? "accepted" : "rejected"}</title></rect>`;
|
||||
html += `<text x="${bx + bw / 2 - 8}" y="${H - 8}" fill="#999">k${h.iteration}</text>`;
|
||||
});
|
||||
svg.innerHTML = html;
|
||||
}
|
||||
e.preventDefault();
|
||||
$("form-error").textContent = "";
|
||||
const cfg = {
|
||||
llm_backend: $("cfg-backend").value,
|
||||
iterations: parseInt($("cfg-iterations").value, 10),
|
||||
max_react_turns: parseInt($("cfg-react").value, 10),
|
||||
workspace_root: $("cfg-root").value.trim() || undefined,
|
||||
};
|
||||
if (cfg.llm_backend === "openai") {
|
||||
cfg.model = $("cfg-model").value.trim();
|
||||
cfg.base_url = $("cfg-base-url").value.trim() || undefined;
|
||||
cfg.api_key = $("cfg-api-key").value.trim() || undefined;
|
||||
cfg.temperature = parseFloat($("cfg-temperature").value);
|
||||
}
|
||||
try {
|
||||
await api("/api/run", { body: cfg });
|
||||
$("console").textContent = "";
|
||||
connectStream();
|
||||
} catch (err) {
|
||||
/* ---------- evaluation ---------- */
|
||||
function showEval(ev) {
|
||||
if (!ev) return;
|
||||
$("eval-emptystate").classList.add("hidden");
|
||||
$("eval-cards").classList.remove("hidden");
|
||||
$("eval-bars").classList.remove("hidden");
|
||||
$("eval-verdict").classList.remove("hidden");
|
||||
const sk = ev.skilled_accuracy, base = ev.baseline_accuracy;
|
||||
const d = sk - base;
|
||||
$("eval-cards").innerHTML = `
|
||||
<div class="kpi"><div class="v">${fmtPct(sk)}</div><div class="l">with evolved skills</div></div>
|
||||
<div class="kpi"><div class="v">${fmtPct(base)}</div><div class="l">no skills (baseline)</div></div>
|
||||
<div class="kpi"><div class="v">${d >= 0 ? "+" : ""}${fmtPct(d)}</div><div class="l">Δ</div></div>
|
||||
<div class="kpi"><div class="v">${ev.bootstrap_p_value.toFixed(4)}</div><div class="l">bootstrap p (1000)</div></div>`;
|
||||
const W = 420, H = 90;
|
||||
const maxV = Math.max(sk, base, 0.01);
|
||||
const bar = (label, v, color) => `
|
||||
<text x="4" y="26" fill="#666">${label}</text>
|
||||
<rect x="4" y="36" width="${Math.max(4, (v / maxV) * (W - 8))}" height="18" fill="${color}" rx="4">
|
||||
<title>${fmtPct(v)}</title></rect>
|
||||
<text x="6" y="76" fill="#444" font-size="11">${fmtPct(v)}</text>`;
|
||||
$("eval-bars").innerHTML = `
|
||||
<svg viewBox="0 0 430 92" class="chart">${bar("evolved skills", sk, "#099268")}
|
||||
${bar("no-skill baseline", base, "#adb5bd")}</svg>`;
|
||||
const verdict = $("eval-verdict");
|
||||
if (ev.significant && d > 0) {
|
||||
verdict.className = "verdict ok";
|
||||
verdict.textContent = `✔ Evolved skills are significantly better than the no-skill baseline (p = ${ev.bootstrap_p_value.toFixed(4)} < 0.05).`;
|
||||
} else if (d < 0) {
|
||||
verdict.className = "verdict bad";
|
||||
verdict.textContent = `✖ Skills underperform the baseline on the test split.`;
|
||||
} else {
|
||||
verdict.className = "verdict neutral";
|
||||
verdict.textContent = `∽ No significant difference from the baseline (p = ${ev.bootstrap_p_value.toFixed(4)}).`;
|
||||
}
|
||||
}
|
||||
$("btn-evaluate").addEventListener("click", async () => {
|
||||
$("eval-error").textContent = "";
|
||||
const root = $("eval-root").value.trim() || S.activeRoot;
|
||||
if (!root) {
|
||||
$("eval-error").textContent = "No workspace yet — run an evolution first, or enter a root.";
|
||||
return;
|
||||
}
|
||||
const btn = $("btn-evaluate");
|
||||
btn.disabled = true; btn.textContent = "Evaluating…";
|
||||
try {
|
||||
$("eval-load-form").classList.remove("hidden");
|
||||
const res = await api("/api/evaluate", { body: { llm_backend: "mock",
|
||||
workspace_root: root, iterations: 3 } });
|
||||
showEval(res.evaluation);
|
||||
S.activeRoot = root;
|
||||
} catch (err) {
|
||||
$("eval-error").textContent = err.message;
|
||||
} finally {
|
||||
btn.disabled = false; btn.textContent = "Re-evaluate current workspace";
|
||||
}
|
||||
});
|
||||
$("btn-eval-load").addEventListener("click", async () => {
|
||||
const root = $("eval-root").value.trim();
|
||||
if (!root) { $("eval-error").textContent = "Enter a workspace root."; return; }
|
||||
try {
|
||||
const res = await api("/api/evaluate", { body: { llm_backend: "mock",
|
||||
workspace_root: root, iterations: 3 } });
|
||||
showEval(res.evaluation);
|
||||
} catch (err) { $("eval-error").textContent = err.message; }
|
||||
});
|
||||
/* ---------- workspace explorer ---------- */
|
||||
async function loadWorkspace(root) {
|
||||
$("ws-error").textContent = "";
|
||||
try {
|
||||
const data = await api(`/api/workspace?root=${encodeURIComponent(root)}`);
|
||||
S.ws = data;
|
||||
S.activeRoot = root;
|
||||
$("ws-split").classList.remove("hidden");
|
||||
renderTree(data.tree);
|
||||
showFile("run_state.json");
|
||||
renderKnowledge(data.files);
|
||||
$("eval-root").value = root;
|
||||
$("ws-root").value = root;
|
||||
} catch (err) {
|
||||
$("ws-error").textContent = err.message;
|
||||
}
|
||||
}
|
||||
$("btn-ws-load").addEventListener("click", () =>
|
||||
loadWorkspace($("ws-root").value.trim()));
|
||||
$("btn-ws-last").addEventListener("click", () => {
|
||||
if (S.activeRoot) { $("ws-root").value = S.activeRoot; loadWorkspace(S.activeRoot); }
|
||||
else $("ws-error").textContent = "No last run yet.";
|
||||
});
|
||||
function renderTree(node) {
|
||||
const ul = document.createElement("ul");
|
||||
node.children.forEach((ch) => ul.appendChild(renderNode(ch)));
|
||||
const tree = $("ws-tree");
|
||||
tree.innerHTML = "";
|
||||
tree.appendChild(ul);
|
||||
}
|
||||
function renderNode(node) {
|
||||
const li = document.createElement("li");
|
||||
if (node.type === "dir") {
|
||||
const label = document.createElement("span");
|
||||
label.className = "node-dir";
|
||||
label.innerHTML = `<span class="toggle">▾</span>📁 ${esc(node.name)}`;
|
||||
li.appendChild(label);
|
||||
const childUl = document.createElement("ul");
|
||||
node.children.forEach((ch) => childUl.appendChild(renderNode(ch)));
|
||||
li.appendChild(childUl);
|
||||
label.addEventListener("click", () => {
|
||||
childUl.classList.toggle("hidden");
|
||||
label.querySelector(".toggle").textContent =
|
||||
childUl.classList.contains("hidden") ? "▸" : "▾";
|
||||
});
|
||||
} else {
|
||||
const f = document.createElement("span");
|
||||
f.className = "node-file";
|
||||
const icon = node.name.endsWith(".json") ? "🧾" :
|
||||
(node.name.endsWith(".md") ? "📝" : "📄");
|
||||
f.innerHTML = `${icon} ${esc(node.name)}`;
|
||||
f.addEventListener("click", () => showFile(node.path));
|
||||
li.appendChild(f);
|
||||
}
|
||||
return li;
|
||||
}
|
||||
function showFile(path) {
|
||||
if (!S.ws || !S.ws.files[path]) { $("ws-content").textContent = "(not available)"; return; }
|
||||
$("ws-file-path").textContent = path;
|
||||
let content = S.ws.files[path];
|
||||
const mode = $("ws-view-mode").value;
|
||||
if (path.endsWith(".json") && (mode === "auto" || mode === "json")) {
|
||||
try { content = JSON.stringify(JSON.parse(content), null, 2); }
|
||||
catch (_) { /* keep raw */ }
|
||||
}
|
||||
$("ws-content").textContent = content;
|
||||
}
|
||||
$("ws-view-mode").addEventListener("change", () => {
|
||||
if (S.ws) showFile($("ws-file-path").textContent);
|
||||
});
|
||||
|
||||
/* ---------- knowledge ---------- */
|
||||
let ktab = "skills";
|
||||
document.querySelectorAll(".ktab").forEach((b) =>
|
||||
b.addEventListener("click", () => {
|
||||
ktab = b.dataset.ktab;
|
||||
document.querySelectorAll(".ktab").forEach((x) =>
|
||||
x.classList.toggle("is-active", x === b));
|
||||
if (S.ws) renderKnowledge(S.ws.files);
|
||||
}));
|
||||
function renderKnowledge(files) {
|
||||
if (!files) return;
|
||||
$("know-emptystate").classList.add("hidden");
|
||||
const out = $("know-content");
|
||||
if (ktab === "skills") {
|
||||
const names = Object.keys(files).filter((p) => p.endsWith("/SKILL.md"));
|
||||
if (!names.length) { out.innerHTML = '<p class="muted">No evolved skills in this workspace.</p>'; return; }
|
||||
out.innerHTML = "";
|
||||
names.forEach((skillPath) => {
|
||||
const purposePath = skillPath.replace("/SKILL.md", "/PURPOSE.md");
|
||||
const card = document.createElement("div");
|
||||
card.className = "skill-card";
|
||||
const name = skillPath.split("/")[1];
|
||||
card.innerHTML = `<h4>📦 ${esc(name)}</h4><h5>SKILL.md</h5>
|
||||
<pre>${esc(files[skillPath])}</pre><h5>PURPOSE.md</h5>
|
||||
<pre>${esc(files[purposePath] || "(none)")}</pre>`;
|
||||
out.appendChild(card);
|
||||
});
|
||||
} else {
|
||||
const map = {
|
||||
patterns: (p) => p.startsWith("wiki/patterns/"),
|
||||
index: (p) => p === "wiki/index.md",
|
||||
logs: (p) => p === "wiki/logs.md",
|
||||
impact: (p) => p === "wiki/skill-impact.md",
|
||||
};
|
||||
const keep = map[ktab];
|
||||
const paths = Object.keys(files).filter(keep).sort();
|
||||
if (!paths.length) { out.innerHTML = '<p class="muted">Nothing here yet.</p>'; return; }
|
||||
out.innerHTML = "";
|
||||
paths.forEach((p) => {
|
||||
const box = document.createElement("div");
|
||||
box.className = "kdoc";
|
||||
box.innerHTML = `<b>${esc(p)}</b>\n\n${esc(files[p])}`;
|
||||
out.appendChild(box);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/* ---------- restore state on load ---------- */
|
||||
(async function init() {
|
||||
try {
|
||||
const st = await api("/api/status");
|
||||
const state = st.state;
|
||||
if (state) {
|
||||
(state.events || []).forEach(renderEvent);
|
||||
if (state.result) {
|
||||
S.lastResult = state.result;
|
||||
showDashboard(state.result.evolution);
|
||||
showEval(state.result.evaluation);
|
||||
S.activeRoot = state.result.workspace_root;
|
||||
$("log-root-label").textContent = `→ ${S.activeRoot}`;
|
||||
$("eval-root").value = S.activeRoot || "";
|
||||
}
|
||||
if (state.status === "done") setStatus("done", "done");
|
||||
if (st.active) connectStream();
|
||||
}
|
||||
} catch (_) { /* server unreachable; keep idle state */ }
|
||||
})();
|
||||
$("form-error").textContent = err.message;
|
||||
}
|
||||
});
|
||||
$("btn-cancel").addEventListener("click", async () => {
|
||||
try { await api("/api/cancel", { body: {} }); logLine("—— cancel requested ——", "ln-cmd"); }
|
||||
catch (_) { /* ignore */ }
|
||||
});
|
||||
@@ -0,0 +1,141 @@
|
||||
:root {
|
||||
--bg: #f7f7f5; --panel: #ffffff; --ink: #1a1a1a; --muted: #6b6b6b;
|
||||
--accent: #3b5bdb; --accent-soft: #e7ebfc; --ok: #099268; --ok-soft: #e6f7f1;
|
||||
--bad: #e03131; --bad-soft: #fdeaea; --warn: #f59f00; --border: #e3e1dc;
|
||||
--mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
||||
--sans: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html, body { margin: 0; }
|
||||
body { background: var(--bg); color: var(--ink); font-family: var(--sans); font-size: 14px; line-height: 1.5; }
|
||||
.hidden { display: none !important; }
|
||||
.muted { color: var(--muted); font-weight: 400; }
|
||||
.small { font-size: 12px; }
|
||||
a { color: var(--accent); }
|
||||
|
||||
/* ---------- header / tabs ---------- */
|
||||
.topbar { display: flex; justify-content: space-between; align-items: center;
|
||||
padding: 14px 22px; background: #fff; border-bottom: 1px solid var(--border); }
|
||||
.brand-kicker { font-size: 11px; letter-spacing: .08em; text-transform: uppercase; color: var(--muted); display: block; }
|
||||
.brand h1 { margin: 0; font-size: 20px; }
|
||||
.brand .sub { color: var(--muted); font-weight: 400; font-size: 14px; }
|
||||
.topbar-right { display: flex; align-items: center; gap: 12px; }
|
||||
.small-link { font-size: 12px; }
|
||||
|
||||
.pill { border-radius: 999px; padding: 3px 12px; font-size: 12px; font-weight: 600; }
|
||||
.pill-idle { background: #eee; color: #555; }
|
||||
.pill-running { background: var(--accent-soft); color: var(--accent); }
|
||||
.pill-evaluating { background: #f3ecfb; color: #7b2cbf; }
|
||||
.pill-done { background: var(--ok-soft); color: var(--ok); }
|
||||
.pill-error { background: var(--bad-soft); color: var(--bad); }
|
||||
|
||||
.tabs { display: flex; gap: 4px; padding: 10px 22px 0; background: #fff;
|
||||
border-bottom: 1px solid var(--border); }
|
||||
.tab { border: 0; background: transparent; padding: 8px 14px; font-size: 13px;
|
||||
font-weight: 600; color: #555; cursor: pointer; border-radius: 8px 8px 0 0; }
|
||||
.tab:hover { background: #f2f2f0; }
|
||||
.tab.is-active { color: var(--accent); background: var(--accent-soft); }
|
||||
|
||||
/* ---------- layout ---------- */
|
||||
main { max-width: 1180px; margin: 18px auto; padding: 0 18px; }
|
||||
.tab-panel { display: none; }
|
||||
.tab-panel.is-active { display: block; }
|
||||
.grid-2 { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; align-items: start; }
|
||||
@media (max-width: 860px) { .grid-2 { grid-template-columns: 1fr; } }
|
||||
.grid-2-inner { display: grid; grid-template-columns: 1fr 1fr; gap: 10px; }
|
||||
|
||||
.card { background: var(--panel); border: 1px solid var(--border); border-radius: 12px;
|
||||
padding: 18px 20px; margin-bottom: 16px; }
|
||||
.card h2 { margin: 0 0 10px; font-size: 16px; }
|
||||
.card h3 { font-size: 14px; margin: 14px 0 6px; }
|
||||
|
||||
/* ---------- forms ---------- */
|
||||
form label { display: block; font-weight: 600; font-size: 13px; margin: 10px 0 4px; }
|
||||
form input, form select { width: 100%; padding: 7px 9px; font-size: 13px;
|
||||
border: 1px solid var(--border); border-radius: 8px; background: #fff; }
|
||||
.actions { display: flex; gap: 10px; margin-top: 14px; }
|
||||
.btn { background: #fff; color: var(--ink); border: 1px solid var(--border);
|
||||
border-radius: 8px; padding: 7px 14px; font-size: 13px; font-weight: 600; cursor: pointer; }
|
||||
.btn:hover { border-color: var(--accent); color: var(--accent); }
|
||||
.btn-primary { background: var(--accent); color: #fff; border-color: var(--accent); }
|
||||
.btn-primary:hover { opacity: .9; color: #fff; }
|
||||
.btn-danger { background: var(--bad); color: #fff; border-color: var(--bad); }
|
||||
.btn-danger:hover { opacity: .9; color: #fff; }
|
||||
.form-error { color: var(--bad); font-size: 12px; min-height: 16px; margin: 8px 0 0; }
|
||||
.load-row { display: flex; gap: 10px; align-items: end; margin-bottom: 12px; }
|
||||
.load-row label { flex: 1; display: block; font-size: 12px; font-weight: 600; }
|
||||
.load-row input { width: 100%; padding: 6px 8px; border: 1px solid var(--border); border-radius: 8px; }
|
||||
/* ---------- console ---------- */
|
||||
.console { background: #15181d; color: #d7e0f0; font-family: var(--mono);
|
||||
font-size: 12px; padding: 12px; border-radius: 10px; height: 460px; overflow: auto; white-space: pre-wrap; }
|
||||
.console .ln { display: block; }
|
||||
.console .ln-tail { color: #8a93a6; }
|
||||
.console .ln-cmd { color: #7cc4ff; }
|
||||
.console .ln-react { color: #f2c55c; }
|
||||
.console .ln-accept { color: #5ee6a7; }
|
||||
.console .ln-reject { color: #ff8d8d; }
|
||||
|
||||
/* ---------- charts ---------- */
|
||||
.chart { width: 100%; height: auto; background: #fff; border: 1px solid var(--border); border-radius: 10px; }
|
||||
.chart text { font-size: 10px; fill: #666; }
|
||||
|
||||
/* ---------- tables ---------- */
|
||||
.table-wrap { overflow: auto; }
|
||||
table { border-collapse: collapse; width: 100%; font-size: 13px; }
|
||||
th, td { text-align: left; padding: 6px 10px; border-bottom: 1px solid var(--border); }
|
||||
th { font-size: 12px; color: var(--muted); }
|
||||
.badge { border-radius: 999px; padding: 2px 10px; font-size: 11px; font-weight: 700; }
|
||||
.badge.accepted { background: var(--ok-soft); color: var(--ok); }
|
||||
.badge.rejected { background: var(--bad-soft); color: var(--bad); }
|
||||
.badge.none { background: #eee; color: #555; }
|
||||
|
||||
/* ---------- KPI / verdict ---------- */
|
||||
.kpi-row { display: flex; gap: 14px; margin: 12px 0; flex-wrap: wrap; }
|
||||
.kpi { flex: 1; min-width: 130px; background: #fff; border: 1px solid var(--border);
|
||||
border-radius: 12px; padding: 14px 16px; text-align: center; }
|
||||
.kpi .v { font-size: 26px; font-weight: 700; }
|
||||
.kpi .l { font-size: 11px; color: var(--muted); text-transform: uppercase; letter-spacing: .04em; }
|
||||
.verdict { font-weight: 600; padding: 10px 14px; border-radius: 10px; display: inline-block; }
|
||||
.verdict.ok { background: var(--ok-soft); color: var(--ok); }
|
||||
.verdict.bad { background: var(--bad-soft); color: var(--bad); }
|
||||
.verdict.neutral { background: #f1f1ee; color: #555; }
|
||||
|
||||
/* ---------- workspace tree / viewer ---------- */
|
||||
.ws-split { display: grid; grid-template-columns: 300px 1fr; gap: 14px; align-items: start; }
|
||||
@media (max-width: 860px) { .ws-split { grid-template-columns: 1fr; } }
|
||||
.tree { border: 1px solid var(--border); border-radius: 10px; padding: 8px; max-height: 560px; overflow: auto; font-size: 13px; }
|
||||
.tree ul { list-style: none; margin: 0; padding-left: 16px; }
|
||||
.tree .node-file { cursor: pointer; padding: 1px 6px; border-radius: 6px; white-space: nowrap; }
|
||||
.tree .node-file:hover { background: var(--accent-soft); }
|
||||
.tree .node-dir { font-weight: 600; }
|
||||
.tree .toggle { cursor: pointer; color: var(--muted); margin-right: 2px; }
|
||||
.viewer { border: 1px solid var(--border); border-radius: 10px; overflow: hidden; }
|
||||
.viewer-head { display: flex; justify-content: space-between; gap: 8px; padding: 8px 10px;
|
||||
/* ---------- knowledge ---------- */
|
||||
.ktabs { display: flex; gap: 4px; margin-bottom: 12px; flex-wrap: wrap; }
|
||||
.ktab { border: 1px solid var(--border); background: #fff; border-radius: 8px;
|
||||
padding: 5px 12px; font-size: 12px; font-weight: 600; color: #555; cursor: pointer; }
|
||||
.ktab.is-active { background: var(--accent); border-color: var(--accent); color: #fff; }
|
||||
.kdoc { background: #fdfdfb; border: 1px solid var(--border); border-radius: 10px;
|
||||
padding: 14px 16px; font-family: var(--mono); font-size: 12px; white-space: pre-wrap; max-height: 480px; overflow: auto; }
|
||||
.skill-card { border: 1px solid var(--border); border-radius: 10px; padding: 12px 14px; margin-bottom: 12px; }
|
||||
.skill-card h4 { margin: 0 0 6px; color: var(--accent); }
|
||||
.skill-card h5 { margin: 8px 0 4px; font-size: 12px; color: var(--muted); text-transform: uppercase; letter-spacing: .04em; }
|
||||
.skill-card pre { margin: 0; font-family: var(--mono); font-size: 12px; white-space: pre-wrap;
|
||||
background: #f8f8f6; border-radius: 8px; padding: 8px 10px; max-height: 220px; overflow: auto; }
|
||||
|
||||
/* ---------- how-it-works ---------- */
|
||||
.pipeline { display: flex; flex-wrap: wrap; gap: 8px; align-items: stretch; margin: 12px 0; }
|
||||
.layer { flex: 1; min-width: 160px; background: var(--accent-soft); border: 1px solid #cfd7f8;
|
||||
border-radius: 10px; padding: 10px 12px; font-size: 13px; }
|
||||
.layer span { display: block; font-weight: 400; font-size: 12px; color: var(--muted); margin-top: 2px; }
|
||||
.arrow { align-self: center; font-size: 20px; color: var(--accent); }
|
||||
.alg { background: #15181d; color: #cfe3d8; font-family: var(--mono); font-size: 12px;
|
||||
border-radius: 10px; padding: 14px 16px; overflow: auto; }
|
||||
.glossary td:first-child { font-family: var(--mono); font-weight: 700; }
|
||||
|
||||
.foot { text-align: center; padding: 18px; color: var(--muted); font-size: 12px; }
|
||||
background: #fafaf8; border-bottom: 1px solid var(--border); align-items: center; }
|
||||
.viewer-head select { font-size: 12px; }
|
||||
.viewer-content { margin: 0; padding: 12px; font-family: var(--mono); font-size: 12px;
|
||||
max-height: 560px; overflow: auto; white-space: pre-wrap; word-break: break-word; }
|
||||
@@ -0,0 +1,202 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>WikiSkill — Skill Evolution Workbench</title>
|
||||
<link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}">
|
||||
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'><text y='13' font-size='13'>📚</text></svg>">
|
||||
</head>
|
||||
<body>
|
||||
<header class="topbar">
|
||||
<div class="brand">
|
||||
<span class="brand-kicker">arXiv:2608.27454 · Google Research</span>
|
||||
<h1>WikiSkill <span class="sub">Skill Evolution Workbench</span></h1>
|
||||
</div>
|
||||
<div class="topbar-right">
|
||||
<span id="status-pill" class="pill pill-idle">idle</span>
|
||||
<a class="small-link" href="https://arxiv.org/abs/2608.27454" target="_blank" rel="noopener">Paper ↗</a>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<nav class="tabs" role="tablist" aria-label="Sections">
|
||||
<button class="tab is-active" data-tab="run" role="tab">▶ Run</button>
|
||||
<button class="tab" data-tab="dashboard" role="tab">Dashboard</button>
|
||||
<button class="tab" data-tab="eval" role="tab">Evaluation</button>
|
||||
<button class="tab" data-tab="workspace" role="tab">Workspace</button>
|
||||
<button class="tab" data-tab="knowledge" role="tab">Knowledge</button>
|
||||
<button class="tab" data-tab="how" role="tab">How it works</button>
|
||||
</nav>
|
||||
|
||||
<main>
|
||||
<!-- ============================ RUN ============================ -->
|
||||
<section id="tab-run" class="tab-panel is-active">
|
||||
<div class="grid-2">
|
||||
<div class="card">
|
||||
<h2>Configuration</h2>
|
||||
<form id="run-form" autocomplete="off">
|
||||
<label>LLM backend
|
||||
<select id="cfg-backend">
|
||||
<option value="mock">Mock (offline, deterministic)</option>
|
||||
<option value="openai">OpenAI-compatible endpoint</option>
|
||||
</select>
|
||||
</label>
|
||||
<div id="openai-fields" class="hidden">
|
||||
<label>Model<input id="cfg-model" placeholder="e.g. qwen3-27b-instruct"></label>
|
||||
<label>Base URL<input id="cfg-base-url" placeholder="https://api.openai.com/v1"></label>
|
||||
<label>API key<input id="cfg-api-key" type="password" placeholder="sk-... (or env WIKISKILL_API_KEY)"></label>
|
||||
<label>Temperature<input id="cfg-temperature" type="number" step="0.1" min="0" max="2" value="0.7"></label>
|
||||
</div>
|
||||
<div class="grid-2-inner">
|
||||
<label>Max iterations (K)<input id="cfg-iterations" type="number" min="1" max="50" value="8"></label>
|
||||
<label>Max ReAct turns<input id="cfg-react" type="number" min="1" max="40" value="20"></label>
|
||||
</div>
|
||||
<label>Workspace root (blank = auto <code>runs/run_<ts></code>)
|
||||
<input id="cfg-root" placeholder="runs/my-experiment"></label>
|
||||
<div class="actions">
|
||||
<button id="btn-start" type="submit" class="btn btn-primary">Start evolution</button>
|
||||
<button id="btn-cancel" type="button" class="btn btn-danger hidden">Cancel</button>
|
||||
<!-- ========================= DASHBOARD ========================= -->
|
||||
<section id="tab-dashboard" class="tab-panel">
|
||||
<div class="card">
|
||||
<h2>Evolution overview</h2>
|
||||
<p class="muted" id="dash-emptystate">No run yet — start one on the <b>Run</b> tab.</p>
|
||||
<div class="grid-2 hidden" id="dash-charts">
|
||||
<div><h3>R_best over iterations</h3><svg id="chart-rbest" viewBox="0 0 400 180" class="chart"></svg></div>
|
||||
<div><h3>Validation gating (proposals)</h3><svg id="chart-gate" viewBox="0 0 400 180" class="chart"></svg></div>
|
||||
</div>
|
||||
<h3>History</h3>
|
||||
<div class="table-wrap">
|
||||
<table id="dash-table">
|
||||
<thead><tr><th>Iter</th><th>Proposal</th><th>Action</th><th>Val score</th><th>Outcome</th></tr></thead>
|
||||
<tbody></tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ========================= EVAL ========================= -->
|
||||
<section id="tab-eval" class="tab-panel">
|
||||
<div class="card">
|
||||
<h2>Test-set evaluation <span class="muted">— final skills vs no-skill baseline (paper §4)</span></h2>
|
||||
<p class="muted" id="eval-emptystate">Evaluation is computed automatically at the end
|
||||
of every run. You can also re-run it for the current workspace below.</p>
|
||||
<div id="eval-cards" class="kpi-row hidden"></div>
|
||||
<div id="eval-bars" class="hidden"></div>
|
||||
<p id="eval-verdict" class="verdict hidden"></p>
|
||||
<div class="actions">
|
||||
<button id="btn-evaluate" class="btn btn-primary">Re-evaluate current workspace</button>
|
||||
</div>
|
||||
<div class="load-row hidden" id="eval-load-form">
|
||||
<label>Workspace root
|
||||
<input id="eval-root" placeholder="runs/run_20260830_120000"></label>
|
||||
<button id="btn-eval-load" class="btn">Load result</button>
|
||||
</div>
|
||||
<p id="eval-error" class="form-error" role="alert"></p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ========================= WORKSPACE ========================= -->
|
||||
<section id="tab-workspace" class="tab-panel">
|
||||
<div class="card">
|
||||
<h2>Workspace explorer</h2>
|
||||
<div class="load-row">
|
||||
<label>Root
|
||||
<input id="ws-root" placeholder="runs/run_20260830_120000"></label>
|
||||
<button id="btn-ws-load" class="btn">Browse</button>
|
||||
<button id="btn-ws-last" class="btn">Last run</button>
|
||||
</div>
|
||||
<p id="ws-error" class="form-error"></p>
|
||||
<div class="ws-split hidden" id="ws-split">
|
||||
<!-- ========================= KNOWLEDGE ========================= -->
|
||||
<section id="tab-knowledge" class="tab-panel">
|
||||
<div class="card">
|
||||
<h2>Knowledge & skills</h2>
|
||||
<div class="ktabs">
|
||||
<button class="ktab is-active" data-ktab="skills">Skills</button>
|
||||
<button class="ktab" data-ktab="patterns">Wiki patterns</button>
|
||||
<button class="ktab" data-ktab="index">Index</button>
|
||||
<button class="ktab" data-ktab="logs">Evolution log</button>
|
||||
<button class="ktab" data-ktab="impact">Skill impact</button>
|
||||
</div>
|
||||
<p class="muted" id="know-emptystate">Load a workspace in the Workspace tab to inspect its knowledge and skills.</p>
|
||||
<div id="know-content"></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ========================= HOW ========================= -->
|
||||
<section id="tab-how" class="tab-panel">
|
||||
<div class="card">
|
||||
<h2>How WikiSkill works</h2>
|
||||
<p><b>WikiSkill</b> co-evolves agent <i>skills</i> with a persistent <i>knowledge base
|
||||
(wiki)</i>. Each evolution iteration runs the loop below; the wiki accumulates
|
||||
across all iterations and is never rolled back.</p>
|
||||
<div class="pipeline">
|
||||
<div class="layer"><b>Raw layer <span>immutable traces τ ~ π(x; S)</span></b></div>
|
||||
<div class="arrow">→</div>
|
||||
<div class="layer"><b>Wiki Maintainer <span>root-cause analysis → patterns</span></b></div>
|
||||
<div class="arrow">→</div>
|
||||
<div class="layer"><b>Skill Proposer <span>ReAct over wiki + traces → P</span></b></div>
|
||||
<div class="arrow">→</div>
|
||||
<div class="layer"><b>Gating <span>accept iff R(T_val) > R_best</span></b></div>
|
||||
</div>
|
||||
<h3>Algorithm 1 (per iteration k)</h3>
|
||||
<pre class="alg">S₀ = ∅, W₀ = ∅; R_best ← R_baseline(D_val, S₀)
|
||||
for k = 1..K:
|
||||
if R_best == 1.0: break
|
||||
T_train,k ← rollouts of π with S_{k-1} # skills injected, wiki hidden
|
||||
T_sample,k ⊂ T_train,k # ≤5 fails + ≤3 passes (App. C)
|
||||
W'_k ← M_WM(W_{k-1}, T_sample,k) # wiki maintenance (patch edits)
|
||||
P_k ← M_P(W'_k, S_{k-1}, T_train,k) # ReAct proposer
|
||||
S'_k ← Apply(S_{k-1}, P_k)
|
||||
if R(T_val,k) > R_best: accept; S_k ← S'_k; R_best ← R(T_val,k)
|
||||
else: roll back skills; S_k ← S_{k-1}
|
||||
W_k ← Update(W'_k, P_k, R(T_val,k), a_k) # harness writes skill-impact.md
|
||||
return S_K, W_K</pre>
|
||||
<h3>Glossary</h3>
|
||||
<table class="glossary">
|
||||
<tr><td>S_k</td><td>active skill set (skills/)</td></tr>
|
||||
<tr><td>W_k</td><td>persistent wiki (patterns, index, logs, skill-impact)</td></tr>
|
||||
<tr><td>P_k</td><td>single atomic skill proposal (create or patch)</td></tr>
|
||||
<tr><td>M_WM</td><td>Wiki Maintainer agent (consolidates traces into patterns)</td></tr>
|
||||
<tr><td>M_P</td><td>Wiki-informed Skill Proposer agent (multi-turn ReAct)</td></tr>
|
||||
<tr><td>R_best</td><td>best validation score; gating threshold (strict >)</td></tr>
|
||||
<tr><td>R(·)</td><td>accuracy metric on a trace set</td></tr>
|
||||
</table>
|
||||
<p class="muted small">Full detail in <code>design.md</code>, <code>prd.md</code>,
|
||||
<code>ui_design.md</code>, and the paper itself.</p>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
<footer class="foot"><span>Independent reimplementation for study · <a href="https://arxiv.org/abs/2608.27454" target="_blank" rel="noopener">Tang et al., Google Research (CC BY 4.0)</a></span></footer>
|
||||
|
||||
<script src="{{ url_for('static', filename='app.js') }}"></script>
|
||||
</body>
|
||||
</html>
|
||||
<div id="ws-tree" class="tree" role="tree"></div>
|
||||
<div class="viewer">
|
||||
<div class="viewer-head"><span id="ws-file-path" class="muted"></span>
|
||||
<select id="ws-view-mode">
|
||||
<option value="auto">auto</option><option value="json">JSON</option>
|
||||
<option value="raw">Raw</option>
|
||||
</select>
|
||||
</div>
|
||||
<pre id="ws-content" class="viewer-content"></pre>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
<p id="form-error" class="form-error" role="alert"></p>
|
||||
</form>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h2>Live console <span class="muted" id="log-root-label"></span></h2>
|
||||
<div id="console" class="console" aria-live="polite"></div>
|
||||
<p class="muted small">Logs stream live; writer events (Wiki Maintainer / Skill
|
||||
Proposer) and validation gating decisions appear as they happen.</p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
@@ -73,8 +73,14 @@ class Workspace:
|
||||
f.write(content)
|
||||
|
||||
# ---------- Raw layer: immutable ----------
|
||||
def add_raw_trace(self, iteration: int, task_id: str, trace: dict) -> str:
|
||||
rel = os.path.join("raw", f"iter_{iteration:03d}", f"{task_id}.json")
|
||||
def add_raw_trace(self, iteration, task_id: str, trace: dict) -> str:
|
||||
"""Write an immutable raw-layer trace under raw/<subdir>/.
|
||||
|
||||
`iteration` may be an int (formatted `iter_NNN`) or any string tag (e.g.
|
||||
`eval_skilled` for post-hoc test evaluations). Existing files are never
|
||||
overwritten — the raw layer preserves the complete execution history."""
|
||||
subdir = f"iter_{iteration:03d}" if isinstance(iteration, int) else str(iteration)
|
||||
rel = os.path.join("raw", subdir, f"{task_id}.json")
|
||||
p = self._abs(rel)
|
||||
if os.path.exists(p):
|
||||
raise FileExistsError(f"raw layer is immutable: {rel}")
|
||||
|
||||
@@ -32,6 +32,22 @@ def test_skills_create_snapshot_restore(tmp_path):
|
||||
assert skills.list_skills() == ["s1"]
|
||||
|
||||
|
||||
def test_skills_restore_reverts_purpose_md(tmp_path):
|
||||
"""A rejected patch must roll back BOTH SKILL.md and PURPOSE.md exactly
|
||||
to S_{k-1} (skills-only rollback, §3.2.4)."""
|
||||
skills = SkillsLayer(_ws(tmp_path))
|
||||
skills.apply_proposal({"action": "create", "skill": "s",
|
||||
"content": "body", "purpose": "v1"})
|
||||
snap = skills.snapshot()
|
||||
skills.apply_proposal({"action": "patch", "skill": "s",
|
||||
"edits": [{"op": "append", "content": "extra"}],
|
||||
"purpose": "v2 (append)"})
|
||||
assert "v2" in skills.purpose("s")
|
||||
skills.restore(snap)
|
||||
assert skills.purpose("s") == "v1\n"
|
||||
assert skills.skill_content("s") == "body"
|
||||
|
||||
|
||||
def test_skill_injection_context(tmp_path):
|
||||
skills = SkillsLayer(_ws(tmp_path))
|
||||
assert "(No active skills.)" in skills.active_skills_context()
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
import os
|
||||
import time
|
||||
import types
|
||||
|
||||
import pytest
|
||||
|
||||
from wikiskill import __version__
|
||||
from wikiskill.webapp import create_app
|
||||
from wikiskill.webapp.api import get_manager
|
||||
from wikiskill.webapp.runner import RunManager, _is_fresh_root
|
||||
|
||||
|
||||
def _wait_status(client, timeout=20.0):
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
r = client.get("/api/status")
|
||||
assert r.status_code == 200
|
||||
st = r.get_json()
|
||||
if st["state"] and st["state"]["status"] in ("done", "error"):
|
||||
return st
|
||||
time.sleep(0.05)
|
||||
raise AssertionError("run did not finish in time")
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path) # relative workspace roots land under tmp_path
|
||||
app = create_app({"TESTING": True})
|
||||
get_manager()._reset_for_tests()
|
||||
with app.test_client() as c:
|
||||
yield c
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- basics
|
||||
def test_index_serves_spa(client):
|
||||
r = client.get("/")
|
||||
assert r.status_code == 200
|
||||
html = r.get_data(as_text=True)
|
||||
assert "WikiSkill" in html
|
||||
assert 'id="run-form"' in html
|
||||
assert 'id="console"' in html
|
||||
assert 'id="chart-rbest"' in html
|
||||
|
||||
|
||||
def test_health(client):
|
||||
r = client.get("/api/health")
|
||||
# ---------------------------------------------------------------- lifecycle
|
||||
def test_full_run_lifecycle(client, tmp_path):
|
||||
root = "ui_test_run"
|
||||
r = client.post("/api/run", json={"llm_backend": "mock", "iterations": 3,
|
||||
"workspace_root": root})
|
||||
assert r.status_code == 202
|
||||
assert r.get_json()["active"] is True
|
||||
|
||||
# single-flight: second start while active must be rejected
|
||||
r2 = client.post("/api/run", json={"llm_backend": "mock", "iterations": 3,
|
||||
"workspace_root": root + "_2"})
|
||||
assert r2.status_code == 409
|
||||
|
||||
st = _wait_status(client)
|
||||
assert st["state"]["status"] == "done"
|
||||
result = st["state"]["result"]
|
||||
assert "unit-conversion" in result["evolution"]["final_skills"]
|
||||
assert result["evaluation"]["significant"] is True
|
||||
|
||||
# workspace artifacts exist on disk
|
||||
assert (tmp_path / root / "wiki" / "index.md").is_file()
|
||||
assert (tmp_path / root / "skills" / "unit-conversion" / "SKILL.md").is_file()
|
||||
# string-tag subdirs for post-hoc test evaluation
|
||||
assert (tmp_path / root / "raw" / "eval_skilled").is_dir()
|
||||
|
||||
|
||||
def test_status_empty(client):
|
||||
st = client.get("/api/status").get_json()
|
||||
assert st["active"] is False
|
||||
assert st["state"] is None
|
||||
# ---------------------------------------------------------------- workspace & eval
|
||||
def test_workspace_endpoint(client, tmp_path):
|
||||
root = "ui_ws_test"
|
||||
client.post("/api/run", json={"llm_backend": "mock", "iterations": 2,
|
||||
"workspace_root": root})
|
||||
_wait_status(client)
|
||||
|
||||
r = client.get(f"/api/workspace?root={root}")
|
||||
assert r.status_code == 200
|
||||
data = r.get_json()
|
||||
assert data["root"] == root
|
||||
names = {c["name"] for c in data["tree"]["children"]}
|
||||
assert {"raw", "wiki", "skills"} <= names
|
||||
assert "wiki/index.md" in data["files"]
|
||||
assert "wiki/skill-impact.md" in data["files"]
|
||||
assert any(k.endswith("SKILL.md") for k in data["files"])
|
||||
|
||||
assert client.get("/api/workspace?root=ui_does_not_exist").status_code == 404
|
||||
assert client.get("/api/workspace?root=../x").status_code == 400
|
||||
|
||||
|
||||
def test_evaluate_endpoint(client, tmp_path):
|
||||
root = "ui_eval_test"
|
||||
client.post("/api/run", json={"llm_backend": "mock", "iterations": 2,
|
||||
"workspace_root": root})
|
||||
_wait_status(client)
|
||||
|
||||
r = client.post("/api/evaluate", json={"llm_backend": "mock",
|
||||
"workspace_root": root})
|
||||
assert r.status_code == 200
|
||||
ev = r.get_json()["evaluation"]
|
||||
assert ev["skilled_accuracy"] == 1.0
|
||||
assert ev["baseline_accuracy"] == 0.0
|
||||
assert ev["significant"] is True
|
||||
assert client.post("/api/evaluate", json={"llm_backend": "mock",
|
||||
"workspace_root": "missing"}).status_code == 404
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- SSE
|
||||
def test_sse_stream(client):
|
||||
r = client.post("/api/run", json={"llm_backend": "mock", "iterations": 1,
|
||||
"workspace_root": "ui_sse_test"})
|
||||
assert r.status_code == 202
|
||||
stream = client.get("/api/stream")
|
||||
body = b""
|
||||
for chunk in stream.response:
|
||||
body += chunk
|
||||
if b'"type": "done"' in body:
|
||||
break
|
||||
assert b"data:" in body
|
||||
assert b'"done"' in body
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- RunManager units
|
||||
def test_runmanager_double_start_rejected():
|
||||
mgr = RunManager()
|
||||
mgr._reset_for_tests()
|
||||
mgr._execute = types.MethodType(lambda self, rid, cfg: time.sleep(2), mgr)
|
||||
a = mgr.start({"llm_backend": "mock"})
|
||||
assert a["active"] is True
|
||||
with pytest.raises(PermissionError):
|
||||
mgr.start({"llm_backend": "mock"})
|
||||
mgr.cancel()
|
||||
mgr._reset_for_tests()
|
||||
|
||||
|
||||
def test_runmanager_fresh_root_guard(tmp_path):
|
||||
assert _is_fresh_root(str(tmp_path / "new")) is True
|
||||
raw = tmp_path / "used" / "raw"
|
||||
raw.mkdir(parents=True)
|
||||
(raw / "iter_000").mkdir()
|
||||
assert _is_fresh_root(str(tmp_path / "used")) is False
|
||||
|
||||
|
||||
def test_cancel(client, tmp_path, monkeypatch):
|
||||
# Slow the orchestrator so cancel lands mid-run (cooperative stop).
|
||||
import wikiskill.orchestrator as orch_mod
|
||||
original_run = orch_mod.EvolutionOrchestrator.run
|
||||
|
||||
def slow_run(self):
|
||||
time.sleep(1.5)
|
||||
return original_run(self)
|
||||
|
||||
monkeypatch.setattr(orch_mod.EvolutionOrchestrator, "run", slow_run)
|
||||
client.post("/api/run", json={"llm_backend": "mock", "iterations": 4,
|
||||
"workspace_root": "ui_cancel_run"})
|
||||
assert client.post("/api/cancel", json={}).status_code == 200
|
||||
st = _wait_status(client, timeout=20)
|
||||
assert st["state"]["status"] in ("done", "error")
|
||||
|
||||
|
||||
def test_run_validation_400s(client):
|
||||
assert client.post("/api/run", json={"llm_backend": "bogus"}).status_code == 400
|
||||
assert client.post("/api/run", json={"llm_backend": "openai"}).status_code == 400 # no model
|
||||
assert client.post("/api/run", json={"llm_backend": "mock",
|
||||
"workspace_root": "../escape"}).status_code == 400
|
||||
assert client.post("/api/run", json={"llm_backend": "mock",
|
||||
"workspace_root": "C:\\abs"}).status_code == 400
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
# WikiSkill — Web UI Design Document
|
||||
|
||||
Companion to `design.md`. Describes the interactive web application that lets users
|
||||
run the WikiSkill evolution loop, watch it live, inspect every produced artifact, and
|
||||
evaluate the evolved skills — a visual, testable harness for the paper's Algorithm 1.
|
||||
|
||||
## 1. Goals
|
||||
|
||||
- **Run**: configure a WikiSkill evolution run (dataset, iterations, LLM backend) and
|
||||
start/stop it from the browser.
|
||||
- **Watch**: stream the evolution live (logs, ReAct proposer steps, gating decisions).
|
||||
- **Inspect**: browse the complete three-layer workspace (`raw/`, `wiki/`, `skills/`)
|
||||
produced by the run.
|
||||
- **Evaluate**: compare final evolved skills vs. no-skill baseline on the test split
|
||||
with paired-bootstrap significance.
|
||||
- **Understand**: an "How it works" view that maps the UI to the paper's architecture
|
||||
(Figure 2 / Algorithm 1).
|
||||
|
||||
Non-goals: skill editing, wiki pruning, multi-user auth. This is a local study tool.
|
||||
|
||||
## 2. User workflows
|
||||
|
||||
1. **Explore** — open `http://127.0.0.1:5000`, read How-it-works, browse an existing
|
||||
workspace from disk.
|
||||
2. **Run** — Configure → Start → watch live events → switch to Dashboard/Workspace.
|
||||
3. **Analyze** — inspect the evolution curve, accepted/rejected proposals, wiki
|
||||
patterns, raw traces, then Evaluate on the test split.
|
||||
4. **Repeat** — tweak parameters (iterations, ReAct turns), start a new run in a fresh
|
||||
workspace, compare.
|
||||
|
||||
## 3. Information architecture (single page, six tabs)
|
||||
|
||||
| Tab | Purpose |
|
||||
|-----|---------|
|
||||
| **Run** | Configuration form + live console + status pill; start/cancel |
|
||||
| **Dashboard** | R_best curve, per-iteration acceptance bars, history table |
|
||||
| **Evaluation** | Test-split comparison (skilled vs baseline), bootstrap p-value |
|
||||
| **Workspace** | File tree + viewer for raw/wiki/skills/run_state.json |
|
||||
| **Knowledge** | Wiki index, pattern pages, evolution log, skill-impact tracker, skills |
|
||||
| **How it works** | Layer diagram + Algorithm 1 pseudocode + glossary |
|
||||
|
||||
## 4. Screen details
|
||||
|
||||
### 4.1 Run tab
|
||||
- **Config form**: LLM backend (`mock`|`openai`), model, base URL, API key (password
|
||||
field), temperature, max iterations (K), max ReAct turns, workspace root (empty =
|
||||
auto `runs/run_<timestamp>`).
|
||||
### 4.3 Evaluation tab
|
||||
- Big-number cards: Skilled accuracy, Baseline accuracy, Δ, bootstrap p-value.
|
||||
- Horizontal bar comparison + verdict ("Skilled is significantly better", etc.).
|
||||
- Re-evaluate button (re-runs `evaluate_test()` for the current workspace).
|
||||
|
||||
### 4.4 Workspace tab
|
||||
- Left: collapsible file tree starting at the workspace root.
|
||||
- Right: content viewer — JSON pretty-printed, markdown shown in raw + rendered form,
|
||||
diffs monospace.
|
||||
|
||||
### 4.5 Knowledge tab
|
||||
- Sub-tabs: Skills (cards with SKILL.md + PURPOSE.md), Wiki patterns (list + pages),
|
||||
Evolution log, Skill-impact history.
|
||||
|
||||
### 4.6 How-it-works tab
|
||||
- Static annotated pipeline of the three layers and four components, Algorithm 1
|
||||
pseudocode, and definitions table (S_k, W_k, P_k, T_train, R_best…).
|
||||
|
||||
## 5. Design system
|
||||
|
||||
- Dark-on-light ("paper" feel): background `#f7f7f5`, panels white `#fff`, ink `#1a1a1a`.
|
||||
- Accent: indigo `#3b5bdb` (brand), success `#099268`, danger `#e03131`, warn `#f59f00`.
|
||||
- Type: system stack; headings 600, mono stack for code/traces.
|
||||
- Badges/pills for Accept/Reject and status; cards with 1px border, 10px radius.
|
||||
- Charts rendered as inline SVG (zero JS chart libraries → works offline).
|
||||
|
||||
## 6. Backend contract
|
||||
|
||||
| Method/Path | Body/Query | Returns |
|
||||
|---|---|---|
|
||||
| `GET /` | — | SPA HTML |
|
||||
| `GET /api/health` | — | `{ok, version}` |
|
||||
| `POST /api/run` | config JSON | `202 {run_id}`; `400` validation; `409` busy |
|
||||
| `GET /api/status` | — | `{active, state:{status, events[], result, error}}` |
|
||||
| `GET /api/stream` | — | Server-Sent Events (`log`/`status`/`react`/`done`/`error`) |
|
||||
| `POST /api/cancel` | — | `200 {cancelled}` |
|
||||
| `GET /api/workspace?root=` | query | `{root, tree, files}` (text files) |
|
||||
| `POST /api/evaluate` | `{workspace_root, llm_backend, …}` | evaluation report JSON |
|
||||
|
||||
Validation rules: workspace_root must be a *relative* path (no abs, no `..`); run
|
||||
rejected with 409 if another run is active; run rejected if root already contains
|
||||
evolution history (choose a fresh root to avoid clobbering the immutable raw layer).
|
||||
|
||||
### SSE protocol
|
||||
Events: `{type:"log", message}`, `{type:"react", turn, kind, path?, found?}`,
|
||||
`{type:"status", phase}`, `{type:"done", result}`, `{type:"error", error}`.
|
||||
Heartbeat comments every 15 s; stream closes after `done`/`error`.
|
||||
|
||||
## 7. State model
|
||||
|
||||
Server (single flask app):
|
||||
- `RunManager` — one active run; holds `RunState {run_id, status, events, result, error}`
|
||||
and an SSE queue; background daemon thread executes the `EvolutionOrchestrator`
|
||||
(MockLLM or OpenAICompatLLM) and wires `log` + proposer `on_step` to the queue;
|
||||
`cancel_event` lets the orchestrator stop between iterations.
|
||||
|
||||
Client (vanilla JS):
|
||||
- `state = {config, status, events[], result, evaluation}`; tab rendering is
|
||||
pure functions over `state`; `EventSource('/api/stream')` drives updates.
|
||||
|
||||
## 8. Error handling & resilience
|
||||
|
||||
- JSON payloads only; unknown/invalid fields → 400 with message.
|
||||
- Proposal failures / malformed LLM JSON are absorbed by the orchestrator (logged).
|
||||
- Run exceptions → `type:"error"` event + status `error` (traceback stored, exposed
|
||||
via status).
|
||||
- EventSource auto-reconnect on transient drops.
|
||||
- All file reads limited to 200 kB per file and whitelisted extension set.
|
||||
|
||||
## 9. Accessibility & usability
|
||||
|
||||
- Semantic HTML (tabs as buttons with aria), keyboard navigation for tree, focus
|
||||
states, sufficient contrast, no motion-only information.
|
||||
|
||||
## 10. Testing strategy
|
||||
|
||||
- Backend: Flask test client (mock backend) — run lifecycle, status, workspace tree,
|
||||
evaluation, single-flight 409, validation 400s, cancel.
|
||||
- Runner: unit test RunManager concurrency (double-start raises).
|
||||
- E2E smoke: boot `wikiskill serve` on a port; fetch `/`, `/api/health`, run
|
||||
`/api/run` with mock, poll status to completion, verify workspace artifacts.
|
||||
- Regression: full `pytest` suite (core + UI) must stay green.
|
||||
|
||||
## 11. Non-functional targets
|
||||
|
||||
- Zero CDN dependencies (offline usable).
|
||||
- Startup < 1 s; mock run completes in seconds; UI remains responsive during runs
|
||||
(threaded Flask).
|
||||
- Flask is the single new runtime dependency, grouped under `[project.optional-dependencies] ui`.
|
||||
- **Actions**: Start, Cancel (appears while running). Status pill: idle | starting |
|
||||
running | evaluating | done | error.
|
||||
- **Live console**: appended log lines; SSE-driven; auto-scrolls; monospace.
|
||||
- **ReAct event log**: marks `read_file` calls (path + found/not-found) and final
|
||||
proposal, rendered inline in the console with icons.
|
||||
|
||||
### 4.2 Dashboard tab
|
||||
- **R_best curve**: SVG polyline (x = iteration incl. baseline, y = R_best value).
|
||||
- **Validation bars**: one bar per evaluated proposal, colored green=Accepted /
|
||||
red=Rejected; dashed baseline marker.
|
||||
- **History table**: iteration, action, target skill, validation score, outcome.
|
||||
Reference in New Issue
Block a user