Configuration
+ +Live console
+ +Logs stream live; writer events (Wiki Maintainer / Skill + Proposer) and validation gating decisions appear as they happen.
+diff --git a/README.md b/README.md index 61ef2fb..09a6f70 100644 --- a/README.md +++ b/README.md @@ -53,11 +53,35 @@ this work: - **Paired bootstrap significance testing** (App. C) in `metrics.py`. - Early stop when `R_best == 1.0`. -## Quickstart +## Web UI + +An interactive workbench for running and inspecting the evolution loop: + +```bash +pip install -e ".[ui]" # installs Flask +wikiskill serve # http://127.0.0.1:5000 +# or: python -m wikiskill.cli serve --port 5000 +``` + +Tabs (see `ui_design.md` for the full design): +- **Run** — configure backend (offline Mock or any OpenAI-compatible endpoint), + iterations, ReAct turns; start/cancel; live SSE console including the Skill + Proposer's ReAct `read_file` steps. +- **Dashboard** — R_best curve, accepted/rejected validation-gating bars, history table. +- **Evaluation** — final skills vs no-skill baseline on the test split with paired + bootstrap p-value (paper §4 + App. C); re-evaluate any workspace. +- **Workspace** — browse the three-layer workspace (`raw/`, `wiki/`, `skills/`) and + view any artifact (JSON pretty-printed, traces, diffs). +- **Knowledge** — evolved skills (SKILL.md + PURPOSE.md), wiki pattern pages, index, + evolution log, skill-impact audit trail. +- **How it works** — the pipeline, Algorithm 1 pseudocode, and glossary mapped to the + paper. + +## Quickstart (CLI) ```bash pip install -e ".[dev]" -pytest # 14 tests incl. end-to-end loop +pytest # 30 tests incl. UI + end-to-end loop python -m wikiskill.demo # offline demo with the deterministic MockLLM ``` @@ -76,20 +100,20 @@ API key is read from `--api-key`, `WIKISKILL_API_KEY`, or `OPENAI_API_KEY`. Any OpenAI-compatible `/chat/completions` endpoint works (including local vLLM, as in the paper). `tasks.jsonl` lines: `{"id", "x", "y", "meta?"}`. -## Layout - +Layout ``` src/wikiskill/ ├── workspace.py # three layers + patch engine + scoped read_file ├── skills.py # skills layer, apply_proposal, skills-only snapshot/rollback ├── agents.py # InferenceAgent, WikiMaintainer, ReAct SkillProposer, stratified_sample -├── orchestrator.py # Algorithm 1 with gating/rollback + audit trail +├── orchestrator.py # Algorithm 1 with gating/rollback + audit trail + test evaluation ├── prompts.py # agent system prompts (adapted from paper Appendix E) ├── llm.py # LLM protocol, OpenAI-compatible client, MockLLM, extract_json ├── metrics.py # accuracy + paired bootstrap (1,000 iterations) ├── datasets.py # dataclasses + synthetic demo dataset -├── demo.py / cli.py -└── tests/… # unit + end-to-end tests +├── webapp/ # Flask UI (runner + api + SPA: see ui_design.md) +├── demo.py / cli.py # `wikiskill demo | evolve | serve` +└── tests/… # unit + end-to-end + UI tests ``` ## Notes & limitations (mirroring the paper) diff --git a/design.md b/design.md index e9e0813..facd36c 100644 --- a/design.md +++ b/design.md @@ -11,17 +11,16 @@ wikiskill/ │ ├── __init__.py │ ├── llm.py # LLM provider abstraction: MockLLM, OpenAICompatLLM │ ├── workspace.py # Three-layer workspace: raw/, wiki/, skills/ + patch engine +│ ├── skills.py # Skills layer: SKILL.md + PURPOSE.md, snapshot/rollback │ ├── prompts.py # System prompts (Inference / Wiki Maintainer / Skill Proposer) │ ├── agents.py # InferenceAgent, WikiMaintainer, SkillProposer (ReAct) -│ ├── orchestrator.py # Algorithm 1: evolution loop, gating, rollback, logging +│ ├── orchestrator.py # Algorithm 1: loop, gating, rollback, test evaluation (§4) │ ├── metrics.py # accuracy metric + paired bootstrap significance test │ ├── datasets.py # Task/Trace dataclasses + demo dataset +│ ├── webapp/ # Web UI (see ui_design.md): runner + Flask api + SPA │ ├── demo.py # End-to-end demo run with mock LLM -│ └── cli.py # `wikiskill evolve` / `wikiskill demo` entry point -└── tests/ - ├── test_workspace.py - ├── test_gating.py - └── test_e2e.py +│ └── cli.py # `wikiskill demo | evolve | serve` entry point +└── tests/ # unit, end-to-end, and UI tests ``` ## 2. Core data model (`datasets.py`) diff --git a/prd.md b/prd.md index 9ca3af8..a2f64ad 100644 --- a/prd.md +++ b/prd.md @@ -91,3 +91,9 @@ experience into the wiki so that subsequent skill updates build on it. workspace with all three layers populated, and accepts/rolls back proposals correctly. 3. Code mirrors paper notation (S_k, W_k, P_k, T_sample, R_best) and cites Algorithm 1 steps in docstrings. +4. **Web UI (ui_design.md)**: `wikiskill serve` boots an interactive workbench that can + start/cancel evolution runs, stream progress live, browse the three-layer workspace, + inspect skills/wiki artifacts, and report test-split evaluation with paired bootstrap. + UI covered by Flask test-client tests plus a live-server smoke test. +5. **Post-run evaluation**: every run reports test-split accuracy for evolved skills vs + no-skill baseline with paired-bootstrap significance (paper §4 + App. C). diff --git a/pyproject.toml b/pyproject.toml index 8d7126d..1d1b064 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -11,6 +11,7 @@ dependencies = [] [project.optional-dependencies] dev = ["pytest>=7"] +ui = ["flask>=3.0"] [project.scripts] wikiskill = "wikiskill.cli:main" diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..f7d47c9 --- /dev/null +++ b/requirements.txt @@ -0,0 +1 @@ +flask>=3.0 \ No newline at end of file diff --git a/smoke_test.py b/smoke_test.py new file mode 100644 index 0000000..704d410 --- /dev/null +++ b/smoke_test.py @@ -0,0 +1,87 @@ +"""Live smoke test: boots the real wikiskill serve stack and exercises it over HTTP.""" +import json +import subprocess +import sys +import time +import urllib.request + +PORT = 8765 +BASE = f"http://127.0.0.1:{PORT}" +ok = True + + +def check(name, cond, extra=""): + global ok + print(f"[{'PASS' if cond else 'FAIL'}] {name} {extra}") + ok = ok and bool(cond) + + +def get(path, timeout=20): + with urllib.request.urlopen(BASE + path, timeout=timeout) as r: + return r.status, r.read().decode("utf-8") + + +proc = subprocess.Popen( + [sys.executable, "-m", "wikiskill.cli", "serve", "--host", "127.0.0.1", + "--port", str(PORT)], + stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + +try: + # wait for server + for _ in range(100): + try: + st, body = get("/api/health", timeout=5) + break + except Exception: + time.sleep(0.3) + else: + raise RuntimeError("server did not start") + + check("health", st == 200 and '"ok":true' in body.replace(" ", "")) + + st, body = get("/") + check("index served", st == 200 and 'WikiSkill' in body and 'run-form' in body) + + req = urllib.request.Request(BASE + "/api/run", + data=json.dumps({"llm_backend": "mock", + "iterations": 2, + "workspace_root": "smoke_run"}).encode(), + headers={"Content-Type": "application/json"}) + with urllib.request.urlopen(req, timeout=10) as r: + st, body = r.status, r.read().decode() + check("run started", st == 202 and '"run_id"' in body) + + done = False + for _ in range(200): + st, body = get("/api/status") + state = json.loads(body)["state"] + if state and state["status"] in ("done", "error"): + done = state["status"] == "done" + break + time.sleep(0.2) + check("run completed (mock, live server)", done) + + st, body = get("/api/workspace?root=smoke_run") + data = json.loads(body) + check("workspace browsable", st == 200 and "wiki/index.md" in data["files"], + f"({sorted(data['files'])[:3]})") + + req = urllib.request.Request(BASE + "/api/evaluate", + data=json.dumps({"llm_backend": "mock", + "workspace_root": "smoke_run"}).encode(), + headers={"Content-Type": "application/json"}) + with urllib.request.urlopen(req, timeout=30) as r: + ev = json.loads(r.read().decode())["evaluation"] + check("evaluation end-to-end", ev["significant"] is True and + ev["skilled_accuracy"] == 1.0, + f"skilled={ev['skilled_accuracy']}") + +finally: + proc.terminate() + try: + proc.wait(timeout=5) + except subprocess.TimeoutExpired: + proc.kill() + +print("SMOKE TEST", "PASSED" if ok else "FAILED") +sys.exit(0 if ok else 1) \ No newline at end of file diff --git a/src/wikiskill/agents.py b/src/wikiskill/agents.py index 90d3517..882299d 100644 --- a/src/wikiskill/agents.py +++ b/src/wikiskill/agents.py @@ -112,9 +112,18 @@ class SkillProposer: """§3.2.3: P_k = M_P(W'_k, S_{k-1}, T_train,k). Multi-turn ReAct agent that reads the wiki index, skill-impact tracker, and traces on demand via read_file.""" - def __init__(self, llm: LLM, ws: Workspace, skills: SkillsLayer, max_turns: int = 20): + def __init__(self, llm: LLM, ws: Workspace, skills: SkillsLayer, max_turns: int = 20, + on_step=None): self.llm, self.ws, self.skills = llm, ws, skills self.max_turns = max_turns + self.on_step = on_step # callable(step: dict) for UI transparency + + def _step(self, **payload): + if self.on_step is not None: + try: + self.on_step({"proposer_step": True, **payload}) + except Exception: # pragma: no cover - never break evolution on UI hooks + pass def propose(self, outcome_summary: str, iteration: int) -> Optional[dict]: index = self.ws.wiki_index() or "(empty wiki)" @@ -126,19 +135,26 @@ class SkillProposer: "(raw/iter_.../...) as needed, then finish with your atomic proposal JSON.") messages = [{"role": "system", "content": SKILL_PROPOSER_SYSTEM}, {"role": "user", "content": user}] - for _ in range(self.max_turns): + for turn in range(1, self.max_turns + 1): text = self.llm.chat(messages, temperature=0.2) parsed = extract_json(text) if parsed and parsed.get("action") in ("create", "patch"): + self._step(turn=turn, kind="proposal", skill=parsed.get("skill"), + action=parsed.get("action")) return parsed fm = re.search(r'"path"\s*:\s*"([^"]+)"', text) if fm: - obs = self.ws.read_file(fm.group(1)) + path = fm.group(1) + obs = self.ws.read_file(path) + self._step(turn=turn, kind="read", path=path, + found=not obs.startswith("ERROR"), + snippet=obs[:200]) messages.append({"role": "assistant", "content": text}) messages.append({"role": "user", "content": f"read_file observation:\n{obs}"}) continue messages.append({"role": "assistant", "content": text}) + self._step(turn=turn, kind="no_tool", snippet=text[:200]) messages.append({"role": "user", "content": "Reply with a read_file tool call or the finish JSON."}) return None diff --git a/src/wikiskill/cli.py b/src/wikiskill/cli.py index 40bb841..8662a47 100644 --- a/src/wikiskill/cli.py +++ b/src/wikiskill/cli.py @@ -25,6 +25,11 @@ def main(argv=None) -> int: e.add_argument("--base-url", default=None) e.add_argument("--api-key", default=None) e.add_argument("--iterations", type=int, default=8) + + s = sub.add_parser("serve", help="run the WikiSkill web UI (Flask)") + s.add_argument("--host", default="127.0.0.1") + s.add_argument("--port", type=int, default=5000) + s.add_argument("--debug", action="store_true") args = p.parse_args(argv) if args.cmd == "demo": @@ -32,6 +37,17 @@ def main(argv=None) -> int: run_demo(args.root, args.iterations) return 0 + if args.cmd == "serve": + try: + from .webapp import create_app + except ImportError as exc: + print("The UI requires Flask. Install it with: " + "pip install -e '.[ui]' (or: pip install flask)", file=sys.stderr) + return 2 + app = create_app() + app.run(host=args.host, port=args.port, debug=args.debug, threaded=True) + return 0 + from .datasets import Dataset, Task, ToolSpec, load_jsonl from .llm import OpenAICompatLLM from .orchestrator import EvolutionOrchestrator diff --git a/src/wikiskill/demo.py b/src/wikiskill/demo.py index 9d8ae37..83b8004 100644 --- a/src/wikiskill/demo.py +++ b/src/wikiskill/demo.py @@ -12,16 +12,21 @@ from .workspace import Workspace def run_demo(root: str = "wikiskill_demo_workspace", K: int = 3) -> dict: ws = Workspace(root) - skills_logger = [] orch = EvolutionOrchestrator( llm=MockLLM(), dataset=make_demo_dataset(), ws=ws, tools=_demo_tools(), K=K) result = orch.run() + eval_res = orch.evaluate_test() + print("\nTest-set evaluation (final skills vs no-skill baseline):") + print(f" skilled accuracy : {eval_res['skilled_accuracy']:.4f}") + print(f" baseline accuracy: {eval_res['baseline_accuracy']:.4f}") + print(f" bootstrap p : {eval_res['bootstrap_p_value']:.4f} " + f"({'significant' if eval_res['significant'] else 'not significant'})") print("\nWorkspace layout:") for dirpath, _, files in os.walk(root): rel = os.path.relpath(dirpath, root) for fn in sorted(files): print(f" {os.path.join(rel, fn)}") - return result + return {**result, "evaluation": eval_res} if __name__ == "__main__": diff --git a/src/wikiskill/orchestrator.py b/src/wikiskill/orchestrator.py index ce3f215..497269d 100644 --- a/src/wikiskill/orchestrator.py +++ b/src/wikiskill/orchestrator.py @@ -31,7 +31,9 @@ class EvolutionOrchestrator: def __init__(self, llm: LLM, dataset: Dataset, ws: Workspace, tools: list, metric: Callable[[List[bool]], float] = None, K: int = 8, - max_react_turns: int = 20, log: Optional[Callable[[str], None]] = None): + max_react_turns: int = 20, log: Optional[Callable[[str], None]] = None, + proposer_on_step=None, + cancel_event=None): from .metrics import accuracy self.llm = llm self.dataset = dataset @@ -41,17 +43,20 @@ class EvolutionOrchestrator: self.skills = SkillsLayer(ws) self.inference = InferenceAgent(llm, tools, ws, self.skills) self.maintainer = WikiMaintainer(llm, ws) - self.proposer = SkillProposer(llm, ws, self.skills, max_turns=max_react_turns) + self.proposer = SkillProposer(llm, ws, self.skills, max_turns=max_react_turns, + on_step=proposer_on_step) self.log = log or (lambda s: print(s, flush=True)) + self.cancel_event = cancel_event # optional threading.Event for mid-run stop self.history: List[dict] = [] self.r_best = -1.0 # ------------------------------------------------------------------ - def _rollout(self, tasks, iteration: int, split: str) -> List[Trajectory]: + def _rollout(self, tasks, iteration, split: str) -> List[Trajectory]: trajs = [self.inference.run(t) for t in tasks] for t in trajs: self.ws.add_raw_trace(iteration, t.task_id, - {**t.to_dict(), "split": split, "iteration": iteration}) + {**t.to_dict(), "split": split, "iteration": + str(iteration)}) return trajs def _score(self, trajs: List[Trajectory]) -> float: @@ -76,6 +81,9 @@ class EvolutionOrchestrator: def run(self) -> dict: self._baseline_validation() for k in range(1, self.K + 1): + if self.cancel_event is not None and self.cancel_event.is_set(): + self.log("Run cancelled by user.") + break if self.r_best >= 1.0: self.log("R_best reached 1.0; early stop.") break @@ -150,3 +158,47 @@ class EvolutionOrchestrator: self.log(f"\nEvolution complete. R_best={self.r_best:.4f}, skills={final_skills}") return result + # ------------------------------------------------------------------ + def evaluate_test(self) -> dict: + """Post-evolution evaluation on the test split (paper §4). + + Compares the final evolved skills against a no-skill baseline on D_test and + reports both accuracies plus a paired bootstrap significance p-value + (paper Appendix C, 1,000 iterations). + """ + from .metrics import paired_bootstrap + # Eval traces are auxiliary (not part of the immutable evolution history); + # clear any previous eval traces so re-evaluation works on the same workspace. + for tag in ("eval_skilled", "eval_baseline"): + d = os.path.join(self.ws.root, "raw", tag) + if os.path.isdir(d): + for fn in os.listdir(d): + os.remove(os.path.join(d, fn)) + os.rmdir(d) + + skilled_trajs = self._rollout(self.dataset.test, "eval_skilled", "test") + skilled = [t.correct for t in skilled_trajs] + skilled_acc = self.metric(skilled) + + snap = self.skills.snapshot() + try: + for name in self.skills.list_skills(): + d = os.path.join(self.ws.root, "skills", name) + for fn in os.listdir(d): + os.remove(os.path.join(d, fn)) + os.rmdir(d) + base_trajs = self._rollout(self.dataset.test, "eval_baseline", "test") + finally: + self.skills.restore(snap) + base = [t.correct for t in base_trajs] + base_acc = self.metric(base) + + p = paired_bootstrap(skilled, base, n=1000, seed=42) + self.log(f"Test evaluation: skilled={skilled_acc:.4f} " + f"baseline={base_acc:.4f} bootstrap_p={p:.4f}") + return {"skilled_accuracy": skilled_acc, + "baseline_accuracy": base_acc, + "bootstrap_p_value": p, + "significant": p < 0.05, + "n_test": len(skilled)} + diff --git a/src/wikiskill/skills.py b/src/wikiskill/skills.py index 481dddf..bd4071d 100644 --- a/src/wikiskill/skills.py +++ b/src/wikiskill/skills.py @@ -76,15 +76,24 @@ class SkillsLayer: raise ValueError(f"unknown proposal action: {action!r}") # ---------- gating rollback (§3.2.4): skills only, wiki untouched ---------- - def snapshot(self) -> Dict[str, str]: - return {name: self.skill_content(name) or "" for name in self.list_skills()} + def snapshot(self) -> Dict[str, dict]: + """Snapshot the active skill state (SKILL.md + PURPOSE.md) so a rejected + proposal can be rolled back exactly to S_{k-1}. The wiki is never included.""" + return { + name: {"SKILL.md": self.skill_content(name) or "", + "PURPOSE.md": self.purpose(name) or ""} + for name in self.list_skills() + } - def restore(self, snap: Dict[str, str]) -> None: + def restore(self, snap: Dict[str, dict]) -> None: for name in self.list_skills(): if name not in snap: d = os.path.join(self.ws.root, "skills", name) for fn in os.listdir(d): os.remove(os.path.join(d, fn)) os.rmdir(d) - for name, content in snap.items(): - self.ws.write_file(os.path.join("skills", name, "SKILL.md"), content) + for name, files in snap.items(): + self.ws.write_file(os.path.join("skills", name, "SKILL.md"), + files["SKILL.md"]) + self.ws.write_file(os.path.join("skills", name, "PURPOSE.md"), + files["PURPOSE.md"]) diff --git a/src/wikiskill/webapp/__init__.py b/src/wikiskill/webapp/__init__.py new file mode 100644 index 0000000..404fdc1 --- /dev/null +++ b/src/wikiskill/webapp/__init__.py @@ -0,0 +1,4 @@ +"""WikiSkill web UI package (ui_design.md).""" +from .api import create_app + +__all__ = ["create_app"] \ No newline at end of file diff --git a/src/wikiskill/webapp/api.py b/src/wikiskill/webapp/api.py new file mode 100644 index 0000000..743bac6 --- /dev/null +++ b/src/wikiskill/webapp/api.py @@ -0,0 +1,183 @@ +"""Flask application factory + REST/SSE endpoints for the WikiSkill UI.""" +from __future__ import annotations + +import json +import os + +from flask import Blueprint, Flask, Response, jsonify, render_template, request + +from .. import __version__ +from .runner import RunManager + +bp = Blueprint("wikiskill_ui", __name__) + +_manager: RunManager = None # module-level singleton + + +def get_manager() -> RunManager: + global _manager + if _manager is None: + _manager = RunManager() + return _manager + + +def create_app(test_config=None) -> Flask: + app = Flask(__name__) + app.config.from_mapping(SECRET_KEY="wikiskill-local-dev") + if test_config: + app.config.update(test_config) + app.register_blueprint(bp) + return app + + +def _safe_root(root: str) -> str: + """Reject absolute paths and path traversal; normalize to a relative name.""" + root = (root or "").strip().replace("\\", "/").strip("/") + if not root: + raise ValueError("workspace root must not be empty") + if os.path.isabs(root): + raise ValueError("workspace root must be a relative path") + if any(part == ".." for part in root.split("/")): + raise ValueError("workspace root must not contain '..'") + return root + + +# --------------------------------------------------------------------------- +@bp.get("/") +def index(): + return render_template("index.html") + + +@bp.get("/api/health") +def health(): + return jsonify({"ok": True, "version": __version__}) + + +@bp.post("/api/run") +def start_run(): + config = request.get_json(silent=True) or {} + backend = config.get("llm_backend", "mock") + if backend not in ("mock", "openai"): + return jsonify({"error": "llm_backend must be 'mock' or 'openai'"}), 400 + if backend == "openai" and not config.get("model"): + return jsonify({"error": "'model' is required when llm_backend=openai"}), 400 + root = config.get("workspace_root") + if root: + try: + _safe_root(root) + except ValueError as e: + return jsonify({"error": str(e)}), 400 + try: + info = get_manager().start(config) + except PermissionError as e: + return jsonify({"error": str(e)}), 409 + return jsonify(info), 202 + + +@bp.get("/api/status") +def status(): + return jsonify(get_manager().status()) + + +@bp.get("/api/stream") +def stream(): + def gen(): + yield from get_manager().stream() + return Response(gen(), mimetype="text/event-stream", + headers={"Cache-Control": "no-cache", + "X-Accel-Buffering": "no"}) + + +@bp.post("/api/cancel") +def cancel(): + return jsonify({"cancelled": get_manager().cancel()}) + + +# --------------------------------------------------------------------------- +_TREE_SKIP = {".pytest_cache", "__pycache__", ".git"} +_TEXT_EXT = {".md", ".json", ".txt", ".py", ".toml", ".diff", ".log"} + + +@bp.get("/api/workspace") +def workspace(): + try: + root = _safe_root(request.args.get("root", "")) + except ValueError as e: + return jsonify({"error": str(e)}), 400 + if not os.path.isdir(root): + return jsonify({"error": f"workspace '{root}' does not exist", + "root": root}), 404 + return jsonify({"root": root, "tree": _build_tree(root), + "files": _read_text_files(root)}) + + +def _build_tree(root: str) -> dict: + def build(path: str) -> dict: + name = os.path.basename(path) or root + rel = os.path.relpath(path, root).replace("\\", "/") + node = {"name": name, "path": rel, "type": "dir", "children": []} + for entry in sorted(os.listdir(path)): + if entry in _TREE_SKIP: + continue + full = os.path.join(path, entry) + if os.path.isdir(full): + node["children"].append(build(full)) + else: + node["children"].append({ + "name": entry, + "path": os.path.relpath(full, root).replace("\\", "/"), + "type": "file"}) + return node + return build(root) + + +def _read_text_files(root: str) -> dict: + """Collect text-file contents (bounded) for the UI viewer.""" + out = {} + for dirpath, dirnames, filenames in os.walk(root): + dirnames[:] = [d for d in dirnames if d not in _TREE_SKIP] + for fn in filenames: + if os.path.splitext(fn)[1] not in _TEXT_EXT: + continue + full = os.path.join(dirpath, fn) + rel = os.path.relpath(full, root).replace("\\", "/") + try: + with open(full, "r", encoding="utf-8") as f: + out[rel] = f.read(200_000) + except (OSError, UnicodeDecodeError): + out[rel] = "(unreadable/binary)" + return out + + +# --------------------------------------------------------------------------- +@bp.post("/api/evaluate") +def evaluate(): + """Re-evaluate final skills vs baseline on the test split (paper §4).""" + config = request.get_json(silent=True) or {} + backend = config.get("llm_backend", "mock") + if backend not in ("mock", "openai"): + return jsonify({"error": "llm_backend must be 'mock' or 'openai'"}), 400 + try: + root = _safe_root(config.get("workspace_root", "")) + except ValueError as e: + return jsonify({"error": str(e)}), 400 + if not os.path.isdir(root): + return jsonify({"error": f"workspace '{root}' does not exist"}), 404 + + from ..datasets import _demo_tools, make_demo_dataset + from ..llm import MockLLM, OpenAICompatLLM + from ..orchestrator import EvolutionOrchestrator + from ..workspace import Workspace + + if backend == "openai": + llm = OpenAICompatLLM(model=config.get("model", ""), + base_url=config.get("base_url") or None, + api_key=config.get("api_key") or None) + else: + llm = MockLLM() + orch = EvolutionOrchestrator(llm=llm, dataset=make_demo_dataset(), + ws=Workspace(root), tools=_demo_tools(), + K=int(config.get("iterations", 8)), + log=lambda msg: None) + res = orch.evaluate_test() + return jsonify({"evaluation": res, "workspace_root": root}) \ No newline at end of file diff --git a/src/wikiskill/webapp/runner.py b/src/wikiskill/webapp/runner.py new file mode 100644 index 0000000..86f7125 --- /dev/null +++ b/src/wikiskill/webapp/runner.py @@ -0,0 +1,166 @@ +"""RunManager: executes the WikiSkill evolution loop in a background thread and +streams progress to the browser via Server-Sent Events (ui_design.md §6–§7).""" +from __future__ import annotations + +import datetime as _dt +import json +import os +import queue +import threading +import traceback as _tb +import uuid +from typing import Optional + +from ..datasets import _demo_tools, make_demo_dataset +from ..llm import MockLLM, OpenAICompatLLM +from ..orchestrator import EvolutionOrchestrator +from ..workspace import Workspace + + +def _is_fresh_root(path: str) -> bool: + """True if the workspace root has no prior evolution history (raw/iter_*).""" + raw = os.path.join(path, "raw") + if not os.path.isdir(raw): + return True + return not any(d.startswith("iter_") for d in os.listdir(raw)) + + +class RunManager: + """Single-flight run manager. `start` spawns a daemon thread; `stream` yields SSE.""" + + def __init__(self): + self._lock = threading.Lock() + self._state: Optional[dict] = None + self._events: "queue.Queue[dict]" = queue.Queue() + self._cancel_event = threading.Event() + + # ---- public API ---------------------------------------------------- + @property + def active(self) -> bool: + with self._lock: + return bool(self._state and self._state.get("status") + in ("starting", "running", "evaluating")) + + def status(self) -> dict: + # Read without holding the lock while computing `active` — the property + # also acquires the (non-reentrant) lock, which would deadlock. + state = self._state + return {"active": bool(state and state.get("status") + in ("starting", "running", "evaluating")), + "state": state} + + def start(self, config: dict) -> dict: + with self._lock: + if self._state and self._state.get("status") in ( + "starting", "running", "evaluating"): + raise PermissionError("another evolution run is already active") + run_id = uuid.uuid4().hex[:12] + self._state = {"run_id": run_id, "config": dict(config), "status": + "starting", "events": [], "result": None, "error": None} + self._cancel_event = threading.Event() + threading.Thread(target=self._execute, args=(run_id, dict(config)), + daemon=True).start() + return {"run_id": run_id, "active": True} + + def cancel(self) -> bool: + self._cancel_event.set() + return True + + def _reset_for_tests(self) -> None: + """Test-only: drop state and queue so a fresh run can start.""" + with self._lock: + self._state = None + self._cancel_event = threading.Event() + while True: + try: + self._events.get_nowait() + except queue.Empty: + break + + # ---- streaming ----------------------------------------------------- + def _emit(self, **payload) -> None: + self._events.put(dict(payload)) + with self._lock: + if self._state is not None: + self._state["events"].append(dict(payload)) + self._state["events"] = self._state["events"][-500:] + + def stream(self): + """SSE generator: emits events; closes after done/error.""" + while True: + try: + payload = self._events.get(timeout=15) + except queue.Empty: + yield ":\n\n" # heartbeat + continue + data = json.dumps(payload) + yield f"data: {data}\n\n" + if payload.get("type") in ("done", "error"): + break + + # ---- execution ----------------------------------------------------- + def _build_llm(self, config: dict): + backend = config.get("llm_backend", "mock") + if backend == "openai": + return OpenAICompatLLM( + model=config.get("model", ""), + base_url=config.get("base_url") or None, + api_key=config.get("api_key") or None, + temperature=float(config.get("temperature", 0.7))) + return MockLLM() + + def _root_for(self, config: dict) -> str: + root = config.get("workspace_root") or "" + if root: + return str(root).strip().strip("/").replace("\\", "/").strip("/") + stamp = _dt.datetime.now().strftime("%Y%m%d_%H%M%S") + return f"runs/run_{stamp}" + + def _execute(self, run_id: str, config: dict) -> None: + try: + self._emit(type="status", phase="starting", + message="Preparing run...") + root = self._root_for(config) + if not _is_fresh_root(root): + raise ValueError( + f"workspace root '{root}' already contains evolution history; " + "choose a fresh root (the raw layer is immutable)") + llm = self._build_llm(config) + ws = Workspace(root) + orch = EvolutionOrchestrator( + llm=llm, dataset=make_demo_dataset(), ws=ws, tools=_demo_tools(), + K=int(config.get("iterations", 8)), + max_react_turns=int(config.get("max_react_turns", 20)), + cancel_event=self._cancel_event, + log=lambda msg: self._emit(type="log", message=msg), + proposer_on_step=lambda ev: self._emit(type="react", **ev)) + self._emit(type="status", phase="running", message=f"root={root}") + result = orch.run() + self._emit(type="status", phase="evaluating", + message="Evaluating evolved skills on the test split...") + evaluation = orch.evaluate_test() + with self._lock: + if self._state is not None: + self._state["status"] = "done" + self._state["result"] = {"evolution": result, + "evaluation": evaluation, + "workspace_root": root} + self._emit(type="done", result=self._state["result"] if self._state else + {"evolution": result, "evaluation": evaluation, + "workspace_root": root}) + except Exception as exc: # noqa: BLE001 - surface any failure to the UI + err = f"{type(exc).__name__}: {exc}" + print(err, flush=True) + print(_tb.format_exc(), flush=True) + with self._lock: + if self._state is not None: + self._state["status"] = "error" + self._state["error"] = err + "\n" + _tb.format_exc() + self._emit(type="error", error=str(exc)) + self._state = None + self._cancel_event = threading.Event() + while True: + try: + self._events.get_nowait() + except queue.Empty: + break \ No newline at end of file diff --git a/src/wikiskill/webapp/static/app.js b/src/wikiskill/webapp/static/app.js new file mode 100644 index 0000000..28e5d94 --- /dev/null +++ b/src/wikiskill/webapp/static/app.js @@ -0,0 +1,437 @@ +"use strict"; +/* WikiSkill UI client — vanilla JS, no dependencies (ui_design.md §6–§7). */ + +const $ = (id) => document.getElementById(id); +const S = { + ws: null, // workspace data {root, tree, files} + lastResult: null, // {evolution, evaluation, workspace_root} + activeRoot: null, +}; + +/* ---------- utils ---------- */ +function esc(s) { + return String(s ?? "").replace(/[&<>"']/g, (c) => ({ + "&": "&", "<": "<", ">": ">", '"': """, "'": "'", + }[c])); +} +async function api(path, opts = {}) { + const res = await fetch(path, { + headers: { "Content-Type": "application/json" }, + method: opts.method || (opts.body ? "POST" : "GET"), + body: opts.body ? JSON.stringify(opts.body) : undefined, + }); + let data = null; + try { data = await res.json(); } catch (_) { /* empty */ } + if (!res.ok) { + const msg = (data && data.error) || `HTTP ${res.status}`; + throw new Error(msg); + } + return data; +} +function fmtPct(v) { return `${(v * 100).toFixed(1)}%`; } + +/* ---------- tabs ---------- */ +function switchTab(name) { + document.querySelectorAll(".tab").forEach((b) => + b.classList.toggle("is-active", b.dataset.tab === name)); + document.querySelectorAll(".tab-panel").forEach((p) => + p.classList.toggle("is-active", p.id === `tab-${name}`)); +} +document.querySelectorAll(".tab").forEach((b) => + b.addEventListener("click", () => switchTab(b.dataset.tab))); + +/* ---------- status pill ---------- */ +function setStatus(status, label) { + const pill = $("status-pill"); + pill.textContent = label || status; + const cls = { idle: "pill-idle", starting: "pill-running", running: "pill-running", + evaluating: "pill-evaluating", done: "pill-done", error: "pill-error" }; + pill.className = `pill ${cls[status] || "pill-idle"}`; + $("btn-cancel").classList.toggle("hidden", + !["starting", "running", "evaluating"].includes(status)); + $("btn-start").disabled = ["starting", "running", "evaluating"].includes(status); +} + +/* ---------- console ---------- */ +function logLine(text, cls) { + const con = $("console"); + const div = document.createElement("span"); + div.className = `ln ${cls || "ln-tail"}`; + div.textContent = text; + con.appendChild(div); + con.scrollTop = con.scrollHeight; +} +function renderEvent(ev) { + if (ev.type === "log") logLine(ev.message, "ln-tail"); + else if (ev.type === "status") { + logLine(`—— ${ev.message || ev.phase} ——`, "ln-cmd"); + if (ev.phase) setStatus(ev.phase === "evaluating" ? "evaluating" : + ["starting", "running"].includes(ev.phase) ? "running" : ev.phase); + } else if (ev.type === "react") { + if (ev.kind === "read") { + logLine(` [proposer] read ${ev.path} ${ev.found ? "" : "(MISSING)"}`, "ln-react"); + } else if (ev.kind === "proposal") { + logLine(` [proposer] → proposal ${ev.action} '${ev.skill}'`, "ln-react"); + } else { + logLine(` [proposer] ${esc(ev.kind)}`, "ln-react"); + } + } else if (ev.type === "done") { + S.lastResult = ev.result; + setStatus("done", "done"); + if (ev.result) { + showDashboard(ev.result.evolution); + showEval(ev.result.evaluation); + S.activeRoot = ev.result.workspace_root; + $("log-root-label").textContent = `→ ${S.activeRoot}`; + } + logLine("✔ run finished", "ln-accept"); + } else if (ev.type === "error") { + logLine(`✖ error: ${ev.error}`, "ln-reject"); + setStatus("error", "error"); + } +} + +/* ---------- run form ---------- */ +$("cfg-backend").addEventListener("change", () => { + $("openai-fields").classList.toggle("hidden", $("cfg-backend").value !== "openai"); +}); +$("run-form").addEventListener("submit", async (e) => { +/* ---------- SSE ---------- */ +let esSource = null; +function connectStream() { + if (esSource) esSource.close(); + setStatus("running", "running…"); + esSource = new EventSource("/api/stream"); + esSource.onmessage = (m) => { + let ev; + try { ev = JSON.parse(m.data); } catch (_) { return; } + renderEvent(ev); + if (ev.type === "done" || ev.type === "error") esSource.close(); + }; + esSource.onerror = () => { if (esSource) esSource.close(); }; +} + +/* ---------- dashboard ---------- */ +function showDashboard(evo) { + $("dash-emptystate").classList.add("hidden"); + $("dash-charts").classList.remove("hidden"); + const history = evo.history || []; + drawRbest(history); + drawGate(history); + const tbody = document.querySelector("#dash-table tbody"); + tbody.innerHTML = ""; + if (!history.length) { + tbody.innerHTML = '
No evolved skills in this workspace.
'; return; } + out.innerHTML = ""; + names.forEach((skillPath) => { + const purposePath = skillPath.replace("/SKILL.md", "/PURPOSE.md"); + const card = document.createElement("div"); + card.className = "skill-card"; + const name = skillPath.split("/")[1]; + card.innerHTML = `${esc(files[skillPath])}${esc(files[purposePath] || "(none)")}`;
+ out.appendChild(card);
+ });
+ } else {
+ const map = {
+ patterns: (p) => p.startsWith("wiki/patterns/"),
+ index: (p) => p === "wiki/index.md",
+ logs: (p) => p === "wiki/logs.md",
+ impact: (p) => p === "wiki/skill-impact.md",
+ };
+ const keep = map[ktab];
+ const paths = Object.keys(files).filter(keep).sort();
+ if (!paths.length) { out.innerHTML = 'Nothing here yet.
'; return; } + out.innerHTML = ""; + paths.forEach((p) => { + const box = document.createElement("div"); + box.className = "kdoc"; + box.innerHTML = `${esc(p)}\n\n${esc(files[p])}`; + out.appendChild(box); + }); + } +} + +/* ---------- restore state on load ---------- */ +(async function init() { + try { + const st = await api("/api/status"); + const state = st.state; + if (state) { + (state.events || []).forEach(renderEvent); + if (state.result) { + S.lastResult = state.result; + showDashboard(state.result.evolution); + showEval(state.result.evaluation); + S.activeRoot = state.result.workspace_root; + $("log-root-label").textContent = `→ ${S.activeRoot}`; + $("eval-root").value = S.activeRoot || ""; + } + if (state.status === "done") setStatus("done", "done"); + if (st.active) connectStream(); + } + } catch (_) { /* server unreachable; keep idle state */ } +})(); + $("form-error").textContent = err.message; + } +}); +$("btn-cancel").addEventListener("click", async () => { + try { await api("/api/cancel", { body: {} }); logLine("—— cancel requested ——", "ln-cmd"); } + catch (_) { /* ignore */ } +}); \ No newline at end of file diff --git a/src/wikiskill/webapp/static/style.css b/src/wikiskill/webapp/static/style.css new file mode 100644 index 0000000..9d2c3ed --- /dev/null +++ b/src/wikiskill/webapp/static/style.css @@ -0,0 +1,141 @@ +:root { + --bg: #f7f7f5; --panel: #ffffff; --ink: #1a1a1a; --muted: #6b6b6b; + --accent: #3b5bdb; --accent-soft: #e7ebfc; --ok: #099268; --ok-soft: #e6f7f1; + --bad: #e03131; --bad-soft: #fdeaea; --warn: #f59f00; --border: #e3e1dc; + --mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; + --sans: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; +} +* { box-sizing: border-box; } +html, body { margin: 0; } +body { background: var(--bg); color: var(--ink); font-family: var(--sans); font-size: 14px; line-height: 1.5; } +.hidden { display: none !important; } +.muted { color: var(--muted); font-weight: 400; } +.small { font-size: 12px; } +a { color: var(--accent); } + +/* ---------- header / tabs ---------- */ +.topbar { display: flex; justify-content: space-between; align-items: center; + padding: 14px 22px; background: #fff; border-bottom: 1px solid var(--border); } +.brand-kicker { font-size: 11px; letter-spacing: .08em; text-transform: uppercase; color: var(--muted); display: block; } +.brand h1 { margin: 0; font-size: 20px; } +.brand .sub { color: var(--muted); font-weight: 400; font-size: 14px; } +.topbar-right { display: flex; align-items: center; gap: 12px; } +.small-link { font-size: 12px; } + +.pill { border-radius: 999px; padding: 3px 12px; font-size: 12px; font-weight: 600; } +.pill-idle { background: #eee; color: #555; } +.pill-running { background: var(--accent-soft); color: var(--accent); } +.pill-evaluating { background: #f3ecfb; color: #7b2cbf; } +.pill-done { background: var(--ok-soft); color: var(--ok); } +.pill-error { background: var(--bad-soft); color: var(--bad); } + +.tabs { display: flex; gap: 4px; padding: 10px 22px 0; background: #fff; + border-bottom: 1px solid var(--border); } +.tab { border: 0; background: transparent; padding: 8px 14px; font-size: 13px; + font-weight: 600; color: #555; cursor: pointer; border-radius: 8px 8px 0 0; } +.tab:hover { background: #f2f2f0; } +.tab.is-active { color: var(--accent); background: var(--accent-soft); } + +/* ---------- layout ---------- */ +main { max-width: 1180px; margin: 18px auto; padding: 0 18px; } +.tab-panel { display: none; } +.tab-panel.is-active { display: block; } +.grid-2 { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; align-items: start; } +@media (max-width: 860px) { .grid-2 { grid-template-columns: 1fr; } } +.grid-2-inner { display: grid; grid-template-columns: 1fr 1fr; gap: 10px; } + +.card { background: var(--panel); border: 1px solid var(--border); border-radius: 12px; + padding: 18px 20px; margin-bottom: 16px; } +.card h2 { margin: 0 0 10px; font-size: 16px; } +.card h3 { font-size: 14px; margin: 14px 0 6px; } + +/* ---------- forms ---------- */ +form label { display: block; font-weight: 600; font-size: 13px; margin: 10px 0 4px; } +form input, form select { width: 100%; padding: 7px 9px; font-size: 13px; + border: 1px solid var(--border); border-radius: 8px; background: #fff; } +.actions { display: flex; gap: 10px; margin-top: 14px; } +.btn { background: #fff; color: var(--ink); border: 1px solid var(--border); + border-radius: 8px; padding: 7px 14px; font-size: 13px; font-weight: 600; cursor: pointer; } +.btn:hover { border-color: var(--accent); color: var(--accent); } +.btn-primary { background: var(--accent); color: #fff; border-color: var(--accent); } +.btn-primary:hover { opacity: .9; color: #fff; } +.btn-danger { background: var(--bad); color: #fff; border-color: var(--bad); } +.btn-danger:hover { opacity: .9; color: #fff; } +.form-error { color: var(--bad); font-size: 12px; min-height: 16px; margin: 8px 0 0; } +.load-row { display: flex; gap: 10px; align-items: end; margin-bottom: 12px; } +.load-row label { flex: 1; display: block; font-size: 12px; font-weight: 600; } +.load-row input { width: 100%; padding: 6px 8px; border: 1px solid var(--border); border-radius: 8px; } +/* ---------- console ---------- */ +.console { background: #15181d; color: #d7e0f0; font-family: var(--mono); + font-size: 12px; padding: 12px; border-radius: 10px; height: 460px; overflow: auto; white-space: pre-wrap; } +.console .ln { display: block; } +.console .ln-tail { color: #8a93a6; } +.console .ln-cmd { color: #7cc4ff; } +.console .ln-react { color: #f2c55c; } +.console .ln-accept { color: #5ee6a7; } +.console .ln-reject { color: #ff8d8d; } + +/* ---------- charts ---------- */ +.chart { width: 100%; height: auto; background: #fff; border: 1px solid var(--border); border-radius: 10px; } +.chart text { font-size: 10px; fill: #666; } + +/* ---------- tables ---------- */ +.table-wrap { overflow: auto; } +table { border-collapse: collapse; width: 100%; font-size: 13px; } +th, td { text-align: left; padding: 6px 10px; border-bottom: 1px solid var(--border); } +th { font-size: 12px; color: var(--muted); } +.badge { border-radius: 999px; padding: 2px 10px; font-size: 11px; font-weight: 700; } +.badge.accepted { background: var(--ok-soft); color: var(--ok); } +.badge.rejected { background: var(--bad-soft); color: var(--bad); } +.badge.none { background: #eee; color: #555; } + +/* ---------- KPI / verdict ---------- */ +.kpi-row { display: flex; gap: 14px; margin: 12px 0; flex-wrap: wrap; } +.kpi { flex: 1; min-width: 130px; background: #fff; border: 1px solid var(--border); + border-radius: 12px; padding: 14px 16px; text-align: center; } +.kpi .v { font-size: 26px; font-weight: 700; } +.kpi .l { font-size: 11px; color: var(--muted); text-transform: uppercase; letter-spacing: .04em; } +.verdict { font-weight: 600; padding: 10px 14px; border-radius: 10px; display: inline-block; } +.verdict.ok { background: var(--ok-soft); color: var(--ok); } +.verdict.bad { background: var(--bad-soft); color: var(--bad); } +.verdict.neutral { background: #f1f1ee; color: #555; } + +/* ---------- workspace tree / viewer ---------- */ +.ws-split { display: grid; grid-template-columns: 300px 1fr; gap: 14px; align-items: start; } +@media (max-width: 860px) { .ws-split { grid-template-columns: 1fr; } } +.tree { border: 1px solid var(--border); border-radius: 10px; padding: 8px; max-height: 560px; overflow: auto; font-size: 13px; } +.tree ul { list-style: none; margin: 0; padding-left: 16px; } +.tree .node-file { cursor: pointer; padding: 1px 6px; border-radius: 6px; white-space: nowrap; } +.tree .node-file:hover { background: var(--accent-soft); } +.tree .node-dir { font-weight: 600; } +.tree .toggle { cursor: pointer; color: var(--muted); margin-right: 2px; } +.viewer { border: 1px solid var(--border); border-radius: 10px; overflow: hidden; } +.viewer-head { display: flex; justify-content: space-between; gap: 8px; padding: 8px 10px; +/* ---------- knowledge ---------- */ +.ktabs { display: flex; gap: 4px; margin-bottom: 12px; flex-wrap: wrap; } +.ktab { border: 1px solid var(--border); background: #fff; border-radius: 8px; + padding: 5px 12px; font-size: 12px; font-weight: 600; color: #555; cursor: pointer; } +.ktab.is-active { background: var(--accent); border-color: var(--accent); color: #fff; } +.kdoc { background: #fdfdfb; border: 1px solid var(--border); border-radius: 10px; + padding: 14px 16px; font-family: var(--mono); font-size: 12px; white-space: pre-wrap; max-height: 480px; overflow: auto; } +.skill-card { border: 1px solid var(--border); border-radius: 10px; padding: 12px 14px; margin-bottom: 12px; } +.skill-card h4 { margin: 0 0 6px; color: var(--accent); } +.skill-card h5 { margin: 8px 0 4px; font-size: 12px; color: var(--muted); text-transform: uppercase; letter-spacing: .04em; } +.skill-card pre { margin: 0; font-family: var(--mono); font-size: 12px; white-space: pre-wrap; + background: #f8f8f6; border-radius: 8px; padding: 8px 10px; max-height: 220px; overflow: auto; } + +/* ---------- how-it-works ---------- */ +.pipeline { display: flex; flex-wrap: wrap; gap: 8px; align-items: stretch; margin: 12px 0; } +.layer { flex: 1; min-width: 160px; background: var(--accent-soft); border: 1px solid #cfd7f8; + border-radius: 10px; padding: 10px 12px; font-size: 13px; } +.layer span { display: block; font-weight: 400; font-size: 12px; color: var(--muted); margin-top: 2px; } +.arrow { align-self: center; font-size: 20px; color: var(--accent); } +.alg { background: #15181d; color: #cfe3d8; font-family: var(--mono); font-size: 12px; + border-radius: 10px; padding: 14px 16px; overflow: auto; } +.glossary td:first-child { font-family: var(--mono); font-weight: 700; } + +.foot { text-align: center; padding: 18px; color: var(--muted); font-size: 12px; } + background: #fafaf8; border-bottom: 1px solid var(--border); align-items: center; } +.viewer-head select { font-size: 12px; } +.viewer-content { margin: 0; padding: 12px; font-family: var(--mono); font-size: 12px; + max-height: 560px; overflow: auto; white-space: pre-wrap; word-break: break-word; } \ No newline at end of file diff --git a/src/wikiskill/webapp/templates/index.html b/src/wikiskill/webapp/templates/index.html new file mode 100644 index 0000000..64e2f82 --- /dev/null +++ b/src/wikiskill/webapp/templates/index.html @@ -0,0 +1,202 @@ + + + + + +Logs stream live; writer events (Wiki Maintainer / Skill + Proposer) and validation gating decisions appear as they happen.
+