WikiSkill: reference implementation of arXiv:2608.27454 (Google Research) - three-layer workspace, evolutionary agents, validation gating

2026-09-22 23:34:25 +08:00 · 2026-08-30 23:07:08 -04:00
commit 0f2df89097
22 changed files with 1596 additions and 0 deletions
@@ -0,0 +1,99 @@
+# WikiSkill — Implementation
+
+Python implementation of **"WikiSkill: Compiling Agent Experience into Persistent
+Knowledge for Skill Evolution"** (arXiv:2608.27454v1, Tang et al., Google Research).
+
+Docs: see [`prd.md`](prd.md) (requirements), [`design.md`](design.md) (architecture),
+[`plan.md`](plan.md) (build plan).
+
+## Credits & Attribution
+
+This project is an independent, open-source reimplementation of the research paper:
+
+> **WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution**
+> Liyan Tang, Cyrus Rashtchian, Chun-Sung Ferng, Andrew Tomkins, Da-Cheng Juan, Tu Vu —
+> **Google Research** (and Virginia Tech).
+> arXiv:2608.27454v1 [cs.AI], 27 Aug 2026. License: CC BY 4.0.
+> Paper: https://arxiv.org/abs/2608.27454
+
+All credit for the WikiSkill **method, algorithm (Algorithm 1), three-layer knowledge
+architecture, agent designs, experimental findings, and agent prompts (Appendix E)**
+belongs to the Google Research authors. This repository reimplements the framework in
+code for study/experimentation purposes and is **not** an official Google product.
+Agent system prompts in [`src/wikiskill/prompts.py`](src/wikiskill/prompts.py) are
+condensed adaptations of those published in the paper (CC BY 4.0); adaptations in this
+repo are released under the same spirit — please cite the original paper if you use
+this work:
+
+```bibtex
+@article{tang2026wikiskill,
+  title   = {WikiSkill: Compiling Agent Experience into Persistent Knowledge
+             for Skill Evolution},
+  author  = {Tang, Liyan and Rashtchian, Cyrus and Ferng, Chun-Sung and
+             Tomkins, Andrew and Juan, Da-Cheng and Vu, Tu},
+  journal = {arXiv preprint arXiv:2608.27454},
+  year    = {2026},
+  note    = {Google Research}
+}
+```
+
+
+## What it implements
+
+- **Three-layer workspace** (paper §3.1): immutable `raw/` traces, persistent `wiki/`
+  (patterns + index + logs + skill-impact tracker), active `skills/` (SKILL.md + PURPOSE.md).
+- **Evolutionary loop** (paper Algorithm 1, §3.2): Inference Agent rollouts with skill
+  injection but **no wiki access**; Wiki Maintainer consolidation (create/update pattern
+  pages via `append`/`replace`/`insert_after` patches); a **ReAct Skill Proposer** that
+  reads the wiki and raw traces on demand and emits one atomic skill proposal.
+- **Gating & rollback** (§3.2.4): accept iff validation score strictly improves `R_best`;
+  otherwise roll back **skills only** — the wiki is never rolled back. The harness
+  programmatically appends proposal + unified diff + outcome to `wiki/skill-impact.md`.
+- **Stratified trace sampling** (App. C): ≤5 failing + ≤3 passing traces, 15,000-char cap.
+- **Paired bootstrap significance testing** (App. C) in `metrics.py`.
+- Early stop when `R_best == 1.0`.
+
+## Quickstart
+
+```bash
+pip install -e ".[dev]"
+pytest                                  # 14 tests incl. end-to-end loop
+python -m wikiskill.demo                # offline demo with the deterministic MockLLM
+```
+
+Demo output: a `wikiskill_demo_workspace/` with `raw/`, `wiki/`, `skills/` populated,
+accepted/rejected proposals logged in `wiki/skill-impact.md`, and `run_state.json`.
+
+## Using a real LLM
+
+```bash
+wikiskill evolve --tasks tasks.jsonl \
+  --train-ids train.txt --val-ids val.txt --test-ids test.txt \
+  --model <model-name> --base-url https://your-endpoint/v1
+```
+
+API key is read from `--api-key`, `WIKISKILL_API_KEY`, or `OPENAI_API_KEY`.
+Any OpenAI-compatible `/chat/completions` endpoint works (including local vLLM, as in
+the paper). `tasks.jsonl` lines: `{"id", "x", "y", "meta?"}`.
+
+## Layout
+
+```
+src/wikiskill/
+├── workspace.py    # three layers + patch engine + scoped read_file
+├── skills.py       # skills layer, apply_proposal, skills-only snapshot/rollback
+├── agents.py       # InferenceAgent, WikiMaintainer, ReAct SkillProposer, stratified_sample
+├── orchestrator.py # Algorithm 1 with gating/rollback + audit trail
+├── prompts.py      # agent system prompts (adapted from paper Appendix E)
+├── llm.py          # LLM protocol, OpenAI-compatible client, MockLLM, extract_json
+├── metrics.py      # accuracy + paired bootstrap (1,000 iterations)
+├── datasets.py     # dataclasses + synthetic demo dataset
+├── demo.py / cli.py
+└── tests/…         # unit + end-to-end tests
+```
+
+## Notes & limitations (mirroring the paper)
+
+- Skills are injected directly into the prompt (no retrieval), matching the paper's setup.
+- Gating requires strict improvement; neutral proposals are rejected (paper Limitations).
+- The wiki accumulates without pruning (paper Limitations).