mirror of
https://github.com/kenhuangus/wikiskill
synced 2026-09-22 23:34:25 +08:00
WikiSkill: reference implementation of arXiv:2608.27454 (Google Research) - three-layer workspace, evolutionary agents, validation gating
This commit is contained in:
DistributedApps.AI
@@ -0,0 +1,140 @@
|
||||
# WikiSkill — Design Document
|
||||
|
||||
Implements: arXiv:2608.27454v1 (WikiSkill). See `prd.md` for requirements.
|
||||
|
||||
## 1. Architecture overview
|
||||
|
||||
```
|
||||
wikiskill/
|
||||
├── pyproject.toml
|
||||
├── src/wikiskill/
|
||||
│ ├── __init__.py
|
||||
│ ├── llm.py # LLM provider abstraction: MockLLM, OpenAICompatLLM
|
||||
│ ├── workspace.py # Three-layer workspace: raw/, wiki/, skills/ + patch engine
|
||||
│ ├── prompts.py # System prompts (Inference / Wiki Maintainer / Skill Proposer)
|
||||
│ ├── agents.py # InferenceAgent, WikiMaintainer, SkillProposer (ReAct)
|
||||
│ ├── orchestrator.py # Algorithm 1: evolution loop, gating, rollback, logging
|
||||
│ ├── metrics.py # accuracy metric + paired bootstrap significance test
|
||||
│ ├── datasets.py # Task/Trace dataclasses + demo dataset
|
||||
│ ├── demo.py # End-to-end demo run with mock LLM
|
||||
│ └── cli.py # `wikiskill evolve` / `wikiskill demo` entry point
|
||||
└── tests/
|
||||
├── test_workspace.py
|
||||
├── test_gating.py
|
||||
└── test_e2e.py
|
||||
```
|
||||
|
||||
## 2. Core data model (`datasets.py`)
|
||||
|
||||
- `Task(x, y, meta)` — task instance + ground truth (paper: 𝒟 = {(x_i, y_i)}).
|
||||
- `ToolCall / Trajectory` — trace `τ = (o1, a1, ..., oT, aT)` captured as messages +
|
||||
structured metadata (final answer, correct flag, tool calls and outputs).
|
||||
- `ToolSpec(name, description, fn)` — environment tools 𝒰 (bash-like demo tools, `read_file`
|
||||
for the proposer).
|
||||
|
||||
## 3. Workspace (`workspace.py`) — the three layers
|
||||
|
||||
`Workspace(root)` manages:
|
||||
|
||||
## 4. LLM abstraction (`llm.py`)
|
||||
|
||||
```python
|
||||
class LLM(Protocol):
|
||||
def chat(self, messages: list[dict], **kw) -> str: ...
|
||||
```
|
||||
|
||||
- `OpenAICompatLLM`: calls any OpenAI-compatible `/chat/completions` endpoint
|
||||
(configurable base_url, api_key via env, model, temperature). Uses `urllib` (no hard deps).
|
||||
- `MockLLM`: deterministic scripted responder for tests/demo. Routes on content:
|
||||
- Maintainer JSON → returns a small valid wiki update JSON.
|
||||
- Proposer JSON → returns a valid create-skill proposal.
|
||||
- Inference → returns a demo answer (extracted from task text).
|
||||
- Robust JSON extraction helper (`extract_json`) shared by all agents (strips code fences).
|
||||
|
||||
## 5. Agents (`agents.py`)
|
||||
|
||||
### 5.1 Inference Agent
|
||||
Runs one task: system prompt = task-type prompt + **injected active skills**
|
||||
(full SKILL.md contents, no wiki access — paper §3.2/ablation). Loop:
|
||||
LLM → tool call → observation → ... → final `ANSWER: ...` line or max turns.
|
||||
Returns `Trajectory` (messages, tool calls, final answer, correct flag).
|
||||
|
||||
### 5.2 Wiki Maintainer (`M_WM`)
|
||||
One LLM call (full-batch; paper Appendix D: B = N_train). Input: current wiki context
|
||||
(index + all pattern pages + logs + skill-impact) and the stratified trace sample.
|
||||
Output JSON parsed into workspace ops:
|
||||
`create_patterns[]`, `update_patterns[] (edits[])`, `update_index`, `append_log`.
|
||||
A **stratified sampler** picks ≤5 failing + ≤3 passing traces, each truncated to 15,000 chars.
|
||||
|
||||
### 5.3 Wiki-Informed Skill Proposer (`M_P`) — ReAct
|
||||
Loop of up to `max_react_turns` (default 20, paper: ~10–20):
|
||||
- Initial context: wiki index, `skill-impact.md`, pass/fail outcome summary (no pre-sampled traces).
|
||||
- Tools: `read_file(path)` (workspace-scoped) + `finish(...)` to emit the final proposal.
|
||||
- Final output JSON: `{"action": "create"|"patch", "skill": name, "content"/"edits": [...],
|
||||
"purpose": mapping to wiki patterns}` — atomic, single-skill per paper §3.2.3.
|
||||
|
||||
## 6. Orchestrator (`orchestrator.py`) — Algorithm 1
|
||||
|
||||
```python
|
||||
EvolutionOrchestrator(llm, tools, dataset, workspace, metric, K, ...).run()
|
||||
```
|
||||
Per iteration k (state S_k, W_k exactly as in the paper):
|
||||
1. early stop if `R_best == 1.0`;
|
||||
2. roll out training tasks with S_{k-1} → raw traces (skill injection, wiki hidden);
|
||||
3. stratified sample T_sample,k;
|
||||
4. `W'_k = M_WM(W_{k-1}, T_sample,k)`;
|
||||
5. `P_k = M_P(W'_k, S_{k-1}, T_train,k)`;
|
||||
6. snapshot skills; `S'_k = Apply(S_{k-1}, P_k)`;
|
||||
7. validate on D_val → `R(T_val,k)`;
|
||||
8. accept iff `R > R_best` else restore snapshot (skills-only rollback);
|
||||
9. `W_k = Update(W'_k, P_k, R, a_k)` → append unified diff + outcome to `skill-impact.md`;
|
||||
10. persist run state (history JSON: per-iteration scores, accept/reject, best).
|
||||
|
||||
## 7. Metrics & significance (`metrics.py`)
|
||||
|
||||
- `accuracy(traces)` — fraction correct.
|
||||
- `paired_bootstrap(mine, base, n=1000, seed)` — resample test tasks with replacement,
|
||||
return p-value that margins ≤ 0; used for "best method" claims (paper Appendix C).
|
||||
|
||||
## 8. Demo dataset (`datasets.py`)
|
||||
|
||||
`make_demo_dataset(seed)`: synthetic "unit-conversion" arithmetic tasks. The no-skill agent
|
||||
often errs (rounding/format pitfalls baked into the mock); a good skill (the mock proposer
|
||||
writes one) nudges the inference prompt to compute correctly — so validation gating can
|
||||
genuinely accept/reject. Also pluggable: any `Dataset` of `Task`s with tools + metric.
|
||||
|
||||
## 9. Error handling & robustness
|
||||
|
||||
- Malformed LLM JSON → retry once, then skip gracefully (maintainer: no-op update with log
|
||||
entry; proposer: proposal = None, iteration recorded as "no proposal").
|
||||
- Missing files, bad patch targets → logged, op skipped (never crashes the loop).
|
||||
- Path traversal blocked in `read_file`.
|
||||
|
||||
## 10. Testing strategy
|
||||
|
||||
- Unit: patch ops (exact/insert_after/append, invalid target), snapshot/restore,
|
||||
path allowlist, immutable raw layer.
|
||||
- Integration: gating math (accept on strict improvement, reject on equal/lower),
|
||||
skill-impact.md written by harness with unified diff.
|
||||
- E2E: MockLLM-driven full loop (K=3) — assert artifacts exist, acceptance behavior,
|
||||
early-stop at R_best=1.0.
|
||||
|
||||
|
||||
- **Raw layer**: `add_raw_trace(iter_k, task_id, trace_dict)` writes
|
||||
`raw/iter_{k}/{task_id}.json`. Immutable: an existing file is never rewritten (enforced).
|
||||
- **Wiki layer** (`wiki/`):
|
||||
- `patterns/` pages: markdown files; updates via a patch engine implementing exactly the
|
||||
three ops from the paper's Maintainer prompt: `append`, `replace` (exact-substring
|
||||
target), `insert_after` (exact-substring target).
|
||||
- `index.md`: full-content rewrite (maintainer always supplies complete index).
|
||||
- `logs.md`: append-only evolution log.
|
||||
- `skill-impact.md`: append-only; the **orchestrator** (not any agent) writes the
|
||||
programmatic entry after gating: iteration, proposal metadata, target skill, unified
|
||||
diff (`difflib.unified_diff` of skill content), validation score, accept/reject.
|
||||
- **Skills layer**: `skills/<name>/SKILL.md` + `PURPOSE.md`.
|
||||
- `apply_proposal(P)`: create (new dir with SKILL.md + PURPOSE.md) or patch (patch ops on
|
||||
SKILL.md, mirrored to PURPOSE.md updates).
|
||||
- `snapshot()` / `restore(snapshot)`: cheap rollback of the Skills layer only (dict of
|
||||
name → content; rollback deletes/rewrites files accordingly). Wiki/raw untouched.
|
||||
- `read_file(rel_path)` with a path-allowlist (agents may only read within the workspace)
|
||||
— used by the ReAct Skill Proposer.
|
||||
Reference in New Issue
Block a user