wikiskill/design.md

WikiSkill — Design Document

Implements: arXiv:2608.27454v1 (WikiSkill). See prd.md for requirements.

1. Architecture overview

wikiskill/
├── pyproject.toml
├── src/wikiskill/
│   ├── __init__.py
│   ├── llm.py            # LLM provider abstraction: MockLLM, OpenAICompatLLM
│   ├── workspace.py      # Three-layer workspace: raw/, wiki/, skills/ + patch engine
│   ├── skills.py         # Skills layer: SKILL.md + PURPOSE.md, snapshot/rollback
│   ├── prompts.py        # System prompts (Inference / Wiki Maintainer / Skill Proposer)
│   ├── agents.py         # InferenceAgent, WikiMaintainer, SkillProposer (ReAct)
│   ├── orchestrator.py   # Algorithm 1: loop, gating, rollback, test evaluation (§4)
│   ├── metrics.py        # accuracy metric + paired bootstrap significance test
│   ├── datasets.py       # Task/Trace dataclasses + demo dataset
│   ├── webapp/           # Web UI (see ui_design.md): runner + Flask api + SPA
│   ├── demo.py           # End-to-end demo run with mock LLM
│   └── cli.py            # `wikiskill demo | evolve | serve` entry point
└── tests/                # unit, end-to-end, and UI tests

2. Core data model (datasets.py)

• Task(x, y, meta) — task instance + ground truth (paper: 𝒟 = {(x_i, y_i)}).
• ToolCall / Trajectory — trace τ = (o1, a1, ..., oT, aT) captured as messages + structured metadata (final answer, correct flag, tool calls and outputs).
• ToolSpec(name, description, fn) — environment tools 𝒰 (bash-like demo tools, read_file for the proposer).

3. Workspace (workspace.py) — the three layers

Workspace(root) manages:

4. LLM abstraction (llm.py)

class LLM(Protocol):
    def chat(self, messages: list[dict], **kw) -> str: ...

• OpenAICompatLLM: calls any OpenAI-compatible /chat/completions endpoint (configurable base_url, api_key via env, model, temperature). Uses urllib (no hard deps).
• MockLLM: deterministic scripted responder for tests/demo. Routes on content:
  • Maintainer JSON → returns a small valid wiki update JSON.
  • Proposer JSON → returns a valid create-skill proposal.
  • Inference → returns a demo answer (extracted from task text).
• Robust JSON extraction helper (extract_json) shared by all agents (strips code fences).

5. Agents (agents.py)

5.1 Inference Agent

Runs one task: system prompt = task-type prompt + injected active skills (full SKILL.md contents, no wiki access — paper §3.2/ablation). Loop: LLM → tool call → observation → ... → final ANSWER: ... line or max turns. Returns Trajectory (messages, tool calls, final answer, correct flag).

5.2 Wiki Maintainer (M_WM)

One LLM call (full-batch; paper Appendix D: B = N_train). Input: current wiki context (index + all pattern pages + logs + skill-impact) and the stratified trace sample. Output JSON parsed into workspace ops: create_patterns[], update_patterns[] (edits[]), update_index, append_log. A stratified sampler picks ≤5 failing + ≤3 passing traces, each truncated to 15,000 chars.

5.3 Wiki-Informed Skill Proposer (M_P) — ReAct

Loop of up to max_react_turns (default 20, paper: ~10–20):

• Initial context: wiki index, skill-impact.md, pass/fail outcome summary (no pre-sampled traces).
• Tools: read_file(path) (workspace-scoped) + finish(...) to emit the final proposal.
• Final output JSON: {"action": "create"|"patch", "skill": name, "content"/"edits": [...], "purpose": mapping to wiki patterns} — atomic, single-skill per paper §3.2.3.

6. Orchestrator (orchestrator.py) — Algorithm 1

EvolutionOrchestrator(llm, tools, dataset, workspace, metric, K, ...).run()

Per iteration k (state S_k, W_k exactly as in the paper):

1. early stop if R_best == 1.0;
2. roll out training tasks with S_{k-1} → raw traces (skill injection, wiki hidden);
3. stratified sample T_sample,k;
4. W'_k = M_WM(W_{k-1}, T_sample,k);
5. P_k = M_P(W'_k, S_{k-1}, T_train,k);
6. snapshot skills; S'_k = Apply(S_{k-1}, P_k);
7. validate on D_val → R(T_val,k);
8. accept iff R > R_best else restore snapshot (skills-only rollback);
9. W_k = Update(W'_k, P_k, R, a_k) → append unified diff + outcome to skill-impact.md;
10. persist run state (history JSON: per-iteration scores, accept/reject, best).

7. Metrics & significance (metrics.py)

• accuracy(traces) — fraction correct.
• paired_bootstrap(mine, base, n=1000, seed) — resample test tasks with replacement, return p-value that margins ≤ 0; used for "best method" claims (paper Appendix C).

8. Demo dataset (datasets.py)

make_demo_dataset(seed): synthetic "unit-conversion" arithmetic tasks. The no-skill agent often errs (rounding/format pitfalls baked into the mock); a good skill (the mock proposer writes one) nudges the inference prompt to compute correctly — so validation gating can genuinely accept/reject. Also pluggable: any Dataset of Tasks with tools + metric.

9. Error handling & robustness

• Malformed LLM JSON → retry once, then skip gracefully (maintainer: no-op update with log entry; proposer: proposal = None, iteration recorded as "no proposal").
• Missing files, bad patch targets → logged, op skipped (never crashes the loop).
• Path traversal blocked in read_file.

10. Testing strategy

• Unit: patch ops (exact/insert_after/append, invalid target), snapshot/restore, path allowlist, immutable raw layer.

• Integration: gating math (accept on strict improvement, reject on equal/lower), skill-impact.md written by harness with unified diff.

• E2E: MockLLM-driven full loop (K=3) — assert artifacts exist, acceptance behavior, early-stop at R_best=1.0.

• Raw layer: add_raw_trace(iter_k, task_id, trace_dict) writes raw/iter_{k}/{task_id}.json. Immutable: an existing file is never rewritten (enforced).

• Wiki layer (wiki/):

  • patterns/ pages: markdown files; updates via a patch engine implementing exactly the three ops from the paper's Maintainer prompt: append, replace (exact-substring target), insert_after (exact-substring target).
  • index.md: full-content rewrite (maintainer always supplies complete index).
  • logs.md: append-only evolution log.
  • skill-impact.md: append-only; the orchestrator (not any agent) writes the programmatic entry after gating: iteration, proposal metadata, target skill, unified diff (difflib.unified_diff of skill content), validation score, accept/reject.

• Skills layer: skills/<name>/SKILL.md + PURPOSE.md.

  • apply_proposal(P): create (new dir with SKILL.md + PURPOSE.md) or patch (patch ops on SKILL.md, mirrored to PURPOSE.md updates).
  • snapshot() / restore(snapshot): cheap rollback of the Skills layer only (dict of name → content; rollback deletes/rewrites files accordingly). Wiki/raw untouched.

• read_file(rel_path) with a path-allowlist (agents may only read within the workspace) — used by the ReAct Skill Proposer.