wikiskill/prd.md

WikiSkill — PRD (Product Requirements Document)

Source paper: "WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution" (Tang, Rashtchian, Ferng, Tomkins, Juan, Vu — Google Research / Virginia Tech, arXiv:2608.27454v1)

1. Background

Agent skills package specialized procedural knowledge into reusable, filesystem-based resources. Recent methods (Trace2Skill, EvoSkill, SkillOpt) discover skills from agent execution traces, but the insights that guide skill development remain scattered across optimization histories, limiting systematic reuse across iterations.

WikiSkill co-evolves agent skills with a persistent knowledge base (wiki). It separates raw experience, accumulated knowledge, and executable skills, and continuously consolidates experience into the wiki so that subsequent skill updates build on it.

2. Goals

• G1: Implement the full WikiSkill evolution loop (paper Algorithm 1) as a runnable Python framework.
• G2: Faithfully reproduce the paper's three-layer knowledge architecture (raw/, wiki/, skills/).
• G3: Implement the three evolutionary agents: Inference Agent, Wiki Maintainer, and a ReAct-style Wiki-Informed Skill Proposer.
• G4: Implement validation gating with skill-only rollback (the wiki is never rolled back).
• G5: Make the system testable end-to-end without any LLM API key (mock LLM provider), while supporting real providers (OpenAI-compatible APIs) for actual use.

3. Non-goals

• No skill retrieval/triggering: active skills are injected directly into the Inference Agent's prompt (per the paper's setup, which isolates skill quality).
• No re-implementation of baselines (Trace2Skill / EvoSkill / SkillOpt).
• No benchmark harnesses for the paper's exact datasets (LiveMath, SealQA, SpreadsheetBench, OfficeQA, ALFWorld); a simple demo dataset with an identical interface is provided instead.

4. Functional requirements

FR1 — Three-layer workspace

• Raw Layer (raw/): immutable execution traces per iteration (reasoning, tool calls, tool outputs, final answers, pass/fail, prediction vs. ground truth). Never modified.
• Wiki Layer (wiki/): persistent, compounding; contains
  • wiki/patterns/*.md — pattern pages (failure modes / successful strategies + workarounds),
  • wiki/index.md — concise catalog of patterns,
  • wiki/logs.md — evolution log appended by the Wiki Maintainer each iteration,
  • wiki/skill-impact.md — skill impact tracker updated programmatically by the harness after gating (proposal metadata, target skill, unified diff, val score, accept/reject).
• Skills Layer (skills/<name>/): each skill contains SKILL.md (frontmatter: name, description, plus procedural instructions) and PURPOSE.md (mapping to motivating wiki patterns).

FR2 — Evolution loop (paper Algorithm 1)

1. Initialize S0 = ∅, W0 = ∅; baseline validation R_best = R(T_val,0).
2. For k = 1..K (early-stop if R_best == 1.0):
   • Roll out Inference Agent on D_train with skills S_{k-1} (NO wiki access).
   • Sample T_sample,k ⊂ T_train,k (stratified: ≤5 failing, ≤3 passing traces; each trace capped at 15,000 characters).
   • Wiki Maintenance: W'_k ← M_WM(W_{k-1}, T_sample,k) — create/update pattern pages via patch operations, update index, append log.
   • Skill Proposal: P_k ← M_P(W'_k, S_{k-1}, T_train,k) — a ReAct agent that reads the wiki index, skill-impact tracker, outcome summary, and on-demand reads pattern pages and raw traces via a read_file tool. Produces ONE atomic proposal targeting a single skill (create new skill OR patch-based edit).
   • Apply: S'_k = Apply(S_{k-1}, P_k).
   • Validate on D_val.
   • Gate: accept iff R(T_val,k) > R_best (strict improvement); otherwise roll back skills to S_{k-1}. Wiki is never rolled back.
   • Update wiki: append proposal metadata + diff + score + outcome to skill-impact.md.
3. Return final skill set S_K and wiki W_K.

FR3 — Agents

• Inference Agent: LLM agent with environment tools; receives task context + injected active skill contents; multi-turn trajectory.
• Wiki Maintainer: single LLM call per iteration; outputs JSON with create_patterns, update_patterns (patch ops: append/replace/insert_after), update_index (full content), append_log.
• Skill Proposer: multi-turn ReAct agent (paper: ~10–20 turns) with read_file and a final structured proposal (create/patch one skill).

FR4 — Gating and persistence

• Strict-improvement acceptance, rollback of skills only, programmatic skill-impact.md updates.
• Wiki state persists across all iterations and rollbacks.

FR5 — Configurability & evaluation

• Pluggable LLM provider (mock / OpenAI-compatible), configurable K, tools, splits.
• Metric per dataset (accuracy); paired bootstrap significance testing helper (1,000 iterations) as in paper Appendix C.

5. Acceptance criteria

1. pytest passes: unit tests for workspace/patches/gating, and an end-to-end test of the full evolution loop using a scripted mock LLM.
2. A demo run (python -m wikiskill.demo) executes ≥2 evolution iterations, produces a workspace with all three layers populated, and accepts/rolls back proposals correctly.
3. Code mirrors paper notation (S_k, W_k, P_k, T_sample, R_best) and cites Algorithm 1 steps in docstrings.
4. Web UI (ui_design.md): wikiskill serve boots an interactive workbench that can start/cancel evolution runs, stream progress live, browse the three-layer workspace, inspect skills/wiki artifacts, and report test-split evaluation with paired bootstrap. UI covered by Flask test-client tests plus a live-server smoke test.
5. Post-run evaluation: every run reports test-split accuracy for evolved skills vs no-skill baseline with paired-bootstrap significance (paper §4 + App. C).