源码快照 @ 2026-09-01 (source-only, 剔除非源码素材)

2026-09-01 19:05:14 +08:00
commit 7be535059e
3358 changed files with 1306357 additions and 0 deletions
@@ -0,0 +1,90 @@
+task_id,maka,codex,claude-code,reasonix,maka_failure_class,codex_failure_class,claude-code_failure_class,reasonix_failure_class
+adaptive-rejection-sampler,passed,passed,failed,unscored,,,budget_exhausted,infra_failed
+bn-fit-modify,passed,passed,failed,passed,,,verification_failed,
+break-filter-js-from-html,passed,passed,passed,passed,,,,
+build-cython-ext,passed,passed,passed,passed,,,,
+build-pmars,passed,passed,passed,passed,,,,
+build-pov-ray,passed,passed,passed,passed,,,,
+caffe-cifar-10,passed,passed,passed,passed,,,,
+cancel-async-tasks,failed,passed,failed,passed,verification_failed,,budget_exhausted,
+chess-best-move,failed,failed,failed,failed,verification_failed,budget_exhausted,budget_exhausted,budget_exhausted
+circuit-fibsqrt,passed,passed,passed,failed,,,,verification_failed
+cobol-modernization,passed,passed,passed,passed,,,,
+code-from-image,failed,passed,failed,passed,budget_exhausted,,budget_exhausted,
+compile-compcert,passed,passed,passed,passed,,,,
+configure-git-webserver,failed,passed,passed,passed,verification_failed,,,
+constraints-scheduling,passed,passed,passed,passed,,,,
+count-dataset-tokens,passed,passed,passed,failed,,,,verification_failed
+crack-7z-hash,passed,passed,passed,failed,,,,budget_exhausted
+custom-memory-heap-crash,passed,passed,passed,passed,,,,
+db-wal-recovery,passed,passed,failed,passed,,,budget_exhausted,
+distribution-search,passed,passed,passed,passed,,,,
+dna-assembly,failed,passed,failed,failed,verification_failed,,budget_exhausted,verification_failed
+dna-insert,failed,passed,passed,failed,verification_failed,,,verification_failed
+extract-elf,passed,failed,failed,failed,,verification_failed,verification_failed,verification_failed
+extract-moves-from-video,failed,failed,failed,failed,budget_exhausted,budget_exhausted,budget_exhausted,budget_exhausted
+feal-differential-cryptanalysis,passed,failed,passed,passed,,verification_failed,,
+feal-linear-cryptanalysis,passed,passed,passed,passed,,,,
+filter-js-from-html,failed,failed,failed,failed,verification_failed,verification_failed,verification_failed,verification_failed
+financial-document-processor,passed,passed,passed,passed,,,,
+fix-code-vulnerability,passed,passed,passed,passed,,,,
+fix-git,failed,passed,passed,passed,verification_failed,,,
+fix-ocaml-gc,passed,passed,passed,passed,,,,
+gcode-to-text,failed,failed,failed,failed,budget_exhausted,budget_exhausted,budget_exhausted,budget_exhausted
+git-leak-recovery,passed,passed,passed,passed,,,,
+git-multibranch,passed,passed,passed,passed,,,,
+gpt2-codegolf,failed,failed,failed,failed,budget_exhausted,budget_exhausted,budget_exhausted,budget_exhausted
+headless-terminal,passed,passed,passed,passed,,,,
+hf-model-inference,passed,passed,passed,passed,,,,
+install-windows-3.11,passed,passed,passed,failed,,,,verification_failed
+kv-store-grpc,passed,failed,passed,failed,,verification_failed,,verification_failed
+large-scale-text-editing,passed,passed,passed,passed,,,,
+largest-eigenval,failed,passed,failed,passed,budget_exhausted,,budget_exhausted,
+llm-inference-batching-scheduler,passed,passed,passed,passed,,,,
+log-summary-date-ranges,passed,passed,passed,passed,,,,
+mailman,passed,passed,passed,passed,,,,
+make-doom-for-mips,failed,failed,failed,failed,budget_exhausted,budget_exhausted,budget_exhausted,budget_exhausted
+make-mips-interpreter,failed,passed,failed,failed,verification_failed,,budget_exhausted,budget_exhausted
+mcmc-sampling-stan,passed,passed,passed,failed,,,,verification_failed
+merge-diff-arc-agi-task,passed,passed,passed,passed,,,,
+model-extraction-relu-logits,passed,passed,failed,passed,,,budget_exhausted,
+modernize-scientific-stack,passed,passed,passed,passed,,,,
+mteb-leaderboard,failed,passed,passed,passed,verification_failed,,,
+mteb-retrieve,passed,passed,passed,failed,,,,verification_failed
+multi-source-data-merger,passed,passed,passed,passed,,,,
+nginx-request-logging,passed,passed,passed,passed,,,,
+openssl-selfsigned-cert,passed,passed,passed,passed,,,,
+overfull-hbox,passed,passed,passed,failed,,,,budget_exhausted
+password-recovery,passed,passed,failed,passed,,,budget_exhausted,
+path-tracing,passed,passed,failed,passed,,,budget_exhausted,
+path-tracing-reverse,failed,failed,failed,failed,budget_exhausted,budget_exhausted,budget_exhausted,budget_exhausted
+polyglot-c-py,passed,passed,passed,failed,,,,verification_failed
+polyglot-rust-c,passed,passed,passed,passed,,,,
+portfolio-optimization,passed,passed,passed,passed,,,,
+protein-assembly,passed,failed,failed,passed,,verification_failed,budget_exhausted,
+prove-plus-comm,passed,passed,passed,passed,,,,
+pypi-server,passed,passed,passed,failed,,,,verification_failed
+pytorch-model-cli,failed,failed,failed,passed,verification_failed,verification_failed,verification_failed,
+pytorch-model-recovery,failed,passed,passed,unscored,verification_failed,,,plumbing_failed
+qemu-alpine-ssh,passed,failed,passed,failed,,verification_failed,,budget_exhausted
+qemu-startup,passed,passed,passed,failed,,,,verification_failed
+query-optimize,passed,passed,passed,failed,,,,verification_failed
+raman-fitting,failed,failed,failed,failed,budget_exhausted,budget_exhausted,budget_exhausted,budget_exhausted
+regex-chess,passed,failed,failed,unscored,,verification_failed,budget_exhausted,infra_failed
+regex-log,passed,passed,passed,passed,,,,
+reshard-c4-data,passed,passed,passed,passed,,,,
+rstan-to-pystan,passed,passed,passed,passed,,,,
+sam-cell-seg,passed,passed,failed,passed,,,verification_failed,
+sanitize-git-repo,passed,failed,passed,passed,,verification_failed,,
+schemelike-metacircular-eval,passed,passed,failed,failed,,,budget_exhausted,verification_failed
+sparql-university,passed,passed,passed,passed,,,,
+sqlite-db-truncate,passed,passed,passed,passed,,,,
+sqlite-with-gcov,passed,passed,passed,passed,,,,
+torch-pipeline-parallelism,failed,passed,failed,failed,budget_exhausted,,verification_failed,budget_exhausted
+torch-tensor-parallelism,failed,failed,failed,failed,verification_failed,budget_exhausted,verification_failed,verification_failed
+train-fasttext,failed,failed,failed,failed,budget_exhausted,budget_exhausted,budget_exhausted,budget_exhausted
+tune-mjcf,passed,passed,failed,passed,,,budget_exhausted,
+video-processing,failed,failed,failed,failed,verification_failed,verification_failed,verification_failed,verification_failed
+vulnerable-secret,passed,passed,passed,passed,,,,
+winning-avg-corewars,passed,passed,failed,passed,,,budget_exhausted,
+write-compressor,failed,passed,failed,failed,budget_exhausted,,budget_exhausted,verification_failed
@@ -0,0 +1,267 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Terminal-Bench 2.1 — DeepSeek V4 Flash: Maka vs Codex vs Claude Code vs Reasonix
+
+This report compares four agent harnesses around the same DeepSeek V4 Flash model on all 89 Terminal-Bench 2.1 tasks. Every arm ran inside the task container under the same executor, the same task-native deadline policy, and the same package mirror, so the only variable that moves between arms is the harness itself.
+
+**Run id:** `deepseek-v4-flash-4arm-tbench-2.1-full-v1`
+
+**Metric:** end-to-end pass@1 by the official task verifier
+
+**Status:** `completed_with_gaps` — 353/356 cells model-scored; the three unscored cells are Reasonix infrastructure and plumbing failures that survived the retry policy
+
+**Per-task outcomes:** [`terminal-bench-2.1-deepseek-v4-flash-four-arm.csv`](./terminal-bench-2.1-deepseek-v4-flash-four-arm.csv)
+
+The experiment was tracked in [issue #1970](https://github.com/maka-agent/maka-agent/issues/1970).
+
+## TL;DR
+
+- **Codex passed 70/89 (78.65%), Maka 65/89 (73.03%), Claude Code 57/89 (64.04%), Reasonix 55/89 (61.80%).**
+- **The Codex–Maka gap is not statistically significant.** Of 19 discordant pairs, Codex won 12 and Maka 7; an exact two-sided McNemar test gives **p = 0.359**. Only two of the six pairwise comparisons reach significance: Codex over Claude Code (p = 0.0072) and Codex over Reasonix (p = 0.0026).
+- **An independent earlier run agrees, including on the non-significance.** Over the same 89 tasks it scored Codex 67, Maka 60, Claude Code 57, with Codex–Maka at p = 0.210. Codex is the one arm configured identically in both runs, and **19% of its tasks changed outcome between them while its net score moved by 3** — that is the noise floor any five-task claim has to clear.
+- **Maka is the cheapest arm per pass at $0.03215**, ahead of Codex ($0.03339), Reasonix ($0.04051), and Claude Code ($0.04800), and it is the fastest arm at the median cell (332.8 s against Codex's 411.7 s).
+- On the 60-task subset where no arm exhausted its budget, the ordering inverts at the top: **Claude Code 90.00%, Codex 88.33%, Maka 86.67%, Reasonix 75.00%.** Claude Code has the highest conditional pass rate and the worst overall score, because it exhausted its budget on 24/89 tasks (26.97%) — more than double any other arm.
+- Three of the four arms are close in solution quality once they finish. The headline spread is driven mostly by whether an arm finishes inside its deadline, not by whether it can solve the task.
+
+## Results
+
+End-to-end pass@1 is the primary result. Budget-exhausted cells remain scored failures in its denominator.
+
+| Arm | Pass@1 | Passed / evaluated | Budget exhausted | Verification failed | Unscored |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Codex | **78.65%** | 70 / 89 | 9 | 10 | 0 |
+| Maka | **73.03%** | 65 / 89 | 11 | 13 | 0 |
+| Claude Code | **64.04%** | 57 / 89 | 24 | 8 | 0 |
+| Reasonix | **61.80%** | 55 / 89 | 13 | 18 | 3 |
+
+Reasonix is the only arm with unscored cells: two infrastructure failures and one plumbing failure survived the retry policy. Its pass@1 is reported over all 89 tasks; over the 86 it actually completed the rate is 63.95%. The other three arms recorded zero infrastructure failures.
+
+## Pairwise significance
+
+Each comparison uses the exact two-sided McNemar test over discordant task pairs, treating the 89 benchmark tasks as the paired units.
+
+| Comparison | A-only passes | B-only passes | Discordant | p |
+| --- | ---: | ---: | ---: | ---: |
+| Codex vs Reasonix | 19 | 4 | 23 | **0.0026** |
+| Codex vs Claude Code | 17 | 4 | 21 | **0.0072** |
+| Maka vs Reasonix | 17 | 7 | 24 | 0.0639 |
+| Maka vs Claude Code | 13 | 5 | 18 | 0.0963 |
+| Maka vs Codex | 7 | 12 | 19 | 0.3593 |
+| Claude Code vs Reasonix | 15 | 13 | 28 | 0.8506 |
+
+Only the two Codex comparisons clear the 0.05 threshold. **The five-task Codex lead over Maka is within what this suite's noise can produce**: the paired outcomes are 58 shared passes, 7 Maka-only, 12 Codex-only, and 12 shared failures. Maka's advantages over Claude Code and Reasonix point in a consistent direction but do not reach significance either.
+
+This is evidence about one frozen run over the fixed 89-task Terminal-Bench 2.1 suite. It is not proof of a universal ordering on other task distributions, other models, or repeated runs.
+
+## Exclusive outcomes
+
+**Maka only, 7:** `extract-elf`, `feal-differential-cryptanalysis`, `kv-store-grpc`, `protein-assembly`, `qemu-alpine-ssh`, `regex-chess`, `sanitize-git-repo`
+
+**Codex only, 12:** `cancel-async-tasks`, `code-from-image`, `configure-git-webserver`, `dna-assembly`, `dna-insert`, `fix-git`, `largest-eigenval`, `make-mips-interpreter`, `mteb-leaderboard`, `pytorch-model-recovery`, `torch-pipeline-parallelism`, `write-compressor`
+
+The two sets differ in kind, not only in size. On all 7 tasks Maka won alone, Codex produced a candidate and the verifier rejected it — none was a deadline loss. On the 12 tasks Codex won alone, Maka's failures split **8 verification failures against 4 budget exhaustions**. Two thirds of the gap is therefore wrong answers rather than unfinished work, which is the opposite of what the per-request timing shape below would predict on its own.
+
+The complete accepted pass/fail outcome and failure class for every task is in the adjacent [CSV](./terminal-bench-2.1-deepseek-v4-flash-four-arm.csv); prompts, payloads, traces, and verifier output are not committed.
+
+## Diagnostic decomposition
+
+These metrics diagnose where the observed gaps appear. Neither replaces end-to-end pass@1.
+
+| Diagnostic | Maka | Codex | Claude Code | Reasonix |
+| --- | ---: | ---: | ---: | ---: |
+| Non-budget conditional pass rate | 52/60 (86.67%) | 53/60 (88.33%) | **54/60 (90.00%)** | 45/60 (75.00%) |
+| Budget exhaustion rate | 11/89 (12.36%) | 9/89 (10.11%) | **24/89 (26.97%)** | 13/89 (14.61%) |
+
+The conditional denominator excludes any task where *any* arm exhausted its budget, leaving the same 60 tasks for all four. It is not an "unlimited-time pass rate": the remaining tasks still ran under their original task-native deadlines, and the excluded set is not random — it is enriched for hard tasks.
+
+The two rows tell different stories. On the conditional subset, Maka, Codex, and Claude Code sit within 3.33 percentage points of each other, and the ordering of the top three reverses relative to the headline. What separates them in the headline is the second row: Claude Code exhausts its budget on more than a quarter of the suite, over twice the rate of Codex.
+
+These observations decompose the gap; this run does not establish their causes.
+
+## Outcome-normalized economics
+
+Cost per pass normalizes recorded usage by successful benchmark outcomes. It includes spending on scored failures.
+
+| Arm | Total cost | Passed | Cost per pass |
+| --- | ---: | ---: | ---: |
+| Maka | $2.0895 | 65 | **$0.03215** |
+| Codex | $2.3376 | 70 | $0.03339 |
+| Reasonix | $2.2281 | 55 | $0.04051 |
+| Claude Code | $2.7363 | 57 | $0.04800 |
+
+Maka is the cheapest per pass despite ranking second on pass@1. Its margin over Codex is 3.7%; the gap to Claude Code is 49%. No equivalence test was performed, so these are descriptive point estimates from a single run.
+
+The resource footprint behind that result:
+
+| Resource diagnostic | Maka | Codex | Claude Code | Reasonix |
+| --- | ---: | ---: | ---: | ---: |
+| Input tokens | 181,448,639 | 295,168,443 | 244,477,053 | 240,741,904 |
+| Cached input tokens | 177,684,352 | 291,031,168 | 240,396,288 | 237,451,776 |
+| Cache-hit share of input | 97.93% | 98.60% | 98.33% | 98.63% |
+| Output tokens | 3,546,226 | 3,081,715 | 4,991,091 | 3,663,490 |
+
+Maka reaches 93% of Codex's score on **61% of its input tokens**. Because DeepSeek prices cache hits at 2% of uncached input, that difference is a smaller share of the bill than of the token count, but it is the mechanism behind Maka's lower total spend at a higher output volume.
+
+Usage was metered by a per-cell host proxy that parses the provider's SSE stream. Clients that close the connection after receiving the terminal event are common — 40% of Reasonix requests end this way — and their usage is retained. Across the run, 2 of 412 Reasonix requests are the only ones without recorded usage.
+
+Costs are cache-aware API-equivalent estimates from the pricing identity frozen in the manifest: $0.145 per million uncached input tokens, $0.0029 per million cache-hit input tokens, and $0.29 per million output tokens. They are not a billing invoice.
+
+## Per-request shape
+
+The four harnesses differ sharply in how they spend a task budget.
+
+All four columns come from the host proxy's per-request telemetry, so they are mutually comparable.
+
+| Arm | Requests per cell | Output tokens per request | Reasoning tokens per request | Seconds per request |
+| --- | ---: | ---: | ---: | ---: |
+| Codex | 44.3 | 781 | 543 | 9.1 |
+| Reasonix | 41.6 | 1,000 | 745 | 11.5 |
+| Maka | 37.4 | 1,064 | 778 | 10.7 |
+| Claude Code | 32.8 | 1,708 | not itemized | 18.2 |
+
+Codex takes the most steps and makes each one the smallest; Claude Code takes the fewest and makes each one the largest and slowest. Maka sits between them, spending 1.43× more reasoning per request than Codex. Anthropic's protocol does not itemize reasoning tokens, so Claude Code's reasoning column is empty rather than zero.
+
+At the cell level this produces a distinctive duration profile:
+
+| Arm | Median cell | Mean cell | p90 cell | Median share of task deadline consumed |
+| --- | ---: | ---: | ---: | ---: |
+| Maka | **332.8 s** | 693.1 s | 1800.0 s | 0.309 |
+| Codex | 411.7 s | 582.8 s | 1334.9 s | 0.303 |
+| Reasonix | 428.7 s | 658.2 s | 1577.5 s | 0.363 |
+| Claude Code | 544.7 s | 840.2 s | 1800.6 s | 0.416 |
+
+Maka has the fastest median cell and the second-slowest mean. The two statistics are not in conflict: on the typical task Maka finishes first and consumes essentially the same share of the deadline as Codex (0.309 against 0.303), and the mean is pulled up by a heavier tail of cells that run to the deadline.
+
+Under a hard deadline this shape has consequences at the tail. On `write-compressor`, whose task-native agent budget is 900 s, Maka spent 890 s inside the model with only 6 s in tools and exhausted the budget after 20 requests; Codex finished the same task in 540 s across 66 requests. Denser steps leave less room to recover from a wrong turn.
+
+## Cross-run stability
+
+An earlier three-arm run over the same 89 tasks and the same model (`deepseek-v4-flash-3arm-tbench-2.1-full-v7`) provides one independent repetition for Codex, Maka, and Claude Code. It predates this branch: Maka ran host-side, no package mirror was applied, and its Claude Code arm lost 17 cells to infrastructure failures. Only the Codex arm is configured identically in both runs.
+
+| Arm | Three-arm run | Four-arm run | Tasks flipping outcome |
+| --- | ---: | ---: | ---: |
+| Codex | 67/89 (75.28%) | 70/89 (78.65%) | 17 (19.10%) |
+| Maka | 60/89 (67.42%) | 65/89 (73.03%) | 19 (21.35%) |
+| Claude Code | 57/89 (64.04%) | 57/89 (64.04%) | 18 (20.22%) |
+
+The Codex row is the interpretable one, and it is the most useful number in this report. Its configuration did not change and it recorded no infrastructure failures in either run, yet **19% of tasks changed outcome between the two runs while the net score moved by 3**. Roughly a fifth of this suite is decided by run-to-run variation rather than by any property of the harness under test.
+
+That is the scale against which the five-task Codex–Maka gap has to be read, and it agrees with the significance tests: the same comparison in the three-arm run gives 15 Codex-only against 8 Maka-only, **p = 0.210** — again not significant, again in Codex's favour. Two independent full runs put Codex nominally ahead of Maka and neither can distinguish them.
+
+The Maka and Claude Code rows cannot be read the same way. Maka's placement and package sourcing both changed between the runs, so its +5 confounds those changes with variance. Claude Code's identical headline conceals a different denominator: 17 of its three-arm cells never scored.
+
+## What this run establishes about Maka
+
+Stated as claims this run supports, with the evidence attached to each.
+
+1. **Maka is competitive with Codex on solution quality, and this suite cannot separate them.** 65 against 70 passes, p = 0.359, against a measured 19% cross-run flip rate. Two independent full runs both put Codex nominally ahead and neither reaches significance.
+2. **Maka is the most economical arm per successful task.** $0.03215 per pass, the lowest of the four, on 61% of Codex's input tokens. Codex buys its extra five passes at 12% higher total spend.
+3. **Maka is the fastest arm at the median cell.** 332.8 s against 411.7 s for Codex, 428.7 s for Reasonix, and 544.7 s for Claude Code, while consuming the same median share of the task deadline as Codex.
+4. **Maka's execution was clean.** Zero infrastructure failures, zero retry admissions, and zero unscored cells across all 89 tasks — matched only by Codex. Claude Code required one retry and Reasonix five, three of which still failed.
+5. **When Maka wins, it wins on correctness.** All 7 tasks Maka passed and Codex failed were Codex verification failures, not Codex deadline losses.
+
+What this run does **not** establish: that Maka is better than Codex, that any of these margins would survive a repetition, or that the mechanisms above cause the outcomes. Every claim here is a description of one frozen run.
+
+## Where Maka improves next
+
+The failure decomposition points somewhere different from where the timing story alone would point.
+
+**First, solution correctness on tasks Codex gets right.** Of the 12 tasks Codex passed and Maka did not, **8 were verification failures and only 4 were deadline losses**. Maka produced a candidate, submitted it, and the official verifier rejected it. That is the larger of the two buckets and it is not a speed problem. The tasks are listed under [Exclusive outcomes](#exclusive-outcomes) and each has a complete trace in the archived run; a per-task read of those 8 is the highest-value next investigation this dataset supports.
+
+**Second, reasoning volume per step under a hard deadline.** Maka spends 778 reasoning tokens per request against Codex's 543, and takes fewer, denser steps. On the four tasks Maka lost to the deadline this is decisive — `write-compressor` spent 99% of its 900 s inside the model. The open question is causal and is not answered here: the same model at the same `max` effort produces substantially more reasoning text under Maka's scaffold than under Codex's, and two candidate explanations (loop repetition and context growth) were checked against the three-arm traces and ruled out. Tracking continues in [issue #1970](https://github.com/maka-agent/maka-agent/issues/1970).
+
+**Third, deadline awareness as a policy, not a timeout.** Maka's median cell consumes 0.309 of its task deadline, so the budget is not generally tight; the 11 exhausted cells are a tail, not a trend. A harness that could detect it was in that tail and shift toward cheaper, more frequent verification steps would convert deadline losses into scored attempts without changing the typical case. This run measures the tail but does not evaluate any such policy.
+
+**Fourth, more repetitions before any of these are treated as settled.** The 19% cross-run flip rate means a single run cannot validate a change worth fewer than roughly ten tasks. Any of the above should be measured with repetitions, not with one more full run.
+
+## Frozen setup
+
+| Dimension | Value |
+| --- | --- |
+| Benchmark | Terminal-Bench 2.1, revision `d49e28f1e4ddd13d289e85a5f312a66750951932`; all 89 tasks |
+| Task-source fingerprint | `sha256:456826aa4c47ed309716c964c96d2a3acc998764ebc84f3e8449c807d74bd4e7` |
+| Run fingerprint | `sha256:869d39bb82d61462afabec5ccc6d9560c9aa19287041a1d46f6c5d7c66fd3fb4` |
+| Toolchain fingerprint | `sha256:7b21118862eb823a7af9c7d90ead64ef459f8a353b7aebae1b164dde6166eb60` |
+| Model | `deepseek-v4-flash` through the DeepSeek provider on all four arms |
+| Reasoning effort | `max` on all four arms |
+| Repetitions | 1 |
+| Metric | Paired pass@1 by the official verifier |
+| Deadline policy | Task-native agent timeout ×1; 30 s agent settlement grace; 900 s outer setup and teardown grace |
+| Task-native budgets | 900 s on 48 tasks, 1800 s on 17, 3600 s on 13, 1200 s on 5, 2400 s on 2, and one each at 600 s, 750 s, 7200 s, and 12000 s |
+| Attempt policy | One accepted model attempt per arm/task; only infrastructure-invalid admissions may be replaced |
+| Placement | All four arms execute inside the task container |
+| Pair execution | 6 task groups concurrently, arms parallel within a group, at most 24 cells concurrently |
+| Task order | `sha256-rank-v1` over seed `terminal-bench-2.1:deepseek-v4-flash:harness-comparison:v1` |
+| Package mirror | `archive.ubuntu.com` and `security.ubuntu.com` redirected to `mirrors.tencentyun.com` for every arm |
+| Billing mode | Metered, using the frozen DeepSeek V4 Flash pricing identity above |
+| Maka arm | `maka_agent:MakaAgent`; `openai-chat`; external system prompt `default-headless`; continuation off; active and stale tool-result prune at a 2,048 estimated-token threshold; semantic compact off; `harbor-local` isolation |
+| Codex arm | `codex_agent:MakaCodexAgent` 0.146.0; `openai-responses`; external system prompt none; `container-full-access` permissions |
+| Claude Code arm | `claude_code_agent:MakaClaudeCodeAgent` 2.1.220; `anthropic-messages`; external system prompt none; `bypassPermissions` |
+| Reasonix arm | `reasonix_agent:MakaReasonixAgent` 1.19.4; `openai-chat`; `stream-json` output; external system prompt none; `auto` permissions |
+
+Redirecting apt is a variance-control measure, not a performance aid. The same command took 16 s or 300 s from the same host depending on which upstream path it landed on, and that noise is charged against whichever agent happened to need more packages. The mirror address enters the manifest fingerprint, so runs with and without it are distinct experiments.
+
+This is a same-model harness comparison, not an isolated model evaluation and not a same-system/same-tool ablation. The arms retain their native agent instructions, tool behavior, context management, and execution loops. Two asymmetries are worth naming explicitly: Maka is the only arm carrying an external system prompt (`default-headless`, where the others send none), and Maka is the only arm running a context-budget tool-result prune policy. The observed differences are attributable to the compared harness systems as a whole; this run does not rank individual harness differences as causes.
+
+## Outcome accounting
+
+The controller WAL records **362 Agent admissions** against 356 scheduled cells: 356 initial admissions plus six infrastructure retries. Those retries covered one Claude Code cell (`custom-memory-heap-crash`, which then completed) and five Reasonix cells (`cancel-async-tasks`, `filter-js-from-html`, and `fix-ocaml-gc` completed; `adaptive-rejection-sampler` and `regex-chess` failed again). No cell received a third admission. Reasonix `pytorch-model-recovery` failed at the plumbing layer and was not retried.
+
+Maka and Codex each received exactly 89 admissions with zero retries and zero infrastructure failures.
+
+The final ledger is 353 `task_completed`, 8 `task_infra_failed`, and 1 `task_plumbing_failed` events, of which the three surviving failures are the unscored cells reported above.
+
+## Limitations
+
+- Two runs of this suite exist, but only the Codex arm is configured identically across both; the comparison in this report rests on one run of the four-arm configuration. Given the 19% cross-run flip rate measured above, differences of this size should not be treated as settled by either run alone.
+- The task-native deadline is a material parameter, not a neutral one. It binds hardest on Claude Code (26.97% exhaustion) and would likely change the ordering if relaxed.
+- Reasonix contributes three unscored cells, so its comparisons rest on 86 shared tasks rather than 89.
+- Maka carries an external system prompt and a tool-result prune policy that the other three arms do not. This is a documented live harness asymmetry, not a controlled variable, and the run has no prune-off or prompt-off control.
+- Cost per pass uses provider list pricing applied to metered usage, not billed invoices.
+- Pairwise McNemar tests are reported without multiple-comparison correction. Applying a Bonferroni correction across the six comparisons moves the threshold to 0.0083; both Codex results survive it (0.0026 and 0.0072) and no other comparison approaches it.
+- Per-request telemetry and the harness token summary disagree on Codex's total reasoning volume (2.14M versus 1.57M tokens); the two agree for Maka. The cause was not identified, so this report makes no cross-source total-volume claim and compares per-request figures only within the telemetry source.
+- No external Oracle registry snapshot was configured, so Oracle annotations are missing for all 89 tasks. The official Terminal-Bench verifier remains the scoring authority.
+
+## Integrity
+
+SHA-256 hashes of the frozen local evidence and the committed outcome projection:
+
+| Source | SHA-256 |
+| --- | --- |
+| `harness-ab-manifest.json` | `aaa3099b365cff01658405d0fc0777aa3c719e8e4cb73627d23bfc124767e3cc` |
+| `harness-cohort-report.json` | `52dff9d3212cbcc9bc927dd0cb280aded72f757e310e75de77f27242e130087e` |
+| `harness-cohort-report.csv` | `bd04d3d0bbb3c2e9d0594d061e1ffc4cd1ba0706e21ddad9f95057415f7d3c2e` |
+| Full run archive `four-arm-full-v1.tgz` | `0816483e35ae2cb85d283c7ee8d74af7e929d8b4898eca4cce877a1341832e95` |
+| Committed outcome CSV | `d0a65eed5c3f8803271819797b292b39c4deaf59178ffbc21595299b740ccfc7` |
+
+## Artifact pointers
+
+Local, git-excluded, under `maka-eval/experiments/harness-ab/terminal-bench-2.1/`:
+
+| Artifact | Path |
+| --- | --- |
+| Four-arm run archive | `deepseek-v4-flash-4arm-tbench-2.1-full-v1-20260805/four-arm-full-v1.tgz` |
+| Generated cohort report | `deepseek-v4-flash-4arm-tbench-2.1-full-v1-20260805/harness-cohort-report.{json,csv,md}` |
+| Immutable manifest | `deepseek-v4-flash-4arm-tbench-2.1-full-v1-20260805/harness-ab-manifest.json` |
+| Controller WAL and attempts | `deepseek-v4-flash-4arm-tbench-2.1-full-v1-20260805/extracted/*/controller/results.jsonl{,.attempts.jsonl}` |
+| Three-arm reference runs | `deepseek-v4-flash-3arm-tbench-2.1-full-20260804/` |
+| Three-arm per-task deep dive | `deepseek-v4-flash-3arm-tbench-2.1-full-20260804/deepdive/` |
+
+The committed CSV contains only the selected final pass/fail and failure-class projections; prompts, trajectories, provider payloads, raw logs, and traces are not published.
@@ -0,0 +1,90 @@
+task_id,maka,opencode,maka_failure_class,opencode_failure_class
+adaptive-rejection-sampler,failed,failed,budget_exhausted,budget_exhausted
+bn-fit-modify,passed,failed,,verification_failed
+break-filter-js-from-html,passed,passed,,
+build-cython-ext,passed,passed,,
+build-pmars,passed,passed,,
+build-pov-ray,passed,passed,,
+caffe-cifar-10,passed,failed,,budget_exhausted
+cancel-async-tasks,failed,failed,verification_failed,verification_failed
+chess-best-move,failed,failed,verification_failed,budget_exhausted
+circuit-fibsqrt,passed,passed,,
+cobol-modernization,passed,passed,,
+code-from-image,passed,failed,,budget_exhausted
+compile-compcert,failed,passed,budget_exhausted,
+configure-git-webserver,passed,passed,,
+constraints-scheduling,passed,passed,,
+count-dataset-tokens,passed,passed,,
+crack-7z-hash,passed,passed,,
+custom-memory-heap-crash,passed,passed,,
+db-wal-recovery,passed,failed,,budget_exhausted
+distribution-search,passed,passed,,
+dna-assembly,failed,failed,verification_failed,verification_failed
+dna-insert,failed,passed,verification_failed,
+extract-elf,failed,failed,verification_failed,verification_failed
+extract-moves-from-video,failed,failed,budget_exhausted,budget_exhausted
+feal-differential-cryptanalysis,passed,passed,,
+feal-linear-cryptanalysis,passed,passed,,
+filter-js-from-html,failed,failed,verification_failed,verification_failed
+financial-document-processor,passed,passed,,
+fix-code-vulnerability,passed,passed,,
+fix-git,passed,failed,,verification_failed
+fix-ocaml-gc,passed,passed,,
+gcode-to-text,failed,failed,budget_exhausted,budget_exhausted
+git-leak-recovery,passed,passed,,
+git-multibranch,passed,passed,,
+gpt2-codegolf,failed,failed,budget_exhausted,budget_exhausted
+headless-terminal,passed,passed,,
+hf-model-inference,passed,passed,,
+install-windows-3.11,failed,failed,budget_exhausted,budget_exhausted
+kv-store-grpc,passed,passed,,
+large-scale-text-editing,passed,passed,,
+largest-eigenval,failed,failed,budget_exhausted,budget_exhausted
+llm-inference-batching-scheduler,passed,passed,,
+log-summary-date-ranges,passed,passed,,
+mailman,passed,passed,,
+make-doom-for-mips,failed,failed,budget_exhausted,budget_exhausted
+make-mips-interpreter,failed,failed,budget_exhausted,budget_exhausted
+mcmc-sampling-stan,passed,passed,,
+merge-diff-arc-agi-task,passed,passed,,
+model-extraction-relu-logits,passed,failed,,budget_exhausted
+modernize-scientific-stack,passed,passed,,
+mteb-leaderboard,failed,failed,verification_failed,budget_exhausted
+mteb-retrieve,passed,failed,,verification_failed
+multi-source-data-merger,passed,passed,,
+nginx-request-logging,passed,passed,,
+openssl-selfsigned-cert,passed,passed,,
+overfull-hbox,failed,failed,verification_failed,verification_failed
+password-recovery,failed,passed,budget_exhausted,
+path-tracing,passed,failed,,budget_exhausted
+path-tracing-reverse,failed,failed,budget_exhausted,budget_exhausted
+polyglot-c-py,passed,passed,,
+polyglot-rust-c,passed,passed,,
+portfolio-optimization,passed,passed,,
+protein-assembly,failed,failed,verification_failed,verification_failed
+prove-plus-comm,passed,passed,,
+pypi-server,passed,passed,,
+pytorch-model-cli,passed,passed,,
+pytorch-model-recovery,passed,passed,,
+qemu-alpine-ssh,failed,failed,budget_exhausted,budget_exhausted
+qemu-startup,failed,passed,budget_exhausted,
+query-optimize,passed,passed,,
+raman-fitting,failed,failed,verification_failed,budget_exhausted
+regex-chess,passed,failed,,budget_exhausted
+regex-log,passed,passed,,
+reshard-c4-data,passed,passed,,
+rstan-to-pystan,failed,failed,budget_exhausted,runtime_error
+sam-cell-seg,passed,passed,,
+sanitize-git-repo,passed,failed,,verification_failed
+schemelike-metacircular-eval,passed,failed,,verification_failed
+sparql-university,passed,failed,,verification_failed
+sqlite-db-truncate,passed,passed,,
+sqlite-with-gcov,passed,failed,,budget_exhausted
+torch-pipeline-parallelism,failed,failed,verification_failed,budget_exhausted
+torch-tensor-parallelism,failed,failed,verification_failed,verification_failed
+train-fasttext,failed,failed,budget_exhausted,budget_exhausted
+tune-mjcf,passed,failed,,budget_exhausted
+video-processing,failed,failed,verification_failed,verification_failed
+vulnerable-secret,passed,passed,,
+winning-avg-corewars,passed,failed,,verification_failed
+write-compressor,passed,failed,,budget_exhausted
@@ -0,0 +1,162 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Terminal-Bench 2.1 — DeepSeek V4 Flash: Maka vs OpenCode
+
+This report compares Maka and OpenCode as two agent harnesses around the same DeepSeek V4 Flash model on all 89 Terminal-Bench 2.1 tasks. It records the frozen experiment, accepted outcomes, paired inference, outcome-normalized economics, and the recovery decisions required to produce a fully scored dataset.
+
+**Run id:** `deepseek-v4-flash-maka-vs-opencode-tbench-2.1-full-v7`
+
+**Local artifacts (git-excluded):** `~/.maka/eval/runs/deepseek-v4-flash-maka-vs-opencode-tbench-2.1-full-v7/`
+
+**Metric:** end-to-end pass@1 by the official task verifier
+
+**Status:** `completed` — 178/178 cells model-scored, no unscored infrastructure cell, and no missing final usage
+
+**Per-task outcomes:** [`terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.csv`](./terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.csv)
+
+## TL;DR
+
+- **Maka passed 61/89 tasks (68.54%); OpenCode passed 49/89 (55.06%).** Maka led by 12 tasks, or 13.48 percentage points.
+- The paired outcomes were 16 Maka-only passes, 4 OpenCode-only passes, and 69 ties. An exact two-sided McNemar test on the 20 discordant pairs gives **p = 0.0118**. This is evidence about the fixed 89-task Terminal-Bench 2.1 suite, not proof of a universal advantage on other task distributions.
+- The observed accepted-dataset API-equivalent cost per pass was effectively the same: **$0.03172 for Maka and $0.03171 for OpenCode**. No cost-equivalence test was performed, so this is a descriptive point estimate rather than a statistical equivalence claim.
+- Two diagnostics accompany, but do not replace, end-to-end pass@1. On the shared 61-pair subset obtained by excluding any pair where either arm exhausted its budget, Maka passed 52/61 (85.25%) and OpenCode 46/61 (75.41%). Budget exhaustion affected 15/89 Maka cells (16.85%) and 24/89 OpenCode cells (26.97%). These observations decompose the final gap; this run does not identify their causal mechanisms.
+
+## Results
+
+End-to-end pass@1 is the primary result. Budget-exhausted cells remain scored failures in its denominator.
+
+| Primary result | Maka | OpenCode | Maka − OpenCode |
+| --- | ---: | ---: | ---: |
+| End-to-end pass@1 | **61/89 (68.54%)** | **49/89 (55.06%)** | **+12 tasks (+13.48 pp)** |
+
+The paired outcome table is:
+
+| | OpenCode pass | OpenCode fail | Total |
+| --- | ---: | ---: | ---: |
+| Maka pass | 45 | 16 | 61 |
+| Maka fail | 4 | 24 | 28 |
+| Total | 49 | 40 | 89 |
+
+For the exact McNemar test, the null assigns equal probability to either direction among discordant pairs. With 16 Maka-only and 4 OpenCode-only outcomes, the two-sided exact binomial probability is `2 × P[Binomial(20, 0.5) ≤ 4] = 0.0118179`. The test treats the 89 benchmark tasks as the paired units. Because this was one frozen run over one fixed suite, the p-value should not be read as a guarantee for other benchmarks, task distributions, provider conditions, or repetitions.
+
+## Diagnostic decomposition
+
+The following metrics diagnose where the observed end-to-end gap appears; neither is an alternate headline score.
+
+| Diagnostic | Maka | OpenCode | Maka − OpenCode |
+| --- | ---: | ---: | ---: |
+| Non-budget Conditional Pass Rate | 52/61 (85.25%) | 46/61 (75.41%) | +9.84 pp |
+| Budget Exhaustion Rate | 15/89 (16.85%) | 24/89 (26.97%) | −10.11 pp |
+
+The conditional denominator excludes the entire pair whenever either arm is `budget_exhausted`, leaving the same 61 tasks for both arms. It is not an “unlimited-time pass rate”: the remaining tasks still ran under their original budgets, and the excluded set is not random. The simultaneous conditional-pass and budget-exhaustion differences show that both dimensions accompany the end-to-end result. They do not by themselves isolate model solution quality, inference speed, or agent turnover efficiency as causes.
+
+## Outcome-normalized economics
+
+Cost per pass is the primary economic denominator because it normalizes recorded usage by successful benchmark outcomes. It includes spending on all accepted scored failures, but excludes superseded infrastructure-invalid admissions from the accepted dataset.
+
+| Economic result | Maka | OpenCode |
+| --- | ---: | ---: |
+| Accepted-dataset API-equivalent cost per pass | **$0.031718** | **$0.031712** |
+| Passed tasks | 61 | 49 |
+
+The observed difference is less than $0.000007 per pass, or 0.02%. At the precision supported by this single run, the two point estimates are effectively the same. This is not a statistical cost-equivalence result: the experiment has one repetition and no uncertainty model for cost per pass.
+
+For completeness, the resource footprint behind that outcome-normalized result is:
+
+| Resource diagnostic | Maka | OpenCode |
+| --- | ---: | ---: |
+| Accepted scored-cell cost | $1.934813 | $1.553888 |
+| Cost per scored cell | $0.021739 | $0.017459 |
+| Total tokens | 173,281,686 | 115,523,680 |
+| Cached input tokens | 166,618,240 | 110,226,048 |
+| Uncached input tokens | 3,315,719 | 2,083,315 |
+| Output tokens | 3,347,727 | 3,214,317 |
+| Cache-hit share of input | 98.05% | 98.15% |
+
+OpenCode had a lower per-cell footprint, but it also produced fewer passes. The lower aggregate spend therefore does not become an observed cost-per-success advantage. The report keeps resource footprint and outcome-normalized economics separate rather than combining them into a composite score.
+
+Costs are cache-aware API-equivalent estimates from the pricing identity frozen in the manifest: $0.145 per million uncached input tokens, $0.0029 per million cache-hit input tokens, and $0.29 per million output tokens. They are not a billing invoice.
+
+## Frozen setup
+
+| Dimension | Value |
+| --- | --- |
+| Benchmark | Terminal-Bench 2.1, revision `d49e28f1e4ddd13d289e85a5f312a66750951932`; all 89 tasks |
+| Task-tree fingerprint | `sha256:456826aa4c47ed309716c964c96d2a3acc998764ebc84f3e8449c807d74bd4e7` |
+| Run fingerprint | `sha256:d18bc4c5ab0dd15161f48838ab0fa4a7a9ba2654f6008afe36d3d398fdf5fff3` |
+| Model | `deepseek-v4-flash` through the DeepSeek provider on both arms |
+| Reasoning effort | `max` on both arms |
+| Repetitions | 1 |
+| Metric | Paired pass@1 |
+| Attempt policy | One accepted model attempt per arm/task; only infrastructure-invalid admissions may be replaced |
+| Deadline policy | Task-native agent timeout ×1; 900-second outer setup and teardown grace |
+| Pair execution | Up to four task pairs concurrently; Maka and OpenCode start in parallel within a pair; at most eight cells concurrently |
+| External system prompt | Empty on both arms |
+| Maka arm | `maka_agent:MakaAgent`; continuation off; active and stale tool-result pruning enabled at a 2,048 estimated-token threshold; semantic compact off |
+| OpenCode arm | `opencode_agent:MakaOpenCodeAgent` 1.17.18; pure mode; automatic permissions; `max` variant |
+| Billing mode | Metered, using the frozen DeepSeek V4 Flash pricing identity above |
+
+This is a same-model harness comparison, not an isolated model evaluation and not a same-system/same-tool ablation. The two arms retain their native agent instructions, tool behavior, context management, and execution loops. The observed difference is therefore attributable to the compared harness systems as a whole; this run does not rank individual harness differences as causes.
+
+## Outcome accounting and recovery
+
+The accepted dataset selects the latest authorized terminal outcome for each arm/task cell under the immutable run fingerprint. The attempts WAL contains **183 Agent admissions**: 178 initial admissions plus five single infrastructure retries. Those retries covered four Maka cells (`winning-avg-corewars`, `polyglot-rust-c`, `path-tracing`, and `qemu-startup`) and one OpenCode cell (`polyglot-rust-c`). No cell received a third Agent admission.
+
+The adjudication ledger contains 25 taxonomy-only corrections that did not change pass/fail: 24 OpenCode cells with authoritative Harbor `AgentTimeoutError` evidence were normalized from `verification_failed` to scored `budget_exhausted`, and OpenCode `rstan-to-pystan` was normalized from harness infrastructure to a scored candidate runtime/resource failure after substantial execution and an official verifier result. Two pre-execution Maka failures were reclassified as unscored infrastructure before receiving their one allowed replacement admission.
+
+The final OpenCode `polyglot-rust-c` admission completed all five provider requests, recorded complete usage, wrote and locally compiled the requested candidate, and then encountered verifier setup infrastructure failure twice because the task container could not reach the Ubuntu archive. The container was deleted, but the single-file candidate was durably and completely represented in the second admission's trajectory. The candidate was reconstructed byte-for-byte (`sha256:dfa4c4cd3b41744e6fd4f971ca544075cc305d625bcd4b515b8ba559438c2cd3`) and replayed only through the unchanged official verifier in the pinned `linux/amd64` task image with the original 1 CPU / 2 GiB limits. The official test passed and returned reward 1. This recovery changed the cell from unscored infrastructure to pass without invoking the model again; the Agent admission count remained two.
+
+The recovery layer is append-only: superseded events remain in the WAL, adjudications identify their source event, and the final verifier-only replay has a separate local evidence manifest. The committed CSV contains only the selected final pass/fail and failure-class projections; prompts, trajectories, provider payloads, raw logs, and local capabilities are not published.
+
+## Caveats
+
+- This is one repetition over a fixed 89-task suite. The exact paired p-value describes asymmetry on this suite and does not establish universal superiority.
+- The arms share the model, provider, task order, instructions, verifier, deadline policy, and infrastructure-retry policy, but they intentionally retain native harness behavior. This is not a single-variable component ablation.
+- Non-budget Conditional Pass Rate is selection-conditional and diagnostic. It cannot be interpreted as an unlimited-time counterfactual.
+- Budget exhaustion records that a valid candidate opportunity reached the benchmark deadline and then received an official verifier result. It does not by itself distinguish provider latency, model generation length, tool time, or agent policy.
+- Cost per pass is an accepted-dataset point estimate from recorded usage and frozen API-equivalent prices. It includes scored failures but excludes superseded infrastructure-invalid admissions, whose complete metering is not always available. “Effectively the same” describes the observed values, not a formal equivalence interval or total experiment-operations cost.
+- No external Oracle registry snapshot was configured. Oracle annotations are missing for all 89 tasks; the official Terminal-Bench verifier remains the scoring authority.
+- Five infrastructure-invalid Agent admissions were replaced under the same bounded policy, and one completed candidate required verifier-only replay. These recoveries are disclosed above because omitting them would make the final coverage and score non-reconstructible.
+
+## Integrity
+
+SHA-256 hashes of the frozen local evidence and committed outcome projection:
+
+| Source | SHA-256 |
+| --- | --- |
+| `harness-ab-manifest.json` | `c364730ac5e3db4ff8a6cf0b22a50b4e4f953ee05d090dc69415cb9012376e9c` |
+| `harness-ab-report.json` | `bd616091c18bed6ad8d4bfe7534c7f4d6a6861c01b1c6f383d408b39673f922b` |
+| `controller/results.jsonl` | `19764b90cbebddeb05c20d58be8760bfdabac1bf99e1c15a0528c0f934cd9835` |
+| `controller/results.jsonl.attempts.jsonl` | `4878ca5b612d20d87a28066456ddcdc32a5c0bb105940cb0e6ef270a2355c7ab` |
+| `controller/adjudications.jsonl` | `7253f3cf043cf5079428b704baf3ee30b8c259a4dc806704cb7016914a4cd667` |
+| Verifier-only replay `evidence.json` | `93b786d4305452bf1f1ef15f5474f5902bc979255e89c61f5669581caab5178b` |
+| Committed outcome CSV | `a1f52e1fbb4463ed1f9a6d591dd50aa8eaff67dbb825d05a36213d0d3a25e766` |
+
+## Artifact pointers
+
+| Artifact | Local path |
+| --- | --- |
+| Generated report | `~/.maka/eval/runs/deepseek-v4-flash-maka-vs-opencode-tbench-2.1-full-v7/harness-ab-report.{json,csv,md}` |
+| Immutable manifest | `~/.maka/eval/runs/deepseek-v4-flash-maka-vs-opencode-tbench-2.1-full-v7/harness-ab-manifest.json` |
+| Controller WAL | `.../controller/results.jsonl` and `.../controller/results.jsonl.attempts.jsonl` |
+| Adjudication ledger | `.../controller/adjudications.jsonl` |
+| Verifier replay evidence | `.../controller/verifier-replays/ab-opencode-r0-polyglot-rust-c-attempt-2/evidence.json` |
+
+The experiment was tracked in [issue #1680](https://github.com/maka-agent/maka-agent/issues/1680).
@@ -0,0 +1,90 @@
+task,codex,maka,pi,dsh,zcode,reasonix,opencode,kimi-code,claude-code
+adaptive-rejection-sampler,pass,pass,pass,fail,fail,fail,pass,fail,fail
+bn-fit-modify,fail,fail,pass,pass,pass,pass,fail,fail,pass
+break-filter-js-from-html,pass,pass,pass,pass,pass,pass,pass,pass,pass
+build-cython-ext,pass,pass,pass,pass,pass,pass,pass,pass,fail
+build-pmars,pass,pass,pass,pass,pass,pass,pass,pass,fail
+build-pov-ray,pass,pass,pass,pass,fail,pass,fail,pass,fail
+caffe-cifar-10,pass,fail,pass,fail,pass,fail,pass,fail,fail
+cancel-async-tasks,fail,pass,fail,pass,pass,pass,fail,fail,pass
+chess-best-move,fail,fail,fail,pass,pass,fail,fail,fail,fail
+circuit-fibsqrt,pass,pass,pass,pass,pass,fail,pass,pass,pass
+cobol-modernization,pass,pass,pass,pass,pass,pass,pass,pass,pass
+code-from-image,pass,pass,pass,pass,pass,pass,fail,fail,fail
+compile-compcert,pass,pass,pass,pass,pass,pass,fail,fail,pass
+configure-git-webserver,pass,pass,pass,pass,pass,fail,pass,fail,pass
+constraints-scheduling,pass,pass,pass,pass,pass,pass,pass,pass,pass
+count-dataset-tokens,pass,pass,pass,pass,pass,pass,pass,pass,pass
+crack-7z-hash,pass,pass,pass,pass,pass,pass,pass,fail,fail
+custom-memory-heap-crash,pass,pass,pass,pass,pass,pass,pass,pass,pass
+db-wal-recovery,pass,fail,pass,fail,pass,pass,pass,fail,pass
+distribution-search,pass,pass,pass,pass,pass,pass,pass,pass,pass
+dna-assembly,pass,pass,fail,fail,fail,fail,fail,fail,fail
+dna-insert,pass,fail,pass,pass,fail,fail,fail,fail,fail
+extract-elf,pass,fail,fail,fail,pass,fail,fail,fail,pass
+extract-moves-from-video,fail,fail,fail,fail,fail,fail,fail,fail,fail
+feal-differential-cryptanalysis,pass,pass,pass,pass,pass,pass,pass,pass,pass
+feal-linear-cryptanalysis,pass,pass,pass,pass,pass,fail,pass,pass,pass
+filter-js-from-html,fail,fail,fail,fail,fail,fail,fail,fail,fail
+financial-document-processor,pass,pass,pass,pass,pass,pass,pass,fail,pass
+fix-code-vulnerability,pass,pass,pass,pass,pass,pass,pass,pass,pass
+fix-git,pass,pass,pass,pass,pass,pass,pass,pass,pass
+fix-ocaml-gc,pass,pass,pass,pass,pass,pass,pass,pass,pass
+gcode-to-text,fail,fail,fail,fail,fail,fail,fail,fail,fail
+git-leak-recovery,pass,pass,pass,pass,pass,pass,pass,pass,fail
+git-multibranch,pass,pass,pass,pass,pass,pass,pass,pass,pass
+gpt2-codegolf,pass,fail,fail,fail,fail,fail,fail,fail,fail
+headless-terminal,pass,pass,pass,pass,pass,pass,pass,pass,pass
+hf-model-inference,pass,pass,pass,pass,pass,pass,pass,pass,pass
+install-windows-3.11,pass,pass,fail,fail,fail,fail,fail,fail,fail
+kv-store-grpc,pass,pass,pass,fail,fail,fail,pass,pass,fail
+large-scale-text-editing,pass,pass,pass,pass,pass,pass,pass,pass,pass
+largest-eigenval,pass,pass,fail,pass,fail,fail,pass,fail,fail
+llm-inference-batching-scheduler,pass,pass,pass,pass,pass,pass,pass,pass,pass
+log-summary-date-ranges,pass,pass,pass,pass,pass,pass,pass,pass,pass
+mailman,pass,pass,pass,fail,pass,pass,fail,pass,fail
+make-doom-for-mips,fail,fail,fail,fail,fail,fail,fail,fail,fail
+make-mips-interpreter,pass,fail,fail,pass,fail,fail,fail,fail,fail
+mcmc-sampling-stan,pass,pass,pass,pass,pass,pass,pass,fail,fail
+merge-diff-arc-agi-task,pass,pass,pass,pass,pass,pass,pass,pass,fail
+model-extraction-relu-logits,fail,pass,fail,pass,pass,pass,fail,pass,fail
+modernize-scientific-stack,pass,pass,pass,pass,pass,pass,pass,pass,pass
+mteb-leaderboard,pass,pass,fail,pass,pass,pass,pass,pass,pass
+mteb-retrieve,pass,pass,pass,pass,pass,pass,fail,fail,pass
+multi-source-data-merger,pass,pass,pass,pass,pass,pass,pass,pass,pass
+nginx-request-logging,pass,pass,pass,pass,pass,pass,pass,pass,fail
+openssl-selfsigned-cert,pass,pass,pass,pass,pass,pass,pass,pass,pass
+overfull-hbox,pass,fail,pass,pass,pass,fail,pass,fail,pass
+password-recovery,pass,pass,pass,fail,pass,fail,pass,pass,pass
+path-tracing,pass,pass,pass,pass,pass,pass,pass,fail,fail
+path-tracing-reverse,pass,pass,pass,pass,fail,fail,fail,fail,pass
+polyglot-c-py,pass,pass,pass,pass,pass,pass,pass,pass,pass
+polyglot-rust-c,pass,pass,pass,pass,pass,pass,pass,pass,pass
+portfolio-optimization,pass,pass,pass,pass,pass,pass,pass,pass,pass
+protein-assembly,fail,fail,fail,fail,fail,pass,fail,fail,pass
+prove-plus-comm,pass,pass,pass,pass,pass,pass,pass,pass,pass
+pypi-server,fail,pass,pass,fail,fail,pass,pass,pass,pass
+pytorch-model-cli,pass,pass,pass,pass,pass,pass,pass,fail,fail
+pytorch-model-recovery,pass,pass,fail,fail,pass,pass,fail,pass,pass
+qemu-alpine-ssh,fail,pass,fail,fail,fail,pass,pass,pass,pass
+qemu-startup,pass,pass,pass,fail,fail,fail,pass,pass,pass
+query-optimize,pass,pass,fail,pass,fail,pass,pass,pass,fail
+raman-fitting,fail,fail,fail,fail,pass,fail,fail,fail,fail
+regex-chess,pass,pass,pass,pass,fail,fail,fail,fail,fail
+regex-log,pass,pass,pass,pass,pass,pass,pass,pass,pass
+reshard-c4-data,pass,pass,pass,pass,pass,pass,pass,pass,pass
+rstan-to-pystan,pass,pass,pass,fail,pass,pass,fail,pass,fail
+sam-cell-seg,pass,pass,pass,pass,pass,fail,pass,fail,fail
+sanitize-git-repo,fail,pass,pass,pass,fail,pass,fail,pass,fail
+schemelike-metacircular-eval,pass,pass,pass,pass,pass,pass,pass,pass,fail
+sparql-university,pass,pass,pass,pass,pass,pass,pass,pass,pass
+sqlite-db-truncate,pass,pass,pass,pass,pass,pass,pass,pass,pass
+sqlite-with-gcov,pass,pass,pass,pass,pass,pass,pass,pass,fail
+torch-pipeline-parallelism,fail,fail,fail,fail,fail,fail,fail,fail,fail
+torch-tensor-parallelism,pass,fail,pass,pass,fail,fail,fail,pass,pass
+train-fasttext,fail,fail,fail,pass,fail,fail,fail,fail,fail
+tune-mjcf,pass,pass,fail,pass,fail,pass,fail,fail,fail
+video-processing,fail,fail,fail,fail,fail,fail,pass,fail,fail
+vulnerable-secret,pass,pass,pass,pass,pass,pass,pass,pass,pass
+winning-avg-corewars,pass,pass,pass,pass,pass,pass,fail,pass,pass
+write-compressor,pass,fail,pass,fail,pass,pass,pass,pass,fail
@@ -0,0 +1,193 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Terminal-Bench 2.1 — DeepSeek V4 Flash nine-harness comparison
+
+This report records the final selected outcomes for nine coding-agent harnesses around the same
+DeepSeek V4 Flash model on all 89 Terminal-Bench 2.1 tasks. It extends the earlier four-arm report
+in [#2208](https://github.com/maka-agent/maka-agent/pull/2208) with OpenCode, Kimi Code, ZCode, Pi,
+and DeepSeek Harness (DSH).
+
+**Benchmark:** Terminal-Bench 2.1, revision
+`d49e28f1e4ddd13d289e85a5f312a66750951932`
+
+**Model:** `deepseek-v4-flash`
+
+**Reasoning effort:** `max`
+
+**Metric:** end-to-end pass@1 by the official task verifier
+
+**Per-task outcomes:**
+[`terminal-bench-2.1-deepseek-v4-flash-nine-arm.csv`](./terminal-bench-2.1-deepseek-v4-flash-nine-arm.csv)
+
+## TL;DR
+
+- **Codex leads at 73/89 (82.0%)**, followed by Maka at 69/89 (77.5%), Pi at 66/89
+  (74.2%), and DSH at 65/89 (73.0%).
+- All nine harnesses pass the same 28 tasks. Five tasks fail on every harness:
+  `extract-moves-from-video`, `filter-js-from-html`, `gcode-to-text`,
+  `make-doom-for-mips`, and `torch-pipeline-parallelism`.
+- Pi records the lowest observed cost per pass at $0.019739. DSH and Maka are nearly tied on this
+  measure at $0.026230 and $0.026307.
+- The original eight arms ran as one cohort. DSH ran later under the same benchmark, model,
+  reasoning, tool-egress, permission, verifier, and task-native timeout contracts, but not in the
+  same simultaneous cohort. The nine-way ranking is therefore descriptive rather than a paired
+  causal comparison.
+
+## Leaderboard
+
+The reporting projection treats the token, cache, and cost values recorded by the selected attempts
+as authoritative and does not extrapolate unobserved provider usage.
+
+| Rank | Harness | Passed | Pass@1 | Tokens | Average/task | Cache rate | Cost |
+| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| 1 | Codex | 73/89 | **82.0%** | 259.62M | 2.92M | 98.6% | $2.105 |
+| 2 | Maka | 69/89 | **77.5%** | 185.71M | 2.09M | 98.8% | $1.815 |
+| 3 | Pi | 66/89 | **74.2%** | 131.40M | 1.48M | 98.5% | $1.303 |
+| 4 | DSH | 65/89 | **73.0%** | 182.84M | 2.05M | 98.6% | $1.705 |
+| 5 | ZCode | 63/89 | **70.8%** | 223.46M | 2.51M | 98.9% | $2.287 |
+| 6 | Reasonix | 60/89 | **67.4%** | 216.17M | 2.43M | 99.0% | $1.935 |
+| 7 | OpenCode | 58/89 | **65.2%** | 166.44M | 1.87M | 98.6% | $1.745 |
+| 8 | Kimi Code | 53/89 | **59.6%** | 193.75M | 2.18M | 98.2% | $2.162 |
+| 9 | Claude Code | 49/89 | **55.1%** | 247.60M | 2.78M | 98.9% | $2.633 |
+
+The 26.9-point spread between Codex and Claude Code is 24 tasks. Every task is worth approximately
+1.12 percentage points on this fixed suite.
+
+## Outcome agreement
+
+| Agreement class | Tasks |
+| --- | ---: |
+| All nine pass | 28 |
+| All nine fail | 5 |
+| Mixed outcome | 56 |
+
+The per-task CSV is the authoritative projection for these counts. It contains exactly 89 unique
+task ids and reproduces every harness total in the leaderboard.
+
+## Outcome-normalized economics
+
+Cost per pass includes the recorded cost of both successful and failed selected cells.
+
+| Harness | Recorded cost | Passed | Cost/pass |
+| --- | ---: | ---: | ---: |
+| Pi | $1.303 | 66 | **$0.019739** |
+| DSH | $1.705 | 65 | **$0.026230** |
+| Maka | $1.815 | 69 | **$0.026307** |
+| Codex | $2.105 | 73 | **$0.028832** |
+| OpenCode | $1.745 | 58 | **$0.030084** |
+| Reasonix | $1.935 | 60 | **$0.032252** |
+| ZCode | $2.287 | 63 | **$0.036303** |
+| Kimi Code | $2.162 | 53 | **$0.040798** |
+| Claude Code | $2.633 | 49 | **$0.053741** |
+
+These are descriptive point estimates from one selected attempt per cell. They are not billing
+invoices or statistical cost-equivalence claims.
+
+## Frozen execution contract
+
+| Dimension | Value |
+| --- | --- |
+| Benchmark | Terminal-Bench 2.1; 89 explicit tasks at revision `d49e28f1e4ddd13d289e85a5f312a66750951932` |
+| Model | `deepseek-v4-flash` on every arm |
+| Reasoning | `max`; provider thinking enabled |
+| Repetitions | 1 |
+| Verifier | Official task-native Terminal-Bench verifier |
+| Deadline | Each task's native agent timeout ×1 |
+| Max steps | 100,000 |
+| Web tools | Removed from the provider-visible tool surface |
+| Shell networking | Enabled, with benchmark-contamination egress filtering |
+| Permissions | Non-interactive benchmark execution |
+| Selection | Valid scored attempts; authorized infrastructure replacements and recorded metric recovery |
+
+The harnesses retain their native prompts, tool schemas, context management, process lifecycle, and
+execution loops. The result compares complete harness systems around one model; it is not a
+single-component ablation.
+
+## Outcome accounting and recovery
+
+The eight-arm result was assembled from the full cohort plus bounded infrastructure, usage, and
+trajectory recovery runs. The final eight-arm trajectory manifest records 712/712 cells with a
+physical trajectory artifact and no missing selected cell.
+
+DSH initially recorded 61/89. Real-machine inspection identified three Eval-specific defects:
+
+1. A five-minute DSH Bash timeout interrupted package installation and left `dpkg` unusable for
+   verification.
+2. Package installation inherited an interactive `tzdata` configuration path instead of the
+   benchmark's non-interactive execution contract.
+3. DSH and the Eval relay terminated task-created background services before the shared-environment
+   verifier could reach them.
+
+Commit `28ebe0949` deferred Bash lifetime to the benchmark deadline, made package installation
+non-interactive, and preserved background descendants only after successful DSH exit. Seven
+affected cells were rerun. Four changed from fail to pass:
+
+- `hf-model-inference`
+- `merge-diff-arc-agi-task`
+- `polyglot-c-py`
+- `regex-log`
+
+Three remained genuine verifier failures:
+
+- `dna-assembly`
+- `extract-moves-from-video`
+- `torch-pipeline-parallelism`
+
+The repaired DSH result is 65/89.
+
+## Caveats
+
+- This is one repetition over one fixed suite.
+- DSH was a configuration-aligned follow-up run, not a ninth arm started simultaneously with the
+  original eight-arm cohort.
+- Some persisted trajectories hit the Eval recording limit. This affects replay completeness, not
+  the official verifier score used here.
+- The report uses the recorded token, cache, and cost aggregates directly, without replacing them
+  with projections.
+- Recovery admissions can sample a different model trajectory. The CSV freezes the selected final
+  outcomes used by this report.
+
+## Artifact pointers
+
+| Artifact | Path |
+| --- | --- |
+| Original eight-arm run | `/mnt/deepswe/eval-runs/deepseek-v4-flash-eight-arm-full-24-903dd1b6-v1` |
+| Eight-arm trajectory overlay | `/mnt/deepswe/eval-runs/deepseek-v4-flash-final-trajectory-overlay-8a1ab4635.json` |
+| DSH full run | `/mnt/deepswe/eval-runs/deepseek-v4-flash-dsh-eightarm-comparable-full-89-v1` |
+| DSH service repair | `/mnt/deepswe/eval-runs/deepseek-v4-flash-dsh-service-canary-v3-74c98a6ac` |
+| DSH six-cell repair | `/mnt/deepswe/eval-runs/deepseek-v4-flash-dsh-verifier-fix-retry-6-v4-28ebe0949` |
+| DSH Eval bundle | `/mnt/deepswe/maka-eval-dsh-verifier-fix-74c98a6ac` |
+| DSH toolchain | `/mnt/deepswe/toolchains/deepseek-harness-0.1.0-rc.6-bullseye-v3` |
+
+The DSH toolchain fingerprint is
+`sha256:a0882b448718ddfb7b64e33a12369c92b0064baf8388fe08a8ff64fe3dd98896`.
+
+## Integrity and verification
+
+- CSV rows: 89
+- Unique task ids: 89
+- Recomputed pass totals: 73, 69, 66, 65, 63, 60, 58, 53, 49
+- All-nine passes: 28
+- All-nine failures: 5
+- CSV SHA-256:
+  `c27c3bcbfc3ebe8e21cc250dc409f02f49ae055032eaf2f19fd6349986e94e6a`
+- DSH repair code: `28ebe0949`
+- DSH repair verification: Eval Node 36/36; Python relay/policy/artifact 33/33; lint,
+  format, and Eval typecheck passed
@@ -0,0 +1,90 @@
+task_id,maka,kimi_code
+adaptive-rejection-sampler,failed,failed
+bn-fit-modify,passed,passed
+break-filter-js-from-html,passed,passed
+build-cython-ext,failed,failed
+build-pmars,passed,passed
+build-pov-ray,passed,failed
+caffe-cifar-10,failed,failed
+cancel-async-tasks,passed,passed
+chess-best-move,passed,passed
+circuit-fibsqrt,passed,passed
+cobol-modernization,failed,failed
+code-from-image,passed,passed
+compile-compcert,passed,passed
+configure-git-webserver,passed,failed
+constraints-scheduling,passed,passed
+count-dataset-tokens,passed,passed
+crack-7z-hash,passed,failed
+custom-memory-heap-crash,passed,passed
+db-wal-recovery,passed,passed
+distribution-search,passed,passed
+dna-assembly,failed,failed
+dna-insert,failed,failed
+extract-elf,failed,failed
+extract-moves-from-video,failed,failed
+feal-differential-cryptanalysis,passed,passed
+feal-linear-cryptanalysis,failed,passed
+filter-js-from-html,failed,failed
+financial-document-processor,passed,failed
+fix-code-vulnerability,passed,failed
+fix-git,passed,passed
+fix-ocaml-gc,passed,passed
+gcode-to-text,failed,failed
+git-leak-recovery,passed,passed
+git-multibranch,passed,passed
+gpt2-codegolf,passed,failed
+headless-terminal,passed,passed
+hf-model-inference,passed,passed
+install-windows-3.11,failed,failed
+kv-store-grpc,passed,passed
+large-scale-text-editing,passed,passed
+largest-eigenval,failed,failed
+llm-inference-batching-scheduler,failed,failed
+log-summary-date-ranges,passed,passed
+mailman,passed,passed
+make-doom-for-mips,failed,failed
+make-mips-interpreter,failed,failed
+mcmc-sampling-stan,passed,failed
+merge-diff-arc-agi-task,passed,passed
+model-extraction-relu-logits,failed,failed
+modernize-scientific-stack,passed,passed
+mteb-leaderboard,passed,passed
+mteb-retrieve,failed,passed
+multi-source-data-merger,passed,passed
+nginx-request-logging,passed,passed
+openssl-selfsigned-cert,passed,passed
+overfull-hbox,passed,failed
+password-recovery,passed,passed
+path-tracing,passed,passed
+path-tracing-reverse,failed,failed
+polyglot-c-py,passed,passed
+polyglot-rust-c,passed,passed
+portfolio-optimization,passed,passed
+protein-assembly,passed,passed
+prove-plus-comm,passed,passed
+pypi-server,passed,passed
+pytorch-model-cli,passed,passed
+pytorch-model-recovery,passed,passed
+qemu-alpine-ssh,failed,failed
+qemu-startup,failed,failed
+query-optimize,failed,failed
+raman-fitting,failed,failed
+regex-chess,passed,failed
+regex-log,passed,passed
+reshard-c4-data,passed,passed
+rstan-to-pystan,passed,passed
+sam-cell-seg,passed,passed
+sanitize-git-repo,passed,passed
+schemelike-metacircular-eval,passed,failed
+sparql-university,passed,passed
+sqlite-db-truncate,passed,passed
+sqlite-with-gcov,passed,passed
+torch-pipeline-parallelism,passed,failed
+torch-tensor-parallelism,passed,passed
+train-fasttext,failed,failed
+tune-mjcf,failed,passed
+video-processing,passed,failed
+vulnerable-secret,passed,passed
+winning-avg-corewars,failed,failed
+write-compressor,failed,failed
@@ -0,0 +1,198 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Terminal-Bench 2.1 — Maka vs Kimi Code (v11)
+
+Paired harness A/B on Terminal-Bench 2.1 with the same model on both arms. This document records the v11 full run: the accepted outcomes and how they were selected, why both absolute scores are a throughput-depressed lower bound, and what explains the relative gap.
+
+**Run id:** `k3-maka-vs-kimi-code-tbench-2.1-full-v11`
+**Local artifacts (git-excluded):** `~/.maka/eval/runs/k3-maka-vs-kimi-code-tbench-2.1-full-v11/`
+**Metric:** pass@1 by Harbor reward
+**Status:** `completed_with_gaps` (178/178 cells accepted and model-scored after recovery; oracle evidence missing)
+**Per-task outcomes:** [`terminal-bench-2.1-maka-vs-kimi-code-v11.csv`](./terminal-bench-2.1-maka-vs-kimi-code-v11.csv)
+
+## TL;DR
+
+- **Headline (accepted outcomes):** Maka **62/89 (69.7%)**, Kimi Code **53/89 (59.6%)** — same model (`k3`) at **max** thinking effort, with thinking history retained on both arms. The Kimi Code headline is operator-adjusted: a mechanical first-verifier count is **50/89** (see Results).
+- **Both absolute scores are a lower bound, not the model's ceiling.** Effective decode ran at ~34–37 output tok/s with heavy first-token latency, and 28 (Maka) / 33 (Kimi Code) of 89 tasks were killed at their task-native deadlines mid-generation; killed tasks passed only 14% / 15% of the time. Tasks that finished in time passed at **95.1%** (Maka) and **85.7%** (Kimi Code), against an **official 88.3%** reported for the same KimiCode harness. Under healthy throughput Maka would plausibly reach or exceed that reference and Kimi Code would plausibly land near it (projection below) — at which point Terminal-Bench 2.1 is close to saturated at the frontier.
+- **The relative +10.1 pp gap (in Maka's favor) is harness asymmetry,** not thinking effort, thinking retention, or network. The main setting differences are Maka's **context-budget tool-result prune** (on, and measured firing), empty system + narrow foreground tools vs Kimi Code's full product CLI (~20KB system, full tool surface). The gap persists among finished-in-time tasks (95.1% vs 85.7%). Differences are listed, not ranked as score causes.
+
+## Results
+
+Accepted final outcomes (after the audited recovery layer below):
+
+| Selection | Maka | Kimi Code | Kimi Code − Maka |
+| --- | ---: | ---: | ---: |
+| Accepted final outcomes | 62/89 (69.7%) | 53/89 (59.6%) | −9 tasks (−10.1 pp) |
+| Mechanical first structured verifier | 62/89 (69.7%) | 50/89 (56.2%) | −12 tasks (−13.5 pp) |
+| Reference: official Kimi K3 + KimiCode harness ([tech blog](https://www.kimi.com/blog/kimi-k3), accessed 2026-07-19) | — | 88.3% | — |
+
+Paired outcomes (accepted): Maka exclusive pass **12**, Kimi Code exclusive pass **3**, both pass **50**, both fail **24** (ties 74).
+
+This is a single-repetition run (`reps = 1`). All numbers are descriptive observations of one frozen run, not estimates of statistical superiority.
+
+## Outcome accounting
+
+The accepted dataset selects the last structured verifier outcome per arm/task cell after recorded recovery admissions. The attempts WAL contains **203 admissions**: 178 initial plus **25 recovery** (Maka 12, Kimi Code 13).
+
+| Recorded recovery reason | Admissions | Changed selected pass/fail |
+| --- | ---: | ---: |
+| `operator_authorized_infrastructure_interruption` | 13 | 0 |
+| `operator_authorized_terminal_infrastructure_failure` | 9 | 0 |
+| `operator_authorized_verifier_infrastructure_misclassification` | 3 | 3 |
+
+The three score-changing recoveries were all on the Kimi Code arm — `prove-plus-comm`, `hf-model-inference`, and `pypi-server` each first recorded a structured verifier failure, were invalidated by an operator as infrastructure misclassification, and passed on their replacement admissions. The WAL records the invalidation reason and prior admission id for each; the frozen run directory no longer retains the superseded trial directories, so the invalidation and replacement outcomes are verifiable but the operator's original judgment is not independently reconstructible.
+
+The generated report's "178/178 attempted, 0 infraFailedCells" describes the accepted final dataset — not the physical admission count, and not the absence of infrastructure failures during the run. Run status is `completed_with_gaps` because final usage is absent for the Maka `adaptive-rejection-sampler` cell; economy aggregates therefore cover 88/89 metered pairs.
+
+## Throughput: why the absolute scores are a lower bound
+
+Agent budgets were task-native (timeout multiplier 1): 48 of 89 tasks at 900 s, the rest spread from 600 s to 12,000 s. Kills fired at the budget on both arms — Maka cells settle ~30 s early, Kimi Code cells run to the deadline (median 0.6 s past it). Both arms decoded far below what those budgets require:
+
+| | Maka | Kimi Code |
+| --- | ---: | ---: |
+| Effective output tok/s, overall | ~37 (est.) | 34.0 (measured) |
+| Effective output tok/s, median task | ~44 (est.) | 35.4 |
+| Tasks killed at the deadline | 28 / 89 | 33 / 89 |
+| Pass rate when killed | 14.3% (4/28) | 15.2% (5/33) |
+| Pass rate when finished in time | **95.1% (58/61)** | **85.7% (48/56)** |
+
+Deadline-killed tasks were still verified; a minority pass because the deliverable was already in place when the budget ran out.
+
+Measurement notes. Kimi Code figures come from per-request provider-proxy telemetry: 1,267 requests with positive generation spans (of 1,300 recorded), 1.16M output tokens over 9.5 h of measured generation. The Maka path has no request-level telemetry, so its figures are estimated from runtime event-stream spans (first-to-last partial event per generation burst: 1.86M tokens over 13.8 h) — treat them as approximate.
+
+First-token latency on the shared proxy path was heavy: median 13.7 s, p90 62.9 s, worst 1,023 s for a single request. With ~14 model requests per task, first-token waits alone consumed roughly 3 minutes of a 15-minute budget at the median. The mechanism is simple: at ~35 tok/s a task can only emit ~30K output tokens per 15 minutes of budget minus tool time and waits, so long-horizon tasks starve mid-generation — and the same starvation killed tasks even at 1,800–3,600 s budgets.
+
+**Projection.** The official 88.3% was reported for the same product harness our Kimi Code arm ran (KimiCode, max reasoning effort). Our finished-in-time subsets pass at 95.1% (Maka) and 85.7% (Kimi Code); on the 49 tasks both arms finished in time, the rates are 95.9% and 89.8%. Under healthy throughput Maka would plausibly reach or exceed the official score, and Kimi Code would plausibly land near it. These subsets are enriched for shorter tasks, so no exact counterfactual is claimed — a healthy-network re-run is the decisive test.
+
+**Benchmark headroom.** The official frontier is already compressed: the current top two leaderboard scores are 88.8% and 88.3% ([llm-stats](https://llm-stats.com/benchmarks/terminal-bench-2.1), accessed 2026-07-19), and our throughput-unconstrained subsets pass at 85.7–95.9%. For frontier model-plus-harness systems, Terminal-Bench 2.1 has little discriminative headroom left — future comparisons will get more signal from a harder benchmark.
+
+## Setup
+
+| Dimension | Value |
+| --- | --- |
+| Benchmark | Terminal-Bench 2.1, revision `d49e28f1` (89 explicit task ids; 4 designated pilot tasks) |
+| Model | `kimi-coding-plan` / `k3` |
+| Thinking effort | `max` on both arms |
+| Thinking retention | Aligned: prior-turn thinking is kept for subsequent requests (Maka: provider-native signed-thinking replay; Kimi Code: product default keep-all) |
+| Agent time budget | Task-native agent timeout ×1 (48/89 tasks at 900 s; rest 600–12,000 s); 900 s outer setup/teardown grace |
+| Repetitions | 1 |
+| External `MAKA_SYSTEM_PROMPT` | Empty for both arms (see asymmetry 1 below) |
+| Attempt policy | `single` for both arms; infrastructure-invalidated admissions excluded via the audited recovery layer |
+| Maka arm | `maka_agent:MakaAgent`; host runtime LLM with tools bridged into the task container; continuation disabled; **context budget ON**: active + stale tool-result prune at 2048 estimated tokens, semantic compact off |
+| Kimi Code arm | `kimi_code_agent:MakaKimiCodeAgent` 0.26.0 (official CLI, print mode, `stream-json`, `prompt-auto`) inside the container; **no Maka context-budget policy** |
+| Pricing profile | Account-plan frozen at $0 recorded cost (token counts remain meaningful) |
+| Concurrency | 2 task pairs, at most 4 concurrent arm attempts; A/B arms of a pair run on the same provider proxy path |
+
+## Ruled out for the relative gap
+
+- **Thinking effort and retention.** Both arms request max effort and both keep prior-turn thinking for multi-step tool loops. Do not read the gap as "Maka thinks harder" or "only one side keeps thinking history." Maka may show `reasoningTokens = 0` in usage projections while streaming thinking, and the Kimi proxy attributes a large share of output to reasoning — those are metering/projection differences, not misalignment.
+- **Network.** Arms ran concurrently on the same degraded path, and wall-clock is essentially tied: mean ~1030 s vs ~1045 s, median ~753 s vs ~901 s, total ~25.5 h vs ~25.8 h, Maka lower on 65/89 tasks, deadline kills 28 vs 33. Network explains the absolute level, not the relative gap.
+- **Maka heavy-task self-check.** The pinned build includes the headless heavy-task self-check machinery (plan/evidence tools plus a bounded finalization gate), but it is off by default and was not enabled here: no `--heavy-task` / `MAKA_HEAVY_TASK_MODE` in any of the 89 trial configs, an identical system-prompt hash across all 89 Maka cells, provider-visible tool count 7 on every completed cell (would be 11 with heavy-task tools), and zero self-check tool calls or gate events in any transcript. The +10.1 pp gap therefore cannot be attributed to Maka-side self-verification. Enabling heavy-task mode is a plausible follow-up lever on the "bad verification / incomplete deliverable / early stop" failure buckets.
+
+## What differed: harness asymmetries
+
+These are the main product/harness setting differences present in v11. They are listed as alignment gaps behind the +10.1 pp, **not** as a ranked causal chain — this run has no single-factor ablations. The gap persists among finished-in-time tasks (95.1% vs 85.7%), so it is not a differential-timeout artifact.
+
+1. **Context-budget tool-result prune (Maka only — a primary harness setting).** Maka ran with active + stale tool-result prune at a 2048 estimated-token threshold (semantic compact off). Before each later model step, oversized *current-turn* tool results in the provider-visible message list are rewritten so they are not re-sent at full size. Kimi Code's print-mode path has **no** equivalent Maka context-budget policy and no structured prune meter in its transcripts. This is one of the largest intentional runtime-policy differences between the two arms.
+
+   Measured on this run (completed cells only; aborted cells emit no prune diagnostics, so counts are a lower bound):
+   - Policy ON: **89/89** Maka cells
+   - Active prune diagnostics: **21/57** completed cells (**21/89** overall)
+   - **576** active rewrite applications; **~1.87M** estimated tokens not re-sent in later steps (~4.4% of Maka cumulative input)
+   - Stale prune recorded **0** events
+   - Top savings: `video-processing` (~471k), `custom-memory-heap-crash` (~262k), `build-pov-ray` (~203k)
+   - Final `contextRemaining` stayed above ~0.95M where measured — no overflow rescues observed under this decode/timeout regime
+   - Among completed cells, pass rates with vs without prune diagnostics were essentially tied (20/21 vs 34/36); without a prune-off control, score impact is not estimated here. Treat prune as a **documented, live harness asymmetry**, not as a proven explanation of the +9 exclusive wins.
+
+2. **System instructions.** Maka sends an empty system string. Kimi Code still bootstraps its product main-agent profile — ~20KB of interactive coding-agent instructions — regardless of the empty external env var.
+
+3. **Tool surface and shell semantics.** Maka headless: narrow file/shell set with foreground Bash. Kimi Code print default: the full product tool list (tasks, cron, agent/swarm, plan/goal family, web) with product Bash/background behavior. The stock print path has no user-facing "minimal main tools" switch.
+
+4. **Completion contract.** Kimi's product system steers toward telling the user when work is done or blocked; Maka relies on the task instruction plus tool schemas.
+
+Background Bash is a real capability difference but not the score story: among the 12 Maka-exclusive passes, only two Kimi Code traces used `run_in_background` and one used `Task*` tools. Most exclusive failures look like ordinary foreground wrongness — bad verification, incomplete deliverable, early stop.
+
+## Exclusive outcomes
+
+**Maka only (12):** `build-pov-ray`, `configure-git-webserver`, `crack-7z-hash`, `financial-document-processor`, `fix-code-vulnerability`, `gpt2-codegolf`, `mcmc-sampling-stan`, `overfull-hbox`, `regex-chess`, `schemelike-metacircular-eval`, `torch-pipeline-parallelism`, `video-processing`
+
+**Kimi Code only (3):** `feal-linear-cryptanalysis`, `mteb-retrieve`, `tune-mjcf`
+
+The complete accepted pass/fail outcome for every task is in the adjacent [CSV](./terminal-bench-2.1-maka-vs-kimi-code-v11.csv); prompts, payloads, traces, and verifier output are not committed.
+
+## Token economy
+
+Recorded USD cost is **$0** for both arms under the frozen account-plan profile — an accounting placeholder, not an estimate of real cost; compare tokens, not dollars.
+
+| Tokens over 88 metered pairs | Maka | Kimi Code |
+| --- | ---: | ---: |
+| Total | 43.96M | 47.77M |
+| Cached input | 37.57M | 44.03M |
+| Uncached input | 4.53M | 2.58M |
+| Output | 1.86M | 1.16M |
+
+Kimi Code's higher cached input is consistent with its larger product system surface and longer contexts, not a different effort setting. Maka's live tool-result prune (asymmetry 1) recorded ~1.87M estimated tokens not re-sent on completed cells — a lower-bound economy effect of that policy, not a bill delta.
+
+Applying the official K3 API list prices ($0.30/MTok cache-hit input, $3.00/MTok cache-miss input, $15.00/MTok output) to the 88 metered pairs:
+
+| List-price estimate | Maka | Kimi Code |
+| --- | ---: | ---: |
+| Cached input | $11.27 | $13.21 |
+| Uncached input | $13.58 | $7.73 |
+| Output | $27.87 | $17.47 |
+| **Total** | **$52.72** | **$38.40** |
+| Per task | $0.60 | $0.44 |
+| Per accepted pass | $0.85 | $0.72 |
+
+These are list-price equivalents of recorded usage, not actual charges — the run was metered at $0 under the subscription account plan. Maka used fewer total tokens yet costs more at list price because output tokens dominate the price sheet and it emitted 60% more of them; Kimi Code's advantage is its large, cheap cached input.
+
+## Caveats
+
+- One frozen run, `reps = 1`: descriptive, not statistical.
+- The Kimi Code 53/89 headline depends on three operator-recorded verifier invalidations; the 50/89 first-verifier sensitivity is included so the adjustment is visible rather than implicit.
+- This is a harness / product-agent comparison with a shared model, not a "same agent, same system, same tools" model bake-off. Empty external system env ≠ empty product system on Kimi Code.
+- No single product dial was shown to explain the net +9; causes are separated into ruled-out vs asymmetric, not ranked.
+- The projection against the official 88.3% rests on the finished-in-time subsets, which are enriched for shorter tasks and differ between arms; a healthy-network re-run is the decisive test.
+- Missing oracle registry configuration: no oracle-backed disagreement layer.
+- $0 recorded cost must not be read as free or equal economics; use the token table.
+- Raw run artifacts remain local and git-excluded; this document commits only aggregate prose and the redaction-minimal outcome CSV.
+
+## Integrity
+
+SHA-256 of the frozen evidence files (local, git-excluded) and the committed CSV:
+
+| Source | SHA-256 |
+| --- | --- |
+| `harness-ab-manifest.json` | `8ffe15ca48348b10aacde90b569c88b598bfd5bd961540fe77896566b8245258` |
+| `harness-ab-report.json` | `83742970810c2ce864b35fba40180ebe41bdda28efe2f6954a376b3d04653269` |
+| `controller/results.jsonl` | `a094d6096e7f5d519f10b65ff64984c46f561313eed8726f2aaefacdbb4270ce` |
+| `controller/results.jsonl.attempts.jsonl` | `975222c4800d2cc65924116f423fc8ea7327b470de559eaefb6381319e9c209c` |
+| Committed outcome CSV | `e7a2e76f883ab8672e72d347b66cf3392965b4e3e21fbac75eb4a9622af849c8` |
+
+## Artifact pointers
+
+| Artifact | Path |
+| --- | --- |
+| Harness report (JSON) | `~/.maka/eval/runs/k3-maka-vs-kimi-code-tbench-2.1-full-v11/harness-ab-report.json` |
+| Harness report (Markdown) | `~/.maka/eval/runs/k3-maka-vs-kimi-code-tbench-2.1-full-v11/harness-ab-report.md` |
+| Run manifest | `~/.maka/eval/runs/k3-maka-vs-kimi-code-tbench-2.1-full-v11/harness-ab-manifest.json` |
+| Controller WAL (final + attempts) | `.../controller/results.jsonl`, `.../controller/results.jsonl.attempts.jsonl` |
+| Per-request telemetry (Kimi Code arm) | `.../jobs/<run>/ab-kimi-code-r0-<task>/<task>/provider-request-telemetry.json` |
+| Per-cell bridge output (both arms) | `.../jobs/<run>/ab-{maka\|kimi-code}-r0-<task>/<task>/trial/<trial>/agent/maka-cell-output.json` |