tutti/docs/conventions/unit-testing.md

Unit Testing

This document contains Tutti's durable test-design rules. The executable authoring and review workflow lives in $tutti-test-audit; test discovery, commands, changed-aware validation, performance scenarios, and Session Replay live in Testing.

Apply these rules to new and materially rewritten tests. They do not require a mass rewrite of legacy suites, but a touched test must not preserve a known low-value pattern without reason.

Quality bar

A test earns its place when it protects a product contract that a plausible implementation mistake could violate. It must execute the production owner and observe a stable result at a boundary capable of exposing that mistake.

Start with this statement:

When ACTION occurs under CONDITION, the owner must produce OUTCOME while
preserving or avoiding NON_EFFECT. A faulty implementation that REGRESSION
must make this test fail.

A merge-worthy test has all of these properties:

• Contract: it protects observable behavior, an invariant, compatibility, or a previously observed failure that matters.
• Sensitivity: a credible wrong implementation, including the old bug for a regression, fails an assertion for the intended reason.
• Fidelity: it invokes the production owner through the lowest credible boundary instead of reproducing or mocking away the behavior.
• Oracle: it asserts stable outcomes and important non-effects rather than incidental structure.
• Determinism: time, concurrency, randomness, state, and external resources produce the same verdict on a busy runner.
• Execution: the blocking repository lane discovers the test, and every platform-specific arm runs on its native platform.

Adding no test is preferable to adding evidence that only restates a constant, type, mock arrangement, or presentation detail. Test count, assertion count, and line coverage are diagnostic metrics, not quality verdicts.

Choose the lowest credible boundary

“Unit” describes isolation, not importance. Use the lowest level that can observe the named risk without replacing it.

Level	Appropriate risk	Boundary that remains real
Pure unit	Parsing, normalization, projection, policy, deterministic algorithms	None
Behavioral component	One controller, store, service, or UI interaction	Internal production wiring
Conformance	A contract shared by implementations or consumers	Public contract and driver
Narrow integration	SQLite, filesystem, Git, HTTP, IPC, parser, subprocess, or OS semantics	The boundary named by the risk
End to end	A journey through a composition root	Public entry point to user-visible result
Visual or performance	Pixels, layout, animation, responsiveness, or timing budgets	Real renderer or trace
Repository check	Imports, generated artifacts, manifests, ownership, or source architecture	Repository files themselves

A test named E2E that bypasses the composition root and replaces all internal collaborators is a component test. A colocated test that launches a real subprocess is an integration test.

Prefer one vertical proof through the owning boundary plus narrow unit cases for important decisions. A helper-only unit test cannot prove that production callers still invoke the helper. Tutti has encountered this exact failure with remote-image materialization; see the troubleshooting entry.

Tutti routing rules

• New Session, Turn, Goal, runtime-operation, or recovery semantics start as a scenario in packages/agent/host/conformance. Consumer adapters implement drivers; they do not clone lifecycle semantics.
• Adapter tests own transport, DTO, authorization, query, presentation, and product-policy behavior.
• SQLite, filesystem, paths, Git, command quoting, executable discovery, process lifecycle, IPC, and native API behavior require a real isolated instance of the relevant boundary.
• UI behavior belongs in a behavioral component test that drives accessible keyboard, pointer, focus, or state transitions. Pixels and layout belong in visual evidence.
• HTTP and provider protocol behavior should cross the real parser, middleware, codec, or receiver when that behavior is the risk.
• Host filesystem, shell, process, registry, signal, permission, and executable behavior must run on the native OS. A request-shape assertion cannot replace the receiving Windows API or process boundary.

Design from risk

Identify the module that owns the decision and the boundary where a caller can observe it. Stable observations include:

• returned domain values and typed errors;
• canonical or persisted state read through the owning store;
• emitted events and observable ordering;
• the final provider, HTTP, IPC, parser, or process request;
• filesystem contents, permissions, and executable behavior;
• accessible UI state, callbacks, enabled actions, and focus;
• absence of a write, dispatch, retry, duplicate, secret, or leaked resource.

Select only input partitions that can change the decision or represent a real failure mode: missing and malformed input, exact boundaries, identity reuse, ordering, partial failure, response loss, restart, cancellation, competing writers, preservation of unrelated state, and native-platform differences.

One test may use several assertions when they jointly prove one behavior. Test names should state cause and observable effect, such as retries transient catalog failures but not permanent failures, rather than works or a private method name.

Prove regression sensitivity

Capture a confirmed bug's failing reproduction before the fix whenever safe. The resulting regression test must fail on pre-fix behavior for the intended reason. If the old revision cannot run, temporarily restore or inject the faulty decision as a local negative control; do not commit that mutation.

For new behavior, first show that the focused test fails because the contract is missing, not because test setup, imports, or fixtures are broken. When sensitivity is not obvious, include the negative-control evidence in the PR.

Assert contracts, not implementation shape

Prefer observations in this order:

1. externally visible result or typed error;
2. durable state and preservation of unrelated state;
3. event or protocol sequence at an owned boundary;
4. side-effect count when count is itself the contract;
5. a narrow internal observation only when no stable boundary exists.

Failure-path tests should normally assert both what happened and what did not: no provider request, no canonical row, no rollback of an earlier commit, no stale output, no secret in diagnostics, or no new identity during a retry. does not throw or err == nil alone is rarely a sufficient oracle.

Prefer invariants to change detectors

A change detector freezes data expected to evolve: an exact catalog size, current model name, schema-version literal, copied defaults object, snapshot of a registry, or duplicated lookup table. Protect the relationship consumers rely on instead: every selectable model has required metadata, migrations end at the declared current version, identities stay unique, or forbidden sets do not overlap.

Use snapshots only for deliberately stable public serialization, generated contracts, or normalized CLI output. Keep them small, normalize paths, IDs, time, width, and platform noise, and pair them with semantic assertions.

Keep source contracts out of runtime unit tests

A runtime test must not read production .go, .ts, or .tsx text and use a string, regex, or indexOf as proof that a call or option is wired correctly. Execute the runtime seam or extract the real decision into its owner.

When source shape is intentionally the contract—imports, generated files, manifests, architecture, or security policy—implement a named repository check under tools/scripts and register it with the repository-check infrastructure. That check is valuable, but it is not runtime unit coverage.

Use real owned resources and narrow doubles

Use temporary real implementations when their semantics are part of the risk: SQLite for transactions and recovery, a filesystem for paths and atomic writes, a local HTTP server for redirects and streaming, a temporary Git repository for worktree behavior, and a subprocess or parser for quoting and wire contracts.

Use a double when the collaborator is externally owned, nondeterministic, unsafe, or irrelevant to the current decision. A double should implement the smallest owning port, return explicit configured outcomes, fail immediately on unexpected calls, and capture only contract observations. Never make every method silently succeed. A test that stubs every layer and verifies that the stubs called one another proves its arrangement, not the product.

Isolate state before production imports can capture it. Tests must not read or write real Tutti state, credentials, provider configuration, user Git config, or developer services. Always restore environment variables, clocks, globals, listeners, timers, stores, sockets, subprocesses, and temporary resources, including on failure.

Make concurrency deterministic

Express the required happens-before relationship with signals:

1. operation A reaches a named gate;
2. operation B is submitted while A is held;
3. the test observes the contract in that state;
4. the gate is released;
5. both operations settle and final state is asserted.

Use channels, barriers, latches, deferred values, or domain events. A timeout is a generous deadlock guard around an awaited signal, not readiness evidence or the behavioral oracle. “Nothing happened for 20 ms” does not prove that nothing can happen.

Inject clocks, timers, deadline errors, random sources, IDs, or schedulers when the decision depends on them. Test immediately before, at, and after thresholds. Retries may expose a flake, but they do not repair it; a pass-on-retry must remain visible as a defect.

Preserve suite and CI integrity

• Keep small suites beside their owner. Split large lifecycle, recovery, concurrency, migration, rendering, or platform scenarios by behavior rather than creating one file per private helper or one giant suite.
• Keep tests order-independent. Parallelize only when they do not share mutable globals, environment, ports, clocks, files, or fixtures.
• Every test must be discovered by its package command and the appropriate root changed-aware lane. A selected suite that collects zero tests must fail.
• Empty files, committed .only, required missing fixtures, and suites whose meaningful cases all skip are defects.
• Platform markers, build tags, and filename filters must map to a blocking native CI lane.
• Keep unit tests offline. Live-provider qualification belongs in an explicit, isolated lane or sanitized record/replay workflow.

Coverage can identify unvisited branches, but a repository-wide percentage rewards shallow render tests, copied constants, and mocked call graphs. For critical lifecycle, authorization, persistence, recovery, and security logic, targeted fault injection or mutation sampling is a stronger audit.

Repository examples

Use these as patterns for their specific risks, not as templates to copy:

• Agent Host conformance expresses lifecycle behavior once for multiple consumers.
• Managed runtime tests distinguish transient and permanent failures, use a local HTTP server, and assert retries, cancellation, and response cleanup.
• Combobox behavior drives keyboard and pointer actions and observes selection, disabled behavior, filtering, accessibility, and closure.
• Windows Codex contract checks behavior against the real Windows receiver instead of trusting a serialized request shape.