tutti/docs/architecture/issue-execution.md

# Issue Execution Coordination

Issue execution combines two independent domains:

- Workspace Issue owns Issue, Task, Run, dependency, acceptance, budget, and
  dispatch-pause facts.
- Agent Host owns Session, Turn, runtime-operation, terminal outcome, and
  lifecycle recovery semantics.

Neither domain mirrors the other's state. `IssueExecutionCoordinator` in
`services/tuttid/service/workspace` is the product-owned integration seam that
maps user intent and canonical Agent facts into Issue commands.

This generic dispatch flow applies to manual and `traditional_plan` Issues.
Accepted `tutti_mode_plan` Issues instead materialize atomically with a
Tutti-owned execution aggregate and active `initial_schedule` checkpoint.
Their materialization creates no Run and never enters the generic eligible-task
dispatcher; later work requires an explicit Tutti execution schedule command.
Every active Tutti checkpoint instead owns one durable main-conversation wake.
The wake asks the source Agent to review canonical execution state and choose a
fenced `schedule`, `mutate`, `acknowledge`, `complete`, or `stop` command;
settling a task never mechanically dispatches a successor.

The source-facing `plan issue get` projection is a read-only composition of the
Tutti execution aggregate and canonical Issue task detail. It exposes the
active checkpoint and graph revision together with launch configuration,
supersession, dependency readiness, allowed actions, and recovery guidance.
The projection is scoped by trusted Agent Session context; callers cannot
supply a source Session ID.

`dispatchPaused` is also a Tutti automation delivery gate, not only a generic
task-launch gate. While it is true, watchdog sweeps retain existing checkpoint
and wake history but do not create a later wake sequence, list or claim a
prepared main wake, or finalize a wake that raced with the pause. Resuming
therefore continues from the same active checkpoint and graph revision instead
of inventing replacement orchestration state. The original source Agent may
resume through `plan issue resume`; the compatibility form
`issue update --dispatch-paused=false` routes to the same source-scoped product
control for Sessions whose frozen command snapshot predates the dedicated
command. Neither path weakens the generic managed-Issue mutation guard.

There is one delivery-race exception to retaining the exact wake identity. If
pause wins the final dispatch CAS after Agent Host has already accepted a
canonical Turn, Tutti cancels that Turn. Once cancellation succeeds, it
atomically marks the leased wake canceled and prepares the next wake sequence
under the same active checkpoint. The new `clientSubmitId` is required because
Host idempotency must never execute an already accepted submit identity twice.
The replacement wake remains hidden while paused and is immediately eligible
after source-scoped resume. If Turn cancellation fails, Tutti retains the lease
until expiry rather than risk overlapping Turns.

Graph transition semantics belong to
`services/tuttid/biz/tuttimodeexecution`. That business layer applies
presence-aware updates, sparse rework inheritance, logical supersession,
dependent-edge rebinding, and active-graph validation as one pure transition.
The SQLite adapter only loads the graph, persists the returned inserts and
updates inside the fenced transaction, and records idempotency history. Product
mutation rules must not be reimplemented in the data adapter.

## Execution flow

Generic Issue dispatch is split into two phases:

1. Under the per-Issue mutation lock, Issue Manager rechecks policy and creates
   a durable running Run. That Run is the claim that prevents duplicate
   dispatch.
2. After releasing the lock, the Issue run launcher prepares any worktree and
   creates the Agent Session. A launch failure settles the claimed Run through
   the normal idempotent completion command.

The source planning Session supplies two independent launch facts: its working
directory is the fallback execution base, while its canonical rail placement
is the delegate Session's logical project/conversation ownership. Preparing an
isolated task worktree changes only the execution directory. Issue dispatch
must carry the source `RailPlacement` unchanged into the Agent Host create
contract and must never infer placement from the temporary worktree path.
When an Issue names a source Session, that projection is a launch
precondition: an unavailable/deleted source leaves the Task unclaimed for a
later reconcile instead of creating a detached Run and Session. Each dispatch
pass resolves the source once, so worktree planning and Agent creation consume
the same immutable source snapshot. A project placement with an empty runtime
cwd falls back to its canonical project path.

Stopping is also split:

1. Under the Issue lock, set `dispatchPaused=true` and snapshot the Issue's
   running Runs.
2. After releasing the lock, request cancellation of each bound Agent Session
   by resolving the exact `issue-run:<runID>` Turn.
3. Settle a Run as canceled only from exact canonical Turn settlement, or from
   a typed adapter result that carries authoritative canceled evidence.

Agent cancellation may synchronously publish a canonical settled-Turn fact.
Because no Issue lock is held across the Agent call, that callback can safely
settle the Run. A failed cancellation leaves the Run running, keeps dispatch
paused, and returns an error; Issue intent must not fabricate Agent outcome.

Stopping a Tutti source conversation uses a stronger, source-scoped boundary.
The canceled canonical source Turn is committed with the Agent transaction's
Tutti activity-inbox marker. The synchronous observer asks the execution
service to archive every nonterminal execution for that workspace/source
Session; it does not discover ownership by scanning a bounded page of running
Runs. In one product transaction, archive fences each execution and Issue,
cancels checkpoints, main wakes, Goal Reviews, prepared or leased Run launches,
and pending workflow/checkpoint/operation recovery. It then cancels exact Run,
main-wake, and reviewer Turns before sealing executions as `archived`.

The same inbox marker is the crash boundary. A watchdog or reviewer sweep
drains it before dispatch, creates any missing archive operations, and runs
archive recovery before another automatic send. Startup create-Issue recovery
also excludes workflows whose source has an undrained canceled-Turn marker.
Only completed source Turns advance the inactivity clock; a canceled source
Turn terminates orchestration instead of postponing the watchdog.

The non-blocking Run launch gate closes the claim-to-launch race without
holding a mutex across external work. Stop records cancel intent and returns
without waiting for an in-flight Agent create call. Launch revalidates the
durable Run and Issue pause fact before external work; when it completes, it
observes any concurrent cancel intent and performs exact-Turn compensation. If
pause wins before launch begins, the unlaunched claim is canceled without
creating an Agent Session.

Tutti-owned launch intents add a cross-process fence to that local gate. Run
terminalization seals any `prepared` or `leased` intent in the same database
transaction as its settlement checkpoint; prepared scans and lease claims also
require the Run to remain `running`. `MarkDispatched` is a strict current-owner
CAS and is the successful delivery linearization point. If an external create
returns after another replica terminalized the Run, either success or an
ambiguous error performs exact-Turn cancellation. A stale owner whose lease was
reclaimed while the Run is still running does not cancel the valid same-ID
Turn; it schedules reconciliation instead. Startup repair repeats the intent
seal even when the settlement checkpoint already exists. When a leased or
dispatched launch may have created a Turn, the settlement transaction also
prepares a durable cancel-compensation operation. Those operations have their
own owner/lease/attempt state, use the same deterministic submit identity, and
are retried on startup and the regular reconciliation cadence until exact
cancellation is accepted. Compensation uses a bounded context detached from
the original delivery cancellation signal. A retryable cancellation outcome
keeps the operation prepared and queues another pass without aborting startup
or starving launch/running-Run recovery in the same workspace. A successful
stale owner also revalidates the Run before compensating: lease reclaim while
the Run remains running is recovery work, not a cancellation signal.

Tutti main-conversation delivery follows the same durable-operation pattern.
Creating or promoting an active checkpoint prepares its wake in the same
SQLite transaction. The canonical identities are
`<checkpointID>:wake:main:1` and
`clientSubmitID=tutti-execution-wake:<wakeID>`; they are interoperability
contracts, not presentation values. An existing row with different checkpoint,
target, sequence, client-submit, or source-session identity fails closed.

Delivery first checks the exact workspace/source Session and leaves the wake
prepared while that Session is busy. A daemon-unique owner leases the wake,
then sends through Agent Service while Host remains the authority for canonical
Session liveness and `clientSubmitID` lookup. Response loss is recovered by
looking up the same deterministic submit ID. Every external send has a bounded
deadline shorter than its remaining lease. Canonical lookup, release, integrity
failure, and final dispatch CAS use their own bounded contexts detached from
caller cancellation, so cancellation cannot strand the row in `leased` after
external work has returned. They are cleanup bounds, not permission to continue
an unbounded Agent call.

Recovery recomputes execution, wake, client-submit, target, sequence, and source
Session identity from durable Issue/checkpoint evidence before claim and again
before delivery. Corrupt identity fails closed and is recorded as `failed`;
immutable workspace/execution/checkpoint relationships are also protected by
the wake table's composite foreign key. Main recovery lists only `main` targets;
a separate integrity scan recognizes deterministic main-form identities whose
target kind was corrupted. Goal Review delivery has its own durable review
operation and lease and therefore is never claimed by the main-wake worker.
Only a lease owner whose lease is still live may enter `SendInput` or win the
final dispatch CAS. The final CAS uses the injected product clock, so a lease
that expires during the external call cannot be finalized by its stale owner.
These fences make restart or replica races converge on one canonical Turn.

The exact Session/Turn settlement changes the wake to `turn_settled`, but does
not resolve the checkpoint: only a correctly fenced checkpoint command may
atomically acknowledge that wake and promote the next backlog checkpoint.

Every nonterminal Tutti execution also carries a fixed inactivity deadline:
`watchdogDueAt = lastOrchestratorActivityAt + 5m`. Valid schedule and
acknowledge commands, Run settlement (including failed or canceled), exact
source-session user input, and exact completed source root-Turn settlement
reset that deadline. A canceled source Turn enters the terminal source-session
stop boundary instead. Delegate/child activity and activity from another
workspace or Session do not. The Agent adapter only projects these facts after Host accepts
the user Turn or reports the canonical root Turn settled; it does not change
Host lifecycle semantics. The projection carries the canonical Turn identity
and user-message occurrence or root-Turn settlement time rather than callback
wall time. SQLite advances the deadline only for a strictly newer event time,
so at-least-once replay and out-of-order delivery cannot drift it. Internal
watchdog prompts are excluded only for their exact target source Session;
reusing the same submit identity in another Session remains user activity.

When a deadline becomes due with no active checkpoint, the daemon atomically
creates a `watchdog` checkpoint and its deterministic sequence-1 main wake. If
the awakened Turn settles without a fenced command, the checkpoint remains
active and the next fixed deadline creates sequence 2 (then 3, and so on) on
that same checkpoint, each with the normal deterministic wake and
`clientSubmitID` identity. A valid command acknowledges every wake for the
resolved checkpoint; later inactivity creates a new watchdog checkpoint
rather than another generation on resolved history.

An Issue dispatch pause freezes this wake loop without resolving or canceling
its active checkpoint. The due deadline may remain in the past, but no wake
sequence is materialized or delivered until the source-scoped resume control
clears the durable pause. The next worker sweep then continues normal recovery
from the retained operation.

Source activity that arrives while a prepared or leased watchdog wake is
suppressed moves that same durable operation to the new five-minute deadline;
it does not replace its identity. A busy source Turn, an already dispatched
main wake, or an active independent reviewer suppresses duplicate delivery.
Reviewer state is read through an injected port and is never written by the
watchdog. Infrastructure leases and retries do not move either the execution
deadline or wake due time.

Accepted source-user submissions and settled source Turns do not depend on a
best-effort post-commit callback for this clock. The canonical Agent SQLite
transaction atomically appends a product-owned activity-inbox marker for the
exact message or Turn mutation. Every watchdog sweep drains those markers
before due materialization, resolves the canonical user-message occurrence or
root-Turn settlement timestamp, and advances the execution clock
monotonically. A transaction cannot commit the canonical activity while
silently losing its marker. Replays are harmless, and user messages whose
`clientSubmitID` and Session match the target of a durable Tutti main wake are
excluded so an internal prompt cannot debounce itself. Direct observers remain
low-latency wake hints only.

When every task is terminal, settlement atomically promotes an
`all_tasks_terminal` checkpoint and changes the execution to
`pending_goal_review`. Completion is never inferred from task count. In `self`
mode the checkpoint owns a normal main wake. In `independent` mode the same
transaction instead prepares one deterministic Goal Review operation:
`reviewID=<checkpointID>:review`, a deterministic reviewer Session, and
`clientSubmitID=tutti-goal-review:<reviewID>`.

The independent reviewer runs in a hidden, strict read-only Session. Runtime
preparation uses an exact capability allowlist: canonical Issue, Task, Run, and
output reads plus the structured
`tutti-goal-review.goal-review.verdict` integration capability. Tutti plan,
Agent control, dynamic app commands, generic Issue/Task/Run mutation, browser,
and computer capabilities are absent. That projection is part of the immutable
Session runtime snapshot so a resumed reviewer receives the same command
surface. The reviewer may inspect canonical evidence, then submit exactly one fenced
`goal_satisfied`, `more_work_required`, or `inconclusive` verdict. A verdict
must match the exact review Session, Turn, checkpoint, graph revision, and
idempotency request.

For session-aware CLI requests, the daemon resolves this projection from the
canonical Agent Host Session and applies it to both capability listing and
command invocation. A missing resolver, unreadable Session, or malformed
snapshot fails closed. Codex runtime preparation also narrows automatic CLI
approval from the whole binary to the exact projected command paths.

This is a product-policy and defense-in-depth boundary, not hostile-process
isolation. The local daemon bearer credential is shared with same-OS-user CLI
clients, and a same-user process that can read the global listener-info file
can deliberately make a sessionless request. Therefore the independent
reviewer contract assumes the configured provider runtime follows Tutti's
session environment and permission policy. It does not claim containment
against a malicious reviewer that strips its environment or reads and reuses
the global credential. Closing that stronger threat model requires a separate
process/filesystem isolation design or removal of unscoped CLI credentials; it
must not be inferred from command projection alone.

Reviewer dispatch uses deterministic Session and submit identities with
canonical Host lookup for response-loss recovery. Verdict admission may race
the dispatch response: while the delivery lease is current it atomically binds
the canonical Turn and records the verdict, and the later dispatch mark is an
idempotent confirmation. Conversely, a Turn observed settled before that bind
fails closed rather than losing a verdict-less settlement. A reviewer Turn
that settles without the structured command also fails the review. Both a
submitted verdict and a failed review atomically prepare the main wake, which
includes the reviewer evidence.

The independent verdict is advisory but cannot be silently ignored.
`more_work_required` and `inconclusive` require an audited disagreement reason
before the source Agent may complete. A failed independent review requires an
explicit audited user action to switch the execution to `self`; there is no
automatic fallback. The final `complete` command is accepted only from the
source Session at the exact active Goal Review checkpoint and graph revision.
It resolves that checkpoint and marks the execution `completed` atomically.
Choosing more work instead changes and schedules the graph through the normal
fenced plan commands.

Goal Review lifecycle remains separate from watchdog delivery. Its activity
reader treats only durable `prepared` or `dispatched` reviews as active
duplicate-suppression ownership; the watchdog never creates or advances a
review row. The daemon worker performs reviewer recovery on its normal sweeps,
including its first listener-ready sweep, so a restart cannot strand a prepared
review.

A leased wake is checked against its current durable due time immediately
before `SendInput`. Source activity in the claim-to-send window therefore
releases the same operation without sending. The final dispatched transition
also compares the due time observed before sending, so activity during the
idempotent Agent submission prevents a stale owner from finalizing an older
generation. A newer canonical marker fences either transition only while its
derived five-minute deadline remains in the future; an already-due marker
cannot strand the wake lease and a later drain does not create another
canonical Turn.

Startup and periodic recovery also inspect dispatched wakes. Using only each
wake's persisted source Session and `clientSubmitID`, the Agent adapter queries
the canonical Turn without resuming its provider. A settled Turn repairs a
lost observer projection with its canonical settlement time, allowing the
next fixed five-minute wake sequence to resume. Pending-only recovery markers
remain quiet, while independent integrity or persistence errors joined with a
pending retry are still reported and do not stop later worker sweeps.

Execution states `orphaned_source`, `completed`, `archiving`, and `archived`
suppress every still-open wake, including `turn_settled`. Suppression is stored
as `canceled` and clears any lease rather than deleting recovery evidence.

## Identity and settlement

Every dispatched Run stamps `clientSubmitID=issue-run:<runID>`. A settled Agent
Turn may complete a Run only when the coordinator resolves that submit ID and
the exact initiating Turn ID matches the settled Turn ID.

Missing, failed, or ambiguous identity resolution is fail-closed: the Run
remains running and reconciliation is scheduled. Reconciliation combines the
same `FindTurnByClientSubmitID` and canonical `GetTurn` queries to recover the
exact settled fact. A different Turn in the same Agent Session must never
settle the Run.

The coordinator consumes `IssueRunSettlement`, a narrow typed fact. Translation
from Agent canonical projection DTOs is isolated at the coordinator adapter;
Issue Manager does not interpret Agent Session or Turn state.

## Lock and transaction rules

The per-Issue mutex serializes local read-modify-write commands in one daemon
process. It must never be held while invoking Agent Host, creating a git
worktree, notifying another Agent conversation, or performing another
potentially re-entrant cross-module action.

The mutex is not a durable transaction boundary. Store commands still need
database-level atomicity or revision/CAS protection for invariants spanning
Run, Task, Issue projection, and budget. Until those store commands are
introduced, the mutex remains a local serialization aid and must not be
described as sufficient cross-process correctness.

## State model

This flow does not need a general-purpose state-machine framework. Durable
facts remain small and direct:

- Issue: `dispatchPaused`, execution policy, budget
- Task: status, acceptance state, latest Run
- Run: running or terminal outcome, Agent Session binding
- Goal Review: mode, reviewer operation/lease, structured verdict, audit
- Agent Host: Session and Turn lifecycle

UI and orchestration phases are derived from those facts. New boolean flags
must not be used to simulate transactions or hide incomplete cross-domain
operations.

The public Issue projection includes `dispatchPaused`. Conversation surfaces
must combine that durable gate with task terminality when presenting aggregate
Tutti execution; Issue rollup status alone is insufficient because a stopped
graph intentionally retains untouched `not_started` successors.

## Recovery

The reconciliation queue is daemon-context-bound and retries transient
failures. It is a fallback for delayed or missed projection delivery, not the
authority for Agent lifecycle semantics. Product timeouts may fail an Issue
Run, but Agent terminal outcomes should come from exact canonical Turn facts.

Durable main wakes enter that queue after Tutti Issue materialization and Run
settlement. Root-Turn settlement also enqueues rather than sending inline, so a
source conversation that was busy is reconsidered without re-entering Agent
delivery from the projection callback. Every queue pass first performs
idempotent suppression and expired-lease repair, then attempts delivery.
Integrity, Session observation, claim, and delivery failures are isolated per
wake: the pass continues through later executions in the workspace and returns
their joined errors afterward. Released delivery failures return a pending
signal so the existing bounded queue cadence retains the workspace even when it
has no running Runs.

Legacy Tutti Issue migration is classification, not dispatch. A legacy
execution remains `running` only when an authoritative Run is still running;
an idle execution becomes `completed`, and a missing source becomes
`orphaned_source`. Its migration checkpoint is terminal and must never create a
main wake. Settlement repair may still seal terminal Run launch bookkeeping,
but it must not create a checkpoint or wake for an `orphaned_source`,
`completed`, `archiving`, or `archived` execution.

The V6 legacy-recovery cleanup recognizes only V5-owned evidence with
`kind = migration` and
`creation_reason = legacy_execution_startup_repair`. It cancels open recovery
wakes and reviews, supersedes the pending/active recovery chain, and normalizes
the execution from authoritative running Runs. This forward migration prevents
already-upgraded local databases from replaying the former V5 startup wake.

During daemon construction, startup performs only the local durable repair.
It does not call Agent `SendInput` or start the queue while CLI routes and the
listener are unavailable. A one-shot listener-ready hook enqueues the
workspaces after listener information has been published. A readiness gate also
turns any earlier internal enqueue into a pending retry without reaching
`SendInput`. A transient repair or Session observation is retained for queue
retry instead of preventing the daemon from serving other workspaces. Startup
never reclaims an unexpired lease owned by another process.

After listener readiness, a daemon-owned watchdog worker scans every current
workspace on a short infrastructure cadence. Each scan materializes all
expired nonterminal deadlines and then delegates delivery to the same durable
main-wake recovery path, then recovers prepared independent Goal Reviews.
Startup requeues only expired leases, preserves live owners, and performs the
same all-execution scan; terminal and orphaned executions never create a
watchdog operation. The scan cadence is not a backoff policy—the persisted
product interval remains exactly five minutes.

The former in-memory Tutti Issue completion notifier and dispatcher are not
orchestration authorities. Checkpoint/wake rows plus canonical Agent Host
queries are the restart-safe source of truth.