fix(company): stop failed children from wedging the tree and soften the dispatch guard
Root-cause fix for the project-4444 class of deadlocks: any FAILED/ CANCELLED work item with downstream dependents used to block its whole tree forever, because the advancement gate required all dependencies to be APPROVED and nothing ever propagated or triaged failures. Settlement mechanism (work_item_transition.py): - compute doomed set (FAILED/CANCELLED seeds contagious through hard deps); settlement-released cards are treated as alive - three-way advancement gate: all-approved (unchanged) / settled-with- failures releases the nearest decision-capable card (manager parent -> failure-triage synthesis turn, rollup delivery/aggregate -> READY) with an atomic dependency_settlement stamp; released cards never oscillate back - claim/park/resume/dispatcher-tick all honor the stamp: runnable gates admit settled failed+stuck deps, parking excludes settled deps and re-arms triage when a failure raced the park, engine resume no longer re-locks released cards, the dispatcher tick releases rollup cards created after the failure - settlement cascade: once the settled card is APPROVED, stuck children the manager did not rebuild are cancelled (transitive closure over stamped stuck seeds, retried until every cancel lands) - info-class deps never block settlement; adaptive runnable gate now shares DEPENDENCY_CLASS_DEFAULT with the release gates Dispatch guard (company_mode.py): - NO_DELEGATION_JUSTIFICATION parsing tolerates markdown decoration (bold/lists/quotes/full-width colon) and rejects placeholder echoes across all artifact/metadata/content channels - retries exhausted no longer FAILs the work item: dispatch is a soft constraint, so the turn output is accepted as normal completion and annotated via manager_dispatch_guard_unresolved; the reminder loop is unchanged, and the mutation flag is now reset per turn so one past delegation can never mute future reminders - manager board context now surfaces failed/cancelled children with their preserved output and pending-cancellation stuck list so the triage turn can rebuild, accept partial results, or escalate Self-produced delegation output goes through review (persisted fact, single predicate): - the DONE transition classifies what a dispatch/intake/plan turn actually delivered from store ground truth (live children => delegated, none => self_produced) and persists turn_output_kind/-source on the WorkItem - is_manager_reviewable_turn honors the persisted marker, so the DONE routing, report spawn, report completion and recovery scans all read the same fact — this closes a pre-existing hole where NO_DELEGATION_JUSTIFICATION output auto-approved with no review at all - escalation requires a real agent manager above; top seats reporting to the human owner keep auto-approve (covered by final delivery's human acceptance) instead of minting unclaimable review cards - dispatcher tick reconciles reviewable cards stuck in AWAITING_MANAGER_REVIEW with no live report/review card by rebuilding the report card idempotently (legacy DBs, crash windows) Verified: 4444 tasks.db replay unwedges end to end; real-store report->review chain exercised without lifecycle mocks (mutation check confirms the tests bite); full suite failure set identical to a same-session HEAD baseline run (all remaining failures pre-existing or environment flakes reproduced at HEAD). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
LZH-YS1998
@@ -11,6 +11,7 @@ from opc.core.events import EventBus
|
||||
from opc.core.models import CompanyMemberSession, DelegationWorkItem, Phase, SeatState, Task, TaskResult, TaskStatus
|
||||
from opc.database.store import OPCStore
|
||||
from opc.layer2_organization.communication import CommunicationManager
|
||||
from opc.layer2_organization.phase import DONE_PHASES
|
||||
from opc.layer2_organization.company_mode import CompanyWorkItemExecutor
|
||||
from opc.layer2_organization.company_runtime import CompanyRuntime
|
||||
from opc.layer2_organization.org_engine import OrgEngine
|
||||
@@ -837,6 +838,77 @@ class ActorRuntimeManagerDispatchGuardTests(unittest.IsolatedAsyncioTestCase):
|
||||
"This turn is a root-only scoping decision before any downstream split.",
|
||||
)
|
||||
|
||||
async def test_manager_dispatch_guard_accepts_markdown_decorated_justification(self) -> None:
|
||||
"""Project-4444 regression: the CTO wrote the escape line as
|
||||
`**NO_DELEGATION_JUSTIFICATION**:` and the bare startswith parser
|
||||
missed it, driving the work item to FAILED. Decorated variants must
|
||||
all parse."""
|
||||
variants = [
|
||||
("**NO_DELEGATION_JUSTIFICATION**: The task needs my own research.",
|
||||
"The task needs my own research."),
|
||||
("**NO_DELEGATION_JUSTIFICATION: Whole line is bold.**",
|
||||
"Whole line is bold."),
|
||||
("- NO_DELEGATION_JUSTIFICATION: Listed as a bullet.",
|
||||
"Listed as a bullet."),
|
||||
("### NO_DELEGATION_JUSTIFICATION: Written as a heading.",
|
||||
"Written as a heading."),
|
||||
("NO_DELEGATION_JUSTIFICATION： Fullwidth colon variant.",
|
||||
"Fullwidth colon variant."),
|
||||
]
|
||||
for content, expected in variants:
|
||||
with self.subTest(content=content):
|
||||
self.task.metadata.pop("manager_no_delegation_justification", None)
|
||||
before = await self.executor._snapshot_manager_dispatch_state(self.task)
|
||||
issues = await self.executor._enforce_manager_dispatch_guard(
|
||||
self.task,
|
||||
TaskResult(status=TaskStatus.DONE, content=f"Some preamble.\n{content}"),
|
||||
before_state=before,
|
||||
)
|
||||
self.assertEqual(issues, [])
|
||||
self.assertEqual(
|
||||
self.task.metadata["manager_no_delegation_justification"],
|
||||
expected,
|
||||
)
|
||||
|
||||
async def test_manager_dispatch_guard_ignores_echoed_instruction_template(self) -> None:
|
||||
"""Echoing the guard's own `NO_DELEGATION_JUSTIFICATION: <specific
|
||||
reason>` template is not a justification — including with trailing
|
||||
punctuation or quoting around the placeholder."""
|
||||
echoes = [
|
||||
"I should finish with `NO_DELEGATION_JUSTIFICATION: <specific reason>` next time.",
|
||||
"NO_DELEGATION_JUSTIFICATION: <specific reason>.",
|
||||
'NO_DELEGATION_JUSTIFICATION: "<specific reason>"',
|
||||
"NO_DELEGATION_JUSTIFICATION: **<specific reason>**",
|
||||
"NO_DELEGATION_JUSTIFICATION: _<specific reason>_",
|
||||
]
|
||||
for content in echoes:
|
||||
with self.subTest(content=content):
|
||||
self.task.metadata.pop("manager_no_delegation_justification", None)
|
||||
before = await self.executor._snapshot_manager_dispatch_state(self.task)
|
||||
issues = await self.executor._enforce_manager_dispatch_guard(
|
||||
self.task,
|
||||
TaskResult(status=TaskStatus.DONE, content=content),
|
||||
before_state=before,
|
||||
)
|
||||
self.assertEqual(len(issues), 1)
|
||||
self.assertNotIn("manager_no_delegation_justification", self.task.metadata)
|
||||
|
||||
async def test_manager_dispatch_guard_ignores_placeholder_in_artifacts(self) -> None:
|
||||
"""The artifact/metadata escape hatch gets the same placeholder
|
||||
filter as the free-text line."""
|
||||
before = await self.executor._snapshot_manager_dispatch_state(self.task)
|
||||
issues = await self.executor._enforce_manager_dispatch_guard(
|
||||
self.task,
|
||||
TaskResult(
|
||||
status=TaskStatus.DONE,
|
||||
content="Done.",
|
||||
artifacts={"no_delegation_justification": "<specific reason>."},
|
||||
),
|
||||
before_state=before,
|
||||
)
|
||||
self.assertEqual(len(issues), 1)
|
||||
self.assertNotIn("manager_no_delegation_justification", self.task.metadata)
|
||||
|
||||
async def test_manager_dispatch_guard_rejects_no_delegation_for_collab_infra_failure(self) -> None:
|
||||
before = await self.executor._snapshot_manager_dispatch_state(self.task)
|
||||
|
||||
@@ -941,6 +1013,256 @@ class ActorRuntimeManagerDispatchGuardTests(unittest.IsolatedAsyncioTestCase):
|
||||
|
||||
self.assertEqual(issues, [])
|
||||
|
||||
async def test_manager_dispatch_guard_exhaustion_accepts_turn_instead_of_failing(self) -> None:
|
||||
# Dispatch is a soft constraint: when the guard reminders run out,
|
||||
# the turn output is accepted as normal completion (annotated via
|
||||
# `manager_dispatch_guard_unresolved`) instead of flipping the work
|
||||
# item to FAILED — not every task needs every seat to delegate.
|
||||
# No lifecycle helpers are mocked: the whole turn loop runs against
|
||||
# the real store, so the review chain must actually materialize.
|
||||
task = await self._make_cto_dispatch_task(
|
||||
metadata_extra={
|
||||
"manager_dispatch_guard_max_retries": 1,
|
||||
"direct_report_role_ids": ["dev"],
|
||||
"direct_report_seat_ids": ["seat::team::cto::dev"],
|
||||
# Stale flag from an imaginary earlier delegating turn: the
|
||||
# per-turn reset must clear it, otherwise the guard would
|
||||
# silently pass without reminders.
|
||||
"manager_board_mutation_performed": True,
|
||||
},
|
||||
)
|
||||
task.status = TaskStatus.PENDING
|
||||
self.executor.execute_task = AsyncMock(
|
||||
side_effect=[
|
||||
TaskResult(status=TaskStatus.DONE, content="Scoped and handled the work directly."),
|
||||
TaskResult(status=TaskStatus.DONE, content="Still handled directly; no delegation needed."),
|
||||
]
|
||||
)
|
||||
|
||||
result = await self.executor._run_work_item(task, {})
|
||||
|
||||
self.assertIsNotNone(result)
|
||||
self.assertNotEqual(result.status, TaskStatus.FAILED)
|
||||
# The reminder loop still ran once before acceptance (proves the
|
||||
# stale mutation flag was reset at turn start).
|
||||
self.assertEqual(self.executor.execute_task.await_count, 2)
|
||||
self.assertIn(
|
||||
"delegate_work",
|
||||
str(task.metadata.get("manager_dispatch_guard_unresolved", "")),
|
||||
)
|
||||
# Self-produced manager output goes through manager review: the
|
||||
# card waits for CEO review and a live report card drives it.
|
||||
work_item = await self.store.get_delegation_work_item("cto-dispatch-item")
|
||||
self.assertEqual(work_item.phase, Phase.AWAITING_MANAGER_REVIEW)
|
||||
self.assertEqual(
|
||||
str((work_item.metadata or {}).get("turn_output_kind", "")), "self_produced"
|
||||
)
|
||||
self.assertEqual(
|
||||
str((work_item.metadata or {}).get("turn_output_source", "")),
|
||||
"dispatch_guard_exhausted",
|
||||
)
|
||||
report_cards = await self._aux_cards_targeting("cto-dispatch-item", "report_target_work_item_id")
|
||||
self.assertEqual(len(report_cards), 1)
|
||||
self.assertNotIn(report_cards[0].phase, DONE_PHASES)
|
||||
|
||||
async def _aux_cards_targeting(self, work_item_id: str, key: str) -> list[DelegationWorkItem]:
|
||||
run_items = await self.store.list_delegation_work_items("run-1")
|
||||
return [
|
||||
item for item in run_items
|
||||
if str((item.metadata or {}).get(key, "") or "").strip() == work_item_id
|
||||
]
|
||||
|
||||
async def _make_cto_dispatch_task(self, *, metadata_extra: dict | None = None) -> Task:
|
||||
task = Task(
|
||||
id="cto-dispatch-task",
|
||||
title="CTO Dispatch",
|
||||
project_id="proj1",
|
||||
assigned_to="cto",
|
||||
status=TaskStatus.DONE,
|
||||
metadata={
|
||||
"execution_mode": "company_mode",
|
||||
"runtime_model": "multi_team_org",
|
||||
"delegation_run_id": "run-1",
|
||||
"delegation_seat_id": "seat::team::ceo::cto",
|
||||
"current_turn_mode": "dispatch_required",
|
||||
"work_kind": "dispatch",
|
||||
"manager_role_id": "ceo",
|
||||
"manager_seat_id": "seat::team::ceo::ceo",
|
||||
**(metadata_extra or {}),
|
||||
},
|
||||
)
|
||||
set_linked_work_item_id(task, "cto-dispatch-item")
|
||||
await self.store.save_delegation_work_item(
|
||||
DelegationWorkItem(
|
||||
work_item_id="cto-dispatch-item",
|
||||
run_id="run-1",
|
||||
cell_id="team::ceo",
|
||||
team_instance_id="team-instance::run-1::team::ceo",
|
||||
team_id="team::ceo",
|
||||
role_id="cto",
|
||||
seat_id="seat::team::ceo::cto",
|
||||
seat_state_id="seat-state::run-1::seat::team::ceo::cto",
|
||||
role_runtime_session_id="role-runtime::run-1::seat::team::ceo::cto",
|
||||
title="CTO Dispatch",
|
||||
summary="Route the engineering work.",
|
||||
kind="dispatch",
|
||||
projection_id="cto-dispatch-item",
|
||||
phase=Phase.RUNNING,
|
||||
manager_role_id="ceo",
|
||||
manager_seat_id="seat::team::ceo::ceo",
|
||||
metadata={"work_item_runtime": True, "runtime_model": "multi_team_org"},
|
||||
)
|
||||
)
|
||||
return task
|
||||
|
||||
async def test_self_produced_dispatch_runs_full_report_review_chain(self) -> None:
|
||||
# Real store, no lifecycle mocks: justified self-produced dispatch
|
||||
# output must route to manager review, actually spawn the report
|
||||
# card, and — once the report turn finishes — actually spawn the
|
||||
# review card in the manager seat.
|
||||
task = await self._make_cto_dispatch_task(
|
||||
metadata_extra={"manager_no_delegation_justification": "single-seat scoping decision"},
|
||||
)
|
||||
|
||||
phase = await self.executor._apply_done_transition(
|
||||
task, result=TaskResult(status=TaskStatus.DONE, content="Scoped the work; no delegation needed."),
|
||||
)
|
||||
|
||||
self.assertEqual(phase, Phase.AWAITING_MANAGER_REVIEW)
|
||||
work_item = await self.store.get_delegation_work_item("cto-dispatch-item")
|
||||
self.assertEqual(work_item.phase, Phase.AWAITING_MANAGER_REVIEW)
|
||||
self.assertEqual(
|
||||
str((work_item.metadata or {}).get("turn_output_source", "")), "justified"
|
||||
)
|
||||
report_cards = await self._aux_cards_targeting("cto-dispatch-item", "report_target_work_item_id")
|
||||
self.assertEqual(len(report_cards), 1)
|
||||
report_card = report_cards[0]
|
||||
self.assertNotIn(report_card.phase, DONE_PHASES)
|
||||
|
||||
# Drive the report turn to completion — the review card must appear.
|
||||
report_task = Task(
|
||||
id="cto-report-task",
|
||||
title=report_card.title,
|
||||
project_id="proj1",
|
||||
assigned_to="cto",
|
||||
status=TaskStatus.DONE,
|
||||
metadata={
|
||||
**dict(report_card.metadata or {}),
|
||||
"execution_mode": "company_mode",
|
||||
"delegation_run_id": "run-1",
|
||||
"delegation_seat_id": "seat::team::ceo::cto",
|
||||
"manager_role_id": "ceo",
|
||||
"manager_seat_id": "seat::team::ceo::ceo",
|
||||
},
|
||||
)
|
||||
set_linked_work_item_id(report_task, report_card.work_item_id)
|
||||
await self.executor._apply_done_transition(
|
||||
report_task,
|
||||
result=TaskResult(status=TaskStatus.DONE, content="Structured handoff report."),
|
||||
)
|
||||
|
||||
review_cards = await self._aux_cards_targeting("cto-dispatch-item", "review_target_work_item_id")
|
||||
self.assertEqual(len(review_cards), 1)
|
||||
self.assertNotIn(review_cards[0].phase, DONE_PHASES)
|
||||
self.assertEqual(str(review_cards[0].role_id or ""), "ceo")
|
||||
|
||||
async def test_dispatch_turn_that_delegated_keeps_auto_approve(self) -> None:
|
||||
# Normal delegation flow: a live child card exists in the store, so
|
||||
# the dispatch exemption applies (children carry the reviewable
|
||||
# output) — classification comes from store ground truth, not from
|
||||
# transient task markers.
|
||||
task = await self._make_cto_dispatch_task()
|
||||
await self.store.save_delegation_work_item(
|
||||
DelegationWorkItem(
|
||||
work_item_id="cto-delegated-child",
|
||||
run_id="run-1",
|
||||
cell_id="team::ceo",
|
||||
team_instance_id="team-instance::run-1::team::ceo",
|
||||
team_id="team::ceo",
|
||||
role_id="cto",
|
||||
seat_id="seat::team::ceo::cto",
|
||||
seat_state_id="seat-state::run-1::seat::team::ceo::cto",
|
||||
role_runtime_session_id="role-runtime::run-1::seat::team::ceo::cto",
|
||||
parent_work_item_id="cto-dispatch-item",
|
||||
title="Delegated child",
|
||||
summary="Build the feature.",
|
||||
kind="execute",
|
||||
projection_id="cto-delegated-child",
|
||||
phase=Phase.READY,
|
||||
manager_role_id="cto",
|
||||
manager_seat_id="seat::team::ceo::cto",
|
||||
metadata={"work_item_runtime": True, "runtime_model": "multi_team_org"},
|
||||
)
|
||||
)
|
||||
|
||||
phase = await self.executor._apply_done_transition(
|
||||
task, result=TaskResult(status=TaskStatus.DONE, content="Delegated to the team."),
|
||||
)
|
||||
|
||||
self.assertEqual(phase, Phase.APPROVED)
|
||||
work_item = await self.store.get_delegation_work_item("cto-dispatch-item")
|
||||
self.assertEqual(
|
||||
str((work_item.metadata or {}).get("turn_output_kind", "")), "delegated"
|
||||
)
|
||||
report_cards = await self._aux_cards_targeting("cto-dispatch-item", "report_target_work_item_id")
|
||||
self.assertEqual(report_cards, [])
|
||||
|
||||
async def test_self_produced_intake_routes_to_manager_review(self) -> None:
|
||||
# The delegation-output rule covers every review-exempt delegation
|
||||
# kind, not just dispatch — an intake turn that answered the work
|
||||
# itself needs review too.
|
||||
task = await self._make_cto_dispatch_task(metadata_extra={"work_kind": "intake"})
|
||||
|
||||
phase = await self.executor._apply_done_transition(
|
||||
task, result=TaskResult(status=TaskStatus.DONE, content="Handled the intake question directly."),
|
||||
)
|
||||
|
||||
self.assertEqual(phase, Phase.AWAITING_MANAGER_REVIEW)
|
||||
work_item = await self.store.get_delegation_work_item("cto-dispatch-item")
|
||||
self.assertEqual(
|
||||
str((work_item.metadata or {}).get("turn_output_kind", "")), "self_produced"
|
||||
)
|
||||
|
||||
async def test_top_seat_self_produced_falls_back_to_auto_approve(self) -> None:
|
||||
# The CEO has no manager to review: the existing no-reviewer
|
||||
# fallback auto-approves instead of stranding the card.
|
||||
self.task.status = TaskStatus.DONE
|
||||
self.task.metadata["work_kind"] = "dispatch"
|
||||
self.task.metadata["manager_dispatch_guard_unresolved"] = "no children"
|
||||
await self.store.update_delegation_work_item("ceo-work-item", phase=Phase.RUNNING)
|
||||
|
||||
phase = await self.executor._apply_done_transition(
|
||||
self.task, result=TaskResult(status=TaskStatus.DONE, content="Handled at the top."),
|
||||
)
|
||||
|
||||
self.assertEqual(phase, Phase.APPROVED)
|
||||
|
||||
async def test_reconcile_rebuilds_missing_report_card(self) -> None:
|
||||
# Legacy shape: card already parked in AWAITING_MANAGER_REVIEW with
|
||||
# the self_produced marker but no live report/review card (the
|
||||
# historical spawn refused dispatch parents). The dispatcher-tick
|
||||
# reconcile must rebuild the report card idempotently.
|
||||
await self._make_cto_dispatch_task()
|
||||
await self.store.update_delegation_work_item(
|
||||
"cto-dispatch-item",
|
||||
phase=Phase.AWAITING_MANAGER_REVIEW,
|
||||
metadata_updates={
|
||||
"turn_output_kind": "self_produced",
|
||||
"turn_output_source": "dispatch_guard_exhausted",
|
||||
},
|
||||
)
|
||||
run_items = await self.store.list_delegation_work_items("run-1")
|
||||
|
||||
run_items = await self.executor._reconcile_missing_review_chain(run_items)
|
||||
|
||||
report_cards = await self._aux_cards_targeting("cto-dispatch-item", "report_target_work_item_id")
|
||||
self.assertEqual(len(report_cards), 1)
|
||||
# Idempotent: a second pass with the live report card present must
|
||||
# not spawn another attempt.
|
||||
run_items = await self.executor._reconcile_missing_review_chain(run_items)
|
||||
report_cards = await self._aux_cards_targeting("cto-dispatch-item", "report_target_work_item_id")
|
||||
self.assertEqual(len(report_cards), 1)
|
||||
|
||||
|
||||
class CompanyModeParallelIsolationTests(unittest.IsolatedAsyncioTestCase):
|
||||
async def test_execute_multi_team_org_isolates_claimed_work_item_exception(self) -> None:
|
||||
|
||||
@@ -1301,6 +1301,108 @@ class CompanyRuntimeSuspendResumeTests(unittest.IsolatedAsyncioTestCase):
|
||||
self.assertEqual(refreshed_item.metadata.get("dispatch_hold"), "")
|
||||
self.assertEqual(resumed_task.status, TaskStatus.BLOCKED)
|
||||
|
||||
async def test_continue_keeps_settlement_released_triage_card_running(self) -> None:
|
||||
"""Stop/Resume must not re-lock a failure-triage card the frontier
|
||||
pass already released over a FAILED dependency: the failed dep is
|
||||
terminal, so after a regression no event would ever wake it again."""
|
||||
store = await self._store()
|
||||
_, task = await self._seed_runtime(store)
|
||||
dependency = DelegationWorkItem(
|
||||
work_item_id="dep-item",
|
||||
run_id="run-1",
|
||||
role_id="designer",
|
||||
seat_id="seat-2",
|
||||
title="Dependency",
|
||||
projection_id="dependency",
|
||||
phase=Phase.FAILED,
|
||||
metadata={"runtime_model": "multi_team_org"},
|
||||
)
|
||||
await store.save_delegation_work_item(dependency)
|
||||
item = await store.get_delegation_work_item("work-item-1")
|
||||
assert item is not None
|
||||
item.metadata = {
|
||||
**dict(item.metadata or {}),
|
||||
"runtime_model": "multi_team_org",
|
||||
"dependency_work_item_ids": ["dep-item"],
|
||||
"dependency_classes": {"dep-item": "hard"},
|
||||
"dependency_settlement": {
|
||||
"failed": ["dep-item"],
|
||||
"cancelled": [],
|
||||
"stuck": [],
|
||||
"settled_at": "2026-07-13T00:00:00",
|
||||
},
|
||||
}
|
||||
await store.save_delegation_work_item(item)
|
||||
engine = self._engine(store)
|
||||
await engine.suspend_company_runtime(
|
||||
origin_task_id=task.id,
|
||||
session_id="sess-parent",
|
||||
reason="user_stop",
|
||||
)
|
||||
captured: dict[str, Any] = {}
|
||||
|
||||
class DummyCompanyExecutor:
|
||||
async def execute(self, plan: CompanyWorkItemRuntimePlan, tasks: list[Task]) -> str:
|
||||
captured["tasks"] = tasks
|
||||
return "runtime resumed"
|
||||
|
||||
engine.company_executor = DummyCompanyExecutor()
|
||||
|
||||
await engine._maybe_resume_checkpoint(
|
||||
"continue",
|
||||
"sess-parent",
|
||||
reply_metadata={"ui_force_resume": True},
|
||||
)
|
||||
refreshed_item = await store.get_delegation_work_item("work-item-1")
|
||||
|
||||
assert refreshed_item is not None
|
||||
self.assertEqual(refreshed_item.phase, Phase.RUNNING)
|
||||
|
||||
def test_resume_dependency_check_honors_settlement_for_soft_stuck_deps(self) -> None:
|
||||
"""The soft-class branch must honor the settlement stamp too: a
|
||||
soft dep parked in WAITING_DEPENDENCIES (not in-progress, not
|
||||
terminal) that the frontier stamped as stuck must not re-lock the
|
||||
released card on resume."""
|
||||
stuck_dep = DelegationWorkItem(
|
||||
work_item_id="soft-dep",
|
||||
run_id="run-soft",
|
||||
role_id="w",
|
||||
seat_id="seat-w",
|
||||
title="Stuck soft dep",
|
||||
projection_id="soft-dep",
|
||||
phase=Phase.WAITING_DEPENDENCIES,
|
||||
)
|
||||
card = DelegationWorkItem(
|
||||
work_item_id="soft-card",
|
||||
run_id="run-soft",
|
||||
role_id="m",
|
||||
seat_id="seat-m",
|
||||
title="Released triage card",
|
||||
projection_id="soft-card",
|
||||
phase=Phase.RUNNING,
|
||||
metadata={
|
||||
"dependency_work_item_ids": ["soft-dep"],
|
||||
"dependency_classes": {"soft-dep": "soft"},
|
||||
"dependency_settlement": {
|
||||
"failed": [],
|
||||
"cancelled": [],
|
||||
"stuck": ["soft-dep"],
|
||||
"settled_at": "2026-07-13T00:00:00",
|
||||
},
|
||||
},
|
||||
)
|
||||
by_id = {"soft-dep": stuck_dep, "soft-card": card}
|
||||
self.assertTrue(
|
||||
OPCEngine._company_runtime_dependencies_satisfied(card, by_id)
|
||||
)
|
||||
card.metadata = {
|
||||
**dict(card.metadata or {}),
|
||||
"dependency_settlement": {},
|
||||
}
|
||||
self.assertFalse(
|
||||
OPCEngine._company_runtime_dependencies_satisfied(card, by_id)
|
||||
)
|
||||
|
||||
async def test_continue_does_not_use_synthetic_external_session_id_as_provider_token(self) -> None:
|
||||
store = await self._store()
|
||||
_, task = await self._seed_runtime(
|
||||
|
||||
@@ -333,8 +333,10 @@ class RefreshDependentsForRunTests(unittest.IsolatedAsyncioTestCase):
|
||||
self.assertEqual(after.metadata.get("waiting_on_work_item_ids"), [])
|
||||
|
||||
async def test_child_cancelled_triggers_parent_refresh(self) -> None:
|
||||
"""Fix 3 core: non-APPROVED terminal (CANCELLED) must still fire
|
||||
the refresh hook. Before Fix 3, only APPROVED did."""
|
||||
"""Fix 3 core + failure-triage release: a non-APPROVED terminal
|
||||
(CANCELLED) fires the refresh hook, and because every dependency is
|
||||
now settled the parent is RELEASED for a triage turn instead of
|
||||
waiting on the dead child forever (the project-4444 deadlock)."""
|
||||
parent = _make_work_item(
|
||||
work_item_id="parent-c",
|
||||
run_id="run-b",
|
||||
@@ -357,14 +359,534 @@ class RefreshDependentsForRunTests(unittest.IsolatedAsyncioTestCase):
|
||||
await self.store.update_delegation_work_item(
|
||||
"child-b", phase=Phase.CANCELLED
|
||||
)
|
||||
# Parent still WAITING_FOR_CHILDREN (not all approved), but the
|
||||
# hook ran — verify by checking waiting_on_work_item_ids.
|
||||
after = await self.store.get_delegation_work_item("parent-c")
|
||||
self.assertEqual(after.phase, Phase.RUNNING)
|
||||
self.assertEqual(after.claimed_by_role_runtime_session_id, "")
|
||||
settlement = dict(after.metadata.get("dependency_settlement", {}) or {})
|
||||
self.assertEqual(list(settlement.get("cancelled", [])), ["child-b"])
|
||||
self.assertEqual(list(settlement.get("failed", [])), [])
|
||||
self.assertEqual(list(after.metadata.get("waiting_on_work_item_ids", [])), [])
|
||||
|
||||
async def test_failed_child_releases_parent_for_triage_synthesis(self) -> None:
|
||||
"""Project-4444 regression: a FAILED child must release the
|
||||
delegating parent into a synthesis/triage turn with the failure
|
||||
stamped, not pin it in WAITING_FOR_CHILDREN forever."""
|
||||
parent = _make_work_item(
|
||||
work_item_id="parent-f",
|
||||
run_id="run-f",
|
||||
phase=Phase.WAITING_FOR_CHILDREN,
|
||||
dependency_ids=["child-ok", "child-bad"],
|
||||
claimed_by="claim-y",
|
||||
metadata={"delegated_children_pending": True},
|
||||
)
|
||||
child_ok = _make_work_item(
|
||||
work_item_id="child-ok", run_id="run-f", phase=Phase.APPROVED
|
||||
)
|
||||
child_bad = _make_work_item(
|
||||
work_item_id="child-bad", run_id="run-f", phase=Phase.RUNNING
|
||||
)
|
||||
await self._save(parent, child_ok, child_bad)
|
||||
|
||||
await self.store.update_delegation_work_item("child-bad", phase=Phase.FAILED)
|
||||
|
||||
after = await self.store.get_delegation_work_item("parent-f")
|
||||
self.assertEqual(after.phase, Phase.READY)
|
||||
self.assertEqual(after.claimed_by_role_runtime_session_id, "")
|
||||
self.assertEqual(after.metadata.get("work_kind"), "synthesize")
|
||||
self.assertEqual(after.metadata.get("current_turn_mode"), "synthesize_required")
|
||||
settlement = dict(after.metadata.get("dependency_settlement", {}) or {})
|
||||
self.assertEqual(list(settlement.get("failed", [])), ["child-bad"])
|
||||
self.assertIn("failed/cancelled", str(after.summary or ""))
|
||||
|
||||
async def test_doomed_chain_marks_stuck_and_releases_parent(self) -> None:
|
||||
"""Transitive settlement: sibling B hard-depends on FAILED A, so B
|
||||
can never run. The parent (deps [A, B]) must still be released,
|
||||
with B recorded as stuck rather than waited on forever."""
|
||||
parent = _make_work_item(
|
||||
work_item_id="parent-chain",
|
||||
run_id="run-chain",
|
||||
phase=Phase.WAITING_FOR_CHILDREN,
|
||||
dependency_ids=["chain-a", "chain-b"],
|
||||
metadata={"runtime_model": "multi_team_org"},
|
||||
)
|
||||
chain_a = _make_work_item(
|
||||
work_item_id="chain-a", run_id="run-chain", phase=Phase.RUNNING
|
||||
)
|
||||
chain_b = _make_work_item(
|
||||
work_item_id="chain-b",
|
||||
run_id="run-chain",
|
||||
phase=Phase.WAITING_DEPENDENCIES,
|
||||
dependency_ids=["chain-a"],
|
||||
)
|
||||
await self._save(parent, chain_a, chain_b)
|
||||
|
||||
await self.store.update_delegation_work_item("chain-a", phase=Phase.FAILED)
|
||||
|
||||
after_parent = await self.store.get_delegation_work_item("parent-chain")
|
||||
after_b = await self.store.get_delegation_work_item("chain-b")
|
||||
self.assertEqual(after_parent.phase, Phase.RUNNING)
|
||||
settlement = dict(after_parent.metadata.get("dependency_settlement", {}) or {})
|
||||
self.assertEqual(list(settlement.get("failed", [])), ["chain-a"])
|
||||
self.assertEqual(list(settlement.get("stuck", [])), ["chain-b"])
|
||||
# The stuck sibling itself is NOT released — the triage turn decides.
|
||||
self.assertEqual(after_b.phase, Phase.WAITING_DEPENDENCIES)
|
||||
# The released parent must actually be claimable despite the
|
||||
# non-terminal stuck dep, or the release is cosmetic.
|
||||
run_items = await self.store.list_delegation_work_items("run-chain")
|
||||
by_id = {item.work_item_id: item for item in run_items}
|
||||
self.assertTrue(
|
||||
CompanyWorkItemExecutor._work_item_is_runnable(by_id["parent-chain"], by_id)
|
||||
)
|
||||
|
||||
async def test_transitive_only_failure_still_releases_parent(self) -> None:
|
||||
"""`parent → B → FAILED A` where A is NOT a direct dep of the
|
||||
parent: no direct dep ever turns FAILED, but B is doomed, so the
|
||||
parent must still be released instead of waiting forever."""
|
||||
parent = _make_work_item(
|
||||
work_item_id="parent-t",
|
||||
run_id="run-t",
|
||||
phase=Phase.WAITING_FOR_CHILDREN,
|
||||
dependency_ids=["t-b"],
|
||||
metadata={"runtime_model": "multi_team_org"},
|
||||
)
|
||||
t_b = _make_work_item(
|
||||
work_item_id="t-b",
|
||||
run_id="run-t",
|
||||
phase=Phase.WAITING_DEPENDENCIES,
|
||||
dependency_ids=["t-a"],
|
||||
)
|
||||
t_a = _make_work_item(work_item_id="t-a", run_id="run-t", phase=Phase.RUNNING)
|
||||
await self._save(t_b, parent, t_a)
|
||||
|
||||
await self.store.update_delegation_work_item("t-a", phase=Phase.FAILED)
|
||||
|
||||
after_parent = await self.store.get_delegation_work_item("parent-t")
|
||||
self.assertEqual(after_parent.phase, Phase.RUNNING)
|
||||
settlement = dict(after_parent.metadata.get("dependency_settlement", {}) or {})
|
||||
self.assertEqual(list(settlement.get("failed", [])), [])
|
||||
self.assertEqual(list(settlement.get("stuck", [])), ["t-b"])
|
||||
run_items = await self.store.list_delegation_work_items("run-t")
|
||||
by_id = {item.work_item_id: item for item in run_items}
|
||||
self.assertTrue(
|
||||
CompanyWorkItemExecutor._work_item_is_runnable(by_id["parent-t"], by_id)
|
||||
)
|
||||
|
||||
async def test_delivery_rollup_released_over_failed_dependency(self) -> None:
|
||||
"""A delivery card WAITING_DEPENDENCIES on a FAILED child is released
|
||||
READY so the failure report reaches the user instead of wedging."""
|
||||
delivery = _make_work_item(
|
||||
work_item_id="delivery-f",
|
||||
run_id="run-dlv",
|
||||
phase=Phase.WAITING_DEPENDENCIES,
|
||||
dependency_ids=["dlv-child"],
|
||||
metadata={"work_kind": "delivery"},
|
||||
)
|
||||
child = _make_work_item(
|
||||
work_item_id="dlv-child", run_id="run-dlv", phase=Phase.RUNNING
|
||||
)
|
||||
await self._save(delivery, child)
|
||||
|
||||
await self.store.update_delegation_work_item("dlv-child", phase=Phase.FAILED)
|
||||
|
||||
after = await self.store.get_delegation_work_item("delivery-f")
|
||||
self.assertEqual(after.phase, Phase.READY)
|
||||
settlement = dict(after.metadata.get("dependency_settlement", {}) or {})
|
||||
self.assertEqual(list(settlement.get("failed", [])), ["dlv-child"])
|
||||
|
||||
async def test_settlement_cascade_cancels_unrebuilt_stuck_children(self) -> None:
|
||||
"""Once the settled triage card is APPROVED, leftover doomed stuck
|
||||
children are cancelled so the run can reach a fully-terminal state."""
|
||||
parent = _make_work_item(
|
||||
work_item_id="parent-casc",
|
||||
run_id="run-casc",
|
||||
phase=Phase.RUNNING,
|
||||
dependency_ids=["casc-a", "casc-b"],
|
||||
metadata={
|
||||
"dependency_settlement": {
|
||||
"failed": ["casc-a"],
|
||||
"cancelled": [],
|
||||
"stuck": ["casc-b"],
|
||||
"settled_at": "2026-07-13T00:00:00",
|
||||
},
|
||||
},
|
||||
)
|
||||
casc_a = _make_work_item(
|
||||
work_item_id="casc-a", run_id="run-casc", phase=Phase.FAILED
|
||||
)
|
||||
casc_b = _make_work_item(
|
||||
work_item_id="casc-b",
|
||||
run_id="run-casc",
|
||||
phase=Phase.WAITING_DEPENDENCIES,
|
||||
dependency_ids=["casc-a"],
|
||||
)
|
||||
await self._save(casc_b, casc_a, parent)
|
||||
|
||||
await self.store.update_delegation_work_item("parent-casc", phase=Phase.APPROVED)
|
||||
|
||||
after_b = await self.store.get_delegation_work_item("casc-b")
|
||||
after_parent = await self.store.get_delegation_work_item("parent-casc")
|
||||
self.assertEqual(after_b.phase, Phase.CANCELLED)
|
||||
self.assertEqual(
|
||||
after_b.metadata.get("last_transition_reason"),
|
||||
"upstream_dependency_failed_parent_settled",
|
||||
)
|
||||
settlement = dict(after_parent.metadata.get("dependency_settlement", {}) or {})
|
||||
self.assertTrue(settlement.get("cascaded_at"))
|
||||
|
||||
async def test_settlement_cascade_cancels_deep_doomed_chain(self) -> None:
|
||||
"""Closure: C hard-depends on stuck B; C is in no released card's
|
||||
direct dependency list, but cancelling B dooms it — the cascade
|
||||
must cancel the whole chain or the run never terminalizes."""
|
||||
parent = _make_work_item(
|
||||
work_item_id="parent-deep",
|
||||
run_id="run-deep",
|
||||
phase=Phase.RUNNING,
|
||||
dependency_ids=["deep-a", "deep-b"],
|
||||
metadata={
|
||||
"dependency_settlement": {
|
||||
"failed": ["deep-a"],
|
||||
"cancelled": [],
|
||||
"stuck": ["deep-b"],
|
||||
"settled_at": "2026-07-13T00:00:00",
|
||||
},
|
||||
},
|
||||
)
|
||||
deep_a = _make_work_item(
|
||||
work_item_id="deep-a", run_id="run-deep", phase=Phase.FAILED
|
||||
)
|
||||
deep_b = _make_work_item(
|
||||
work_item_id="deep-b",
|
||||
run_id="run-deep",
|
||||
phase=Phase.WAITING_DEPENDENCIES,
|
||||
dependency_ids=["deep-a"],
|
||||
)
|
||||
deep_c = _make_work_item(
|
||||
work_item_id="deep-c",
|
||||
run_id="run-deep",
|
||||
phase=Phase.WAITING_DEPENDENCIES,
|
||||
dependency_ids=["deep-b"],
|
||||
)
|
||||
await self._save(deep_c, deep_b, deep_a, parent)
|
||||
|
||||
await self.store.update_delegation_work_item("parent-deep", phase=Phase.APPROVED)
|
||||
|
||||
after_b = await self.store.get_delegation_work_item("deep-b")
|
||||
after_c = await self.store.get_delegation_work_item("deep-c")
|
||||
self.assertEqual(after_b.phase, Phase.CANCELLED)
|
||||
self.assertEqual(after_c.phase, Phase.CANCELLED)
|
||||
|
||||
async def test_settlement_cascade_spares_rewired_children(self) -> None:
|
||||
"""Stuck children the manager rewired onto a live replacement stay
|
||||
alive: rewiring drops them out of the doomed set."""
|
||||
parent = _make_work_item(
|
||||
work_item_id="parent-rw",
|
||||
run_id="run-rw",
|
||||
phase=Phase.RUNNING,
|
||||
dependency_ids=["rw-a", "rw-b"],
|
||||
metadata={
|
||||
"dependency_settlement": {
|
||||
"failed": ["rw-a"],
|
||||
"cancelled": [],
|
||||
"stuck": ["rw-b"],
|
||||
"settled_at": "2026-07-13T00:00:00",
|
||||
},
|
||||
},
|
||||
)
|
||||
# Failed child was manager-deleted with a replacement, so rw-b's
|
||||
# dependency normalizes onto the live replacement card.
|
||||
rw_a = _make_work_item(
|
||||
work_item_id="rw-a",
|
||||
run_id="run-rw",
|
||||
phase=Phase.CANCELLED,
|
||||
metadata={
|
||||
"deleted_by_manager_tool": True,
|
||||
"replacement_dependency_work_item_ids": ["rw-a2"],
|
||||
},
|
||||
)
|
||||
rw_a2 = _make_work_item(
|
||||
work_item_id="rw-a2", run_id="run-rw", phase=Phase.RUNNING
|
||||
)
|
||||
rw_b = _make_work_item(
|
||||
work_item_id="rw-b",
|
||||
run_id="run-rw",
|
||||
phase=Phase.WAITING_DEPENDENCIES,
|
||||
dependency_ids=["rw-a"],
|
||||
)
|
||||
await self._save(rw_a2, rw_b, rw_a, parent)
|
||||
|
||||
await self.store.update_delegation_work_item("parent-rw", phase=Phase.APPROVED)
|
||||
|
||||
after_b = await self.store.get_delegation_work_item("rw-b")
|
||||
self.assertEqual(after_b.phase, Phase.WAITING_DEPENDENCIES)
|
||||
|
||||
async def test_park_does_not_wait_on_settled_failed_or_doomed_deps(self) -> None:
|
||||
"""A triage turn that accepts partial results must be able to
|
||||
finish: settled deps (terminal or doomed) are not pending, so the
|
||||
card completes instead of re-parking forever on the FAILED child
|
||||
it just triaged."""
|
||||
parent = _make_work_item(
|
||||
work_item_id="park-parent",
|
||||
run_id="run-park",
|
||||
phase=Phase.RUNNING,
|
||||
dependency_ids=["park-ok", "park-bad", "park-stuck"],
|
||||
metadata={
|
||||
"dependency_settlement": {
|
||||
"failed": ["park-bad"],
|
||||
"cancelled": [],
|
||||
"stuck": ["park-stuck"],
|
||||
"settled_at": "2026-07-13T00:00:00",
|
||||
},
|
||||
},
|
||||
)
|
||||
park_ok = _make_work_item(
|
||||
work_item_id="park-ok", run_id="run-park", phase=Phase.APPROVED
|
||||
)
|
||||
park_bad = _make_work_item(
|
||||
work_item_id="park-bad", run_id="run-park", phase=Phase.FAILED
|
||||
)
|
||||
park_stuck = _make_work_item(
|
||||
work_item_id="park-stuck",
|
||||
run_id="run-park",
|
||||
phase=Phase.WAITING_DEPENDENCIES,
|
||||
dependency_ids=["park-bad"],
|
||||
)
|
||||
await self._save(park_ok, park_bad, park_stuck, parent)
|
||||
|
||||
task = Task(
|
||||
id="park-task",
|
||||
title="Triage turn",
|
||||
project_id="proj1",
|
||||
assigned_to="m",
|
||||
status=TaskStatus.RUNNING,
|
||||
metadata={"work_item_projection_id": "park-parent"},
|
||||
)
|
||||
set_linked_work_item_id(task, "park-parent")
|
||||
await self.store.save_task(task)
|
||||
|
||||
executor = self._executor()
|
||||
parked = await executor._park_for_delegated_children(task)
|
||||
|
||||
self.assertFalse(parked)
|
||||
after = await self.store.get_delegation_work_item("park-parent")
|
||||
self.assertEqual(after.phase, Phase.RUNNING)
|
||||
# Settlement stamp survives (needed by the cascade at APPROVED).
|
||||
self.assertTrue(dict(after.metadata.get("dependency_settlement", {}) or {}))
|
||||
|
||||
async def test_park_race_releases_triage_when_failure_beat_the_park(self) -> None:
|
||||
"""Race: children failed while the manager turn was still running,
|
||||
so the failure hook fired before the card parked (and may have
|
||||
regressed it to WAITING_FOR_CHILDREN without a stamp). Park must
|
||||
re-arm the triage release instead of silently returning False."""
|
||||
parent = _make_work_item(
|
||||
work_item_id="race-parent",
|
||||
run_id="run-race",
|
||||
phase=Phase.WAITING_FOR_CHILDREN,
|
||||
dependency_ids=["race-bad"],
|
||||
)
|
||||
race_bad = _make_work_item(
|
||||
work_item_id="race-bad", run_id="run-race", phase=Phase.FAILED
|
||||
)
|
||||
await self._save(race_bad, parent)
|
||||
|
||||
task = Task(
|
||||
id="race-task",
|
||||
title="Dispatch turn",
|
||||
project_id="proj1",
|
||||
assigned_to="m",
|
||||
status=TaskStatus.RUNNING,
|
||||
metadata={"work_item_projection_id": "race-parent"},
|
||||
)
|
||||
set_linked_work_item_id(task, "race-parent")
|
||||
await self.store.save_task(task)
|
||||
|
||||
executor = self._executor()
|
||||
parked = await executor._park_for_delegated_children(task)
|
||||
|
||||
self.assertTrue(parked)
|
||||
after = await self.store.get_delegation_work_item("race-parent")
|
||||
self.assertNotEqual(after.phase, Phase.WAITING_FOR_CHILDREN)
|
||||
settlement = dict(after.metadata.get("dependency_settlement", {}) or {})
|
||||
self.assertEqual(list(settlement.get("failed", [])), ["race-bad"])
|
||||
|
||||
async def test_released_triage_card_is_not_doomed_for_upper_parents(self) -> None:
|
||||
"""A released (stamped) triage card is alive: it must not count as
|
||||
doomed, or the upper parent settles early and its cascade could
|
||||
cancel a triage card that is about to run."""
|
||||
triage = _make_work_item(
|
||||
work_item_id="alive-triage",
|
||||
run_id="run-alive",
|
||||
phase=Phase.READY,
|
||||
dependency_ids=["alive-bad"],
|
||||
metadata={
|
||||
"runtime_model": "multi_team_org",
|
||||
"work_kind": "synthesize",
|
||||
"dependency_settlement": {
|
||||
"failed": ["alive-bad"],
|
||||
"cancelled": [],
|
||||
"stuck": [],
|
||||
"settled_at": "2026-07-13T00:00:00",
|
||||
},
|
||||
},
|
||||
)
|
||||
alive_bad = _make_work_item(
|
||||
work_item_id="alive-bad", run_id="run-alive", phase=Phase.FAILED
|
||||
)
|
||||
upper = _make_work_item(
|
||||
work_item_id="alive-upper",
|
||||
run_id="run-alive",
|
||||
phase=Phase.WAITING_FOR_CHILDREN,
|
||||
dependency_ids=["alive-triage"],
|
||||
)
|
||||
await self._save(alive_bad, triage, upper)
|
||||
|
||||
from opc.layer2_organization.work_item_transition import (
|
||||
compute_doomed_work_item_ids,
|
||||
)
|
||||
|
||||
run_items = await self.store.list_delegation_work_items("run-alive")
|
||||
by_id = {item.work_item_id: item for item in run_items}
|
||||
doomed = compute_doomed_work_item_ids(by_id)
|
||||
self.assertNotIn("alive-triage", doomed)
|
||||
|
||||
changed = await refresh_dependents_for_run(self.store, run_id="run-alive")
|
||||
after_upper = await self.store.get_delegation_work_item("alive-upper")
|
||||
# The upper parent keeps waiting for the live triage card — no
|
||||
# premature settlement over it.
|
||||
self.assertEqual(after_upper.phase, Phase.WAITING_FOR_CHILDREN)
|
||||
self.assertFalse(
|
||||
dict(after_upper.metadata.get("dependency_settlement", {}) or {})
|
||||
)
|
||||
|
||||
async def test_settlement_cascade_retries_after_partial_failure(self) -> None:
|
||||
"""Retry: B (stamped stuck) was cancelled by a previous cascade
|
||||
attempt but deeper C failed to cancel. The next pass must still
|
||||
traverse through terminal B and cancel C before stamping."""
|
||||
parent = _make_work_item(
|
||||
work_item_id="retry-parent",
|
||||
run_id="run-retry",
|
||||
phase=Phase.APPROVED,
|
||||
dependency_ids=["retry-a", "retry-b"],
|
||||
metadata={
|
||||
"dependency_settlement": {
|
||||
"failed": ["retry-a"],
|
||||
"cancelled": [],
|
||||
"stuck": ["retry-b"],
|
||||
"settled_at": "2026-07-13T00:00:00",
|
||||
},
|
||||
},
|
||||
)
|
||||
retry_a = _make_work_item(
|
||||
work_item_id="retry-a", run_id="run-retry", phase=Phase.FAILED
|
||||
)
|
||||
retry_b = _make_work_item(
|
||||
work_item_id="retry-b",
|
||||
run_id="run-retry",
|
||||
phase=Phase.CANCELLED,
|
||||
dependency_ids=["retry-a"],
|
||||
)
|
||||
retry_c = _make_work_item(
|
||||
work_item_id="retry-c",
|
||||
run_id="run-retry",
|
||||
phase=Phase.WAITING_DEPENDENCIES,
|
||||
dependency_ids=["retry-b"],
|
||||
)
|
||||
await self._save(retry_c, retry_b, retry_a, parent)
|
||||
|
||||
await refresh_dependents_for_run(self.store, run_id="run-retry")
|
||||
|
||||
after_c = await self.store.get_delegation_work_item("retry-c")
|
||||
after_parent = await self.store.get_delegation_work_item("retry-parent")
|
||||
self.assertEqual(after_c.phase, Phase.CANCELLED)
|
||||
settlement = dict(after_parent.metadata.get("dependency_settlement", {}) or {})
|
||||
self.assertTrue(settlement.get("cascaded_at"))
|
||||
|
||||
async def test_info_dependency_does_not_block_settlement(self) -> None:
|
||||
"""Info-class deps never gate claiming, so an in-flight info dep
|
||||
must not keep a failed board from settling either."""
|
||||
parent = _make_work_item(
|
||||
work_item_id="info-parent",
|
||||
run_id="run-info",
|
||||
phase=Phase.WAITING_FOR_CHILDREN,
|
||||
dependency_ids=["info-bad", "info-fyi"],
|
||||
metadata={"dependency_classes": {"info-fyi": "info"}},
|
||||
)
|
||||
info_bad = _make_work_item(
|
||||
work_item_id="info-bad", run_id="run-info", phase=Phase.RUNNING
|
||||
)
|
||||
info_fyi = _make_work_item(
|
||||
work_item_id="info-fyi", run_id="run-info", phase=Phase.RUNNING
|
||||
)
|
||||
await self._save(info_fyi, info_bad, parent)
|
||||
|
||||
await self.store.update_delegation_work_item("info-bad", phase=Phase.FAILED)
|
||||
|
||||
after = await self.store.get_delegation_work_item("info-parent")
|
||||
self.assertNotEqual(after.phase, Phase.WAITING_FOR_CHILDREN)
|
||||
settlement = dict(after.metadata.get("dependency_settlement", {}) or {})
|
||||
self.assertEqual(list(settlement.get("failed", [])), ["info-bad"])
|
||||
self.assertNotIn("info-fyi", list(settlement.get("stuck", [])))
|
||||
|
||||
async def test_late_created_rollup_card_gets_settlement_on_dispatcher_tick(self) -> None:
|
||||
"""A delivery card created AFTER its dependency failed sees no
|
||||
failure hook; the dispatcher tick must run the frontier for it."""
|
||||
late_bad = _make_work_item(
|
||||
work_item_id="late-bad", run_id="run-late", phase=Phase.FAILED
|
||||
)
|
||||
await self._save(late_bad)
|
||||
delivery = _make_work_item(
|
||||
work_item_id="late-delivery",
|
||||
run_id="run-late",
|
||||
phase=Phase.WAITING_DEPENDENCIES,
|
||||
dependency_ids=["late-bad"],
|
||||
metadata={"work_kind": "delivery"},
|
||||
)
|
||||
await self._save(delivery)
|
||||
|
||||
executor = self._executor()
|
||||
run_items = await self.store.list_delegation_work_items("run-late")
|
||||
await executor._refresh_ready_work_items(run_items, tasks=[])
|
||||
|
||||
after = await self.store.get_delegation_work_item("late-delivery")
|
||||
self.assertEqual(after.phase, Phase.READY)
|
||||
settlement = dict(after.metadata.get("dependency_settlement", {}) or {})
|
||||
self.assertEqual(list(settlement.get("failed", [])), ["late-bad"])
|
||||
|
||||
async def test_park_still_waits_on_live_children(self) -> None:
|
||||
"""Control: genuinely in-flight children still park the manager."""
|
||||
parent = _make_work_item(
|
||||
work_item_id="live-parent",
|
||||
run_id="run-live",
|
||||
phase=Phase.RUNNING,
|
||||
dependency_ids=["live-ok", "live-running"],
|
||||
)
|
||||
live_ok = _make_work_item(
|
||||
work_item_id="live-ok", run_id="run-live", phase=Phase.APPROVED
|
||||
)
|
||||
live_running = _make_work_item(
|
||||
work_item_id="live-running", run_id="run-live", phase=Phase.RUNNING
|
||||
)
|
||||
await self._save(live_ok, live_running, parent)
|
||||
|
||||
task = Task(
|
||||
id="live-task",
|
||||
title="Dispatch turn",
|
||||
project_id="proj1",
|
||||
assigned_to="m",
|
||||
status=TaskStatus.RUNNING,
|
||||
metadata={"work_item_projection_id": "live-parent"},
|
||||
)
|
||||
set_linked_work_item_id(task, "live-parent")
|
||||
await self.store.save_task(task)
|
||||
|
||||
executor = self._executor()
|
||||
parked = await executor._park_for_delegated_children(task)
|
||||
|
||||
self.assertTrue(parked)
|
||||
after = await self.store.get_delegation_work_item("live-parent")
|
||||
self.assertEqual(after.phase, Phase.WAITING_FOR_CHILDREN)
|
||||
# Waiting list should reflect the deps (hook wrote it).
|
||||
self.assertEqual(
|
||||
list(after.metadata.get("waiting_on_work_item_ids", [])),
|
||||
["child-a", "child-b"],
|
||||
["live-running"],
|
||||
)
|
||||
|
||||
async def test_manager_deleted_child_is_pruned_from_parent_dependencies(self) -> None:
|
||||
|
||||
@@ -183,12 +183,21 @@ class WorkerExecuteDoneSpawnsReportTests(unittest.IsolatedAsyncioTestCase):
|
||||
org_engine = _make_org_engine(root)
|
||||
executor = _build_executor(store, org_engine)
|
||||
|
||||
# CEO dispatch card. work_kind=dispatch routes directly
|
||||
# to APPROVED — no review, and no report turn.
|
||||
# Dispatch card that actually delegated (a live child card
|
||||
# exists in the store). Delegated output routes directly to
|
||||
# APPROVED — the children carry the reviewable output, so no
|
||||
# review and no report turn for the dispatch card itself.
|
||||
# (A dispatch card WITHOUT children is the self-produced
|
||||
# case and does get the report/review chain.)
|
||||
child = _build_child_work_item()
|
||||
child.metadata = dict(child.metadata or {})
|
||||
child.metadata["work_kind"] = "dispatch"
|
||||
await store.save_delegation_work_item(child)
|
||||
delegated = _build_child_work_item()
|
||||
delegated.work_item_id = "wi-grandchild"
|
||||
delegated.projection_id = "wi-grandchild"
|
||||
delegated.parent_work_item_id = "wi-child"
|
||||
await store.save_delegation_work_item(delegated)
|
||||
worker_task = _build_worker_task()
|
||||
worker_task.metadata = dict(worker_task.metadata or {})
|
||||
worker_task.metadata["work_kind"] = "dispatch"
|
||||
|
||||
Reference in New Issue
Block a user