Gitinbox/OpenOPC
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(company): stop failed children from wedging the tree and soften the dispatch guard

Browse Source 
Root-cause fix for the project-4444 class of deadlocks: any FAILED/
CANCELLED work item with downstream dependents used to block its whole
tree forever, because the advancement gate required all dependencies to
be APPROVED and nothing ever propagated or triaged failures.

Settlement mechanism (work_item_transition.py):
- compute doomed set (FAILED/CANCELLED seeds contagious through hard
  deps); settlement-released cards are treated as alive
- three-way advancement gate: all-approved (unchanged) / settled-with-
  failures releases the nearest decision-capable card (manager parent ->
  failure-triage synthesis turn, rollup delivery/aggregate -> READY)
  with an atomic dependency_settlement stamp; released cards never
  oscillate back
- claim/park/resume/dispatcher-tick all honor the stamp: runnable gates
  admit settled failed+stuck deps, parking excludes settled deps and
  re-arms triage when a failure raced the park, engine resume no longer
  re-locks released cards, the dispatcher tick releases rollup cards
  created after the failure
- settlement cascade: once the settled card is APPROVED, stuck children
  the manager did not rebuild are cancelled (transitive closure over
  stamped stuck seeds, retried until every cancel lands)
- info-class deps never block settlement; adaptive runnable gate now
  shares DEPENDENCY_CLASS_DEFAULT with the release gates

Dispatch guard (company_mode.py):
- NO_DELEGATION_JUSTIFICATION parsing tolerates markdown decoration
  (bold/lists/quotes/full-width colon) and rejects placeholder echoes
  across all artifact/metadata/content channels
- retries exhausted no longer FAILs the work item: dispatch is a soft
  constraint, so the turn output is accepted as normal completion and
  annotated via manager_dispatch_guard_unresolved; the reminder loop is
  unchanged, and the mutation flag is now reset per turn so one past
  delegation can never mute future reminders
- manager board context now surfaces failed/cancelled children with
  their preserved output and pending-cancellation stuck list so the
  triage turn can rebuild, accept partial results, or escalate

Self-produced delegation output goes through review (persisted fact,
single predicate):
- the DONE transition classifies what a dispatch/intake/plan turn
  actually delivered from store ground truth (live children => delegated,
  none => self_produced) and persists turn_output_kind/-source on the
  WorkItem
- is_manager_reviewable_turn honors the persisted marker, so the DONE
  routing, report spawn, report completion and recovery scans all read
  the same fact — this closes a pre-existing hole where
  NO_DELEGATION_JUSTIFICATION output auto-approved with no review at all
- escalation requires a real agent manager above; top seats reporting to
  the human owner keep auto-approve (covered by final delivery's human
  acceptance) instead of minting unclaimable review cards
- dispatcher tick reconciles reviewable cards stuck in
  AWAITING_MANAGER_REVIEW with no live report/review card by rebuilding
  the report card idempotently (legacy DBs, crash windows)

Verified: 4444 tasks.db replay unwedges end to end; real-store
report->review chain exercised without lifecycle mocks (mutation check
confirms the tests bite); full suite failure set identical to a
same-session HEAD baseline run (all remaining failures pre-existing or
environment flakes reproduced at HEAD).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
LZH-YS1998
2026-07-13 16:17:29 +08:00
parent a0402522da
commit 4e7aa75ba5
8 changed files with 1891 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_actor_runtime_company_mode.py
+322
View File
@@ -11,6 +11,7 @@ from opc.core.events import EventBus
from opc.core.models import CompanyMemberSession, DelegationWorkItem, Phase, SeatState, Task, TaskResult, TaskStatus
from opc.database.store import OPCStore
from opc.layer2_organization.communication import CommunicationManager
from opc.layer2_organization.phase import DONE_PHASES
from opc.layer2_organization.company_mode import CompanyWorkItemExecutor
from opc.layer2_organization.company_runtime import CompanyRuntime
from opc.layer2_organization.org_engine import OrgEngine
@@ -837,6 +838,77 @@ class ActorRuntimeManagerDispatchGuardTests(unittest.IsolatedAsyncioTestCase):
"This turn is a root-only scoping decision before any downstream split.",
)
async def test_manager_dispatch_guard_accepts_markdown_decorated_justification(self) -> None:
"""Project-4444 regression: the CTO wrote the escape line as
`**NO_DELEGATION_JUSTIFICATION**:` and the bare startswith parser
missed it, driving the work item to FAILED. Decorated variants must
all parse."""
variants = [
("**NO_DELEGATION_JUSTIFICATION**: The task needs my own research.",
"The task needs my own research."),
("**NO_DELEGATION_JUSTIFICATION: Whole line is bold.**",
"Whole line is bold."),
("- NO_DELEGATION_JUSTIFICATION: Listed as a bullet.",
"Listed as a bullet."),
("### NO_DELEGATION_JUSTIFICATION: Written as a heading.",
"Written as a heading."),
("NO_DELEGATION_JUSTIFICATION： Fullwidth colon variant.",
"Fullwidth colon variant."),
]
for content, expected in variants:
with self.subTest(content=content):
self.task.metadata.pop("manager_no_delegation_justification", None)
before = await self.executor._snapshot_manager_dispatch_state(self.task)
issues = await self.executor._enforce_manager_dispatch_guard(
self.task,
TaskResult(status=TaskStatus.DONE, content=f"Some preamble.\n{content}"),
before_state=before,
)
self.assertEqual(issues, [])
self.assertEqual(
self.task.metadata["manager_no_delegation_justification"],
expected,
)
async def test_manager_dispatch_guard_ignores_echoed_instruction_template(self) -> None:
"""Echoing the guard's own `NO_DELEGATION_JUSTIFICATION: <specific
reason>` template is not a justification — including with trailing
punctuation or quoting around the placeholder."""
echoes = [
"I should finish with `NO_DELEGATION_JUSTIFICATION: <specific reason>` next time.",
"NO_DELEGATION_JUSTIFICATION: <specific reason>.",
'NO_DELEGATION_JUSTIFICATION: "<specific reason>"',
"NO_DELEGATION_JUSTIFICATION: **<specific reason>**",
"NO_DELEGATION_JUSTIFICATION: _<specific reason>_",
]
for content in echoes:
with self.subTest(content=content):
self.task.metadata.pop("manager_no_delegation_justification", None)
before = await self.executor._snapshot_manager_dispatch_state(self.task)
issues = await self.executor._enforce_manager_dispatch_guard(
self.task,
TaskResult(status=TaskStatus.DONE, content=content),
before_state=before,
)
self.assertEqual(len(issues), 1)
self.assertNotIn("manager_no_delegation_justification", self.task.metadata)
async def test_manager_dispatch_guard_ignores_placeholder_in_artifacts(self) -> None:
"""The artifact/metadata escape hatch gets the same placeholder
filter as the free-text line."""
before = await self.executor._snapshot_manager_dispatch_state(self.task)
issues = await self.executor._enforce_manager_dispatch_guard(
self.task,
TaskResult(
status=TaskStatus.DONE,
content="Done.",
artifacts={"no_delegation_justification": "<specific reason>."},
),
before_state=before,
)
self.assertEqual(len(issues), 1)
self.assertNotIn("manager_no_delegation_justification", self.task.metadata)
async def test_manager_dispatch_guard_rejects_no_delegation_for_collab_infra_failure(self) -> None:
before = await self.executor._snapshot_manager_dispatch_state(self.task)
@@ -941,6 +1013,256 @@ class ActorRuntimeManagerDispatchGuardTests(unittest.IsolatedAsyncioTestCase):
self.assertEqual(issues, [])
async def test_manager_dispatch_guard_exhaustion_accepts_turn_instead_of_failing(self) -> None:
# Dispatch is a soft constraint: when the guard reminders run out,
# the turn output is accepted as normal completion (annotated via
# `manager_dispatch_guard_unresolved`) instead of flipping the work
# item to FAILED — not every task needs every seat to delegate.
# No lifecycle helpers are mocked: the whole turn loop runs against
# the real store, so the review chain must actually materialize.
task = await self._make_cto_dispatch_task(
metadata_extra={
"manager_dispatch_guard_max_retries": 1,
"direct_report_role_ids": ["dev"],
"direct_report_seat_ids": ["seat::team::cto::dev"],
# Stale flag from an imaginary earlier delegating turn: the
# per-turn reset must clear it, otherwise the guard would
# silently pass without reminders.
"manager_board_mutation_performed": True,
},
)
task.status = TaskStatus.PENDING
self.executor.execute_task = AsyncMock(
side_effect=[
TaskResult(status=TaskStatus.DONE, content="Scoped and handled the work directly."),
TaskResult(status=TaskStatus.DONE, content="Still handled directly; no delegation needed."),
]
)
result = await self.executor._run_work_item(task, {})
self.assertIsNotNone(result)
self.assertNotEqual(result.status, TaskStatus.FAILED)
# The reminder loop still ran once before acceptance (proves the
# stale mutation flag was reset at turn start).
self.assertEqual(self.executor.execute_task.await_count, 2)
self.assertIn(
"delegate_work",
str(task.metadata.get("manager_dispatch_guard_unresolved", "")),
)
# Self-produced manager output goes through manager review: the
# card waits for CEO review and a live report card drives it.
work_item = await self.store.get_delegation_work_item("cto-dispatch-item")
self.assertEqual(work_item.phase, Phase.AWAITING_MANAGER_REVIEW)
self.assertEqual(
str((work_item.metadata or {}).get("turn_output_kind", "")), "self_produced"
)
self.assertEqual(
str((work_item.metadata or {}).get("turn_output_source", "")),
"dispatch_guard_exhausted",
)
report_cards = await self._aux_cards_targeting("cto-dispatch-item", "report_target_work_item_id")
self.assertEqual(len(report_cards), 1)
self.assertNotIn(report_cards[0].phase, DONE_PHASES)
async def _aux_cards_targeting(self, work_item_id: str, key: str) -> list[DelegationWorkItem]:
run_items = await self.store.list_delegation_work_items("run-1")
return [
item for item in run_items
if str((item.metadata or {}).get(key, "") or "").strip() == work_item_id
]
async def _make_cto_dispatch_task(self, *, metadata_extra: dict | None = None) -> Task:
task = Task(
id="cto-dispatch-task",
title="CTO Dispatch",
project_id="proj1",
assigned_to="cto",
status=TaskStatus.DONE,
metadata={
"execution_mode": "company_mode",
"runtime_model": "multi_team_org",
"delegation_run_id": "run-1",
"delegation_seat_id": "seat::team::ceo::cto",
"current_turn_mode": "dispatch_required",
"work_kind": "dispatch",
"manager_role_id": "ceo",
"manager_seat_id": "seat::team::ceo::ceo",
**(metadata_extra or {}),
},
)
set_linked_work_item_id(task, "cto-dispatch-item")
await self.store.save_delegation_work_item(
DelegationWorkItem(
work_item_id="cto-dispatch-item",
run_id="run-1",
cell_id="team::ceo",
team_instance_id="team-instance::run-1::team::ceo",
team_id="team::ceo",
role_id="cto",
seat_id="seat::team::ceo::cto",
seat_state_id="seat-state::run-1::seat::team::ceo::cto",
role_runtime_session_id="role-runtime::run-1::seat::team::ceo::cto",
title="CTO Dispatch",
summary="Route the engineering work.",
kind="dispatch",
projection_id="cto-dispatch-item",
phase=Phase.RUNNING,
manager_role_id="ceo",
manager_seat_id="seat::team::ceo::ceo",
metadata={"work_item_runtime": True, "runtime_model": "multi_team_org"},
)
)
return task
async def test_self_produced_dispatch_runs_full_report_review_chain(self) -> None:
# Real store, no lifecycle mocks: justified self-produced dispatch
# output must route to manager review, actually spawn the report
# card, and — once the report turn finishes — actually spawn the
# review card in the manager seat.
task = await self._make_cto_dispatch_task(
metadata_extra={"manager_no_delegation_justification": "single-seat scoping decision"},
)
phase = await self.executor._apply_done_transition(
task, result=TaskResult(status=TaskStatus.DONE, content="Scoped the work; no delegation needed."),
)
self.assertEqual(phase, Phase.AWAITING_MANAGER_REVIEW)
work_item = await self.store.get_delegation_work_item("cto-dispatch-item")
self.assertEqual(work_item.phase, Phase.AWAITING_MANAGER_REVIEW)
self.assertEqual(
str((work_item.metadata or {}).get("turn_output_source", "")), "justified"
)
report_cards = await self._aux_cards_targeting("cto-dispatch-item", "report_target_work_item_id")
self.assertEqual(len(report_cards), 1)
report_card = report_cards[0]
self.assertNotIn(report_card.phase, DONE_PHASES)
# Drive the report turn to completion — the review card must appear.
report_task = Task(
id="cto-report-task",
title=report_card.title,
project_id="proj1",
assigned_to="cto",
status=TaskStatus.DONE,
metadata={
**dict(report_card.metadata or {}),
"execution_mode": "company_mode",
"delegation_run_id": "run-1",
"delegation_seat_id": "seat::team::ceo::cto",
"manager_role_id": "ceo",
"manager_seat_id": "seat::team::ceo::ceo",
},
)
set_linked_work_item_id(report_task, report_card.work_item_id)
await self.executor._apply_done_transition(
report_task,
result=TaskResult(status=TaskStatus.DONE, content="Structured handoff report."),
)
review_cards = await self._aux_cards_targeting("cto-dispatch-item", "review_target_work_item_id")
self.assertEqual(len(review_cards), 1)
self.assertNotIn(review_cards[0].phase, DONE_PHASES)
self.assertEqual(str(review_cards[0].role_id or ""), "ceo")
async def test_dispatch_turn_that_delegated_keeps_auto_approve(self) -> None:
# Normal delegation flow: a live child card exists in the store, so
# the dispatch exemption applies (children carry the reviewable
# output) — classification comes from store ground truth, not from
# transient task markers.
task = await self._make_cto_dispatch_task()
await self.store.save_delegation_work_item(
DelegationWorkItem(
work_item_id="cto-delegated-child",
run_id="run-1",
cell_id="team::ceo",
team_instance_id="team-instance::run-1::team::ceo",
team_id="team::ceo",
role_id="cto",
seat_id="seat::team::ceo::cto",
seat_state_id="seat-state::run-1::seat::team::ceo::cto",
role_runtime_session_id="role-runtime::run-1::seat::team::ceo::cto",
parent_work_item_id="cto-dispatch-item",
title="Delegated child",
summary="Build the feature.",
kind="execute",
projection_id="cto-delegated-child",
phase=Phase.READY,
manager_role_id="cto",
manager_seat_id="seat::team::ceo::cto",
metadata={"work_item_runtime": True, "runtime_model": "multi_team_org"},
)
)
phase = await self.executor._apply_done_transition(
task, result=TaskResult(status=TaskStatus.DONE, content="Delegated to the team."),
)
self.assertEqual(phase, Phase.APPROVED)
work_item = await self.store.get_delegation_work_item("cto-dispatch-item")
self.assertEqual(
str((work_item.metadata or {}).get("turn_output_kind", "")), "delegated"
)
report_cards = await self._aux_cards_targeting("cto-dispatch-item", "report_target_work_item_id")
self.assertEqual(report_cards, [])
async def test_self_produced_intake_routes_to_manager_review(self) -> None:
# The delegation-output rule covers every review-exempt delegation
# kind, not just dispatch — an intake turn that answered the work
# itself needs review too.
task = await self._make_cto_dispatch_task(metadata_extra={"work_kind": "intake"})
phase = await self.executor._apply_done_transition(
task, result=TaskResult(status=TaskStatus.DONE, content="Handled the intake question directly."),
)
self.assertEqual(phase, Phase.AWAITING_MANAGER_REVIEW)
work_item = await self.store.get_delegation_work_item("cto-dispatch-item")
self.assertEqual(
str((work_item.metadata or {}).get("turn_output_kind", "")), "self_produced"
)
async def test_top_seat_self_produced_falls_back_to_auto_approve(self) -> None:
# The CEO has no manager to review: the existing no-reviewer
# fallback auto-approves instead of stranding the card.
self.task.status = TaskStatus.DONE
self.task.metadata["work_kind"] = "dispatch"
self.task.metadata["manager_dispatch_guard_unresolved"] = "no children"
await self.store.update_delegation_work_item("ceo-work-item", phase=Phase.RUNNING)
phase = await self.executor._apply_done_transition(
self.task, result=TaskResult(status=TaskStatus.DONE, content="Handled at the top."),
)
self.assertEqual(phase, Phase.APPROVED)
async def test_reconcile_rebuilds_missing_report_card(self) -> None:
# Legacy shape: card already parked in AWAITING_MANAGER_REVIEW with
# the self_produced marker but no live report/review card (the
# historical spawn refused dispatch parents). The dispatcher-tick
# reconcile must rebuild the report card idempotently.
await self._make_cto_dispatch_task()
await self.store.update_delegation_work_item(
"cto-dispatch-item",
phase=Phase.AWAITING_MANAGER_REVIEW,
metadata_updates={
"turn_output_kind": "self_produced",
"turn_output_source": "dispatch_guard_exhausted",
},
)
run_items = await self.store.list_delegation_work_items("run-1")
run_items = await self.executor._reconcile_missing_review_chain(run_items)
report_cards = await self._aux_cards_targeting("cto-dispatch-item", "report_target_work_item_id")
self.assertEqual(len(report_cards), 1)
# Idempotent: a second pass with the live report card present must
# not spawn another attempt.
run_items = await self.executor._reconcile_missing_review_chain(run_items)
report_cards = await self._aux_cards_targeting("cto-dispatch-item", "report_target_work_item_id")
self.assertEqual(len(report_cards), 1)
class CompanyModeParallelIsolationTests(unittest.IsolatedAsyncioTestCase):
async def test_execute_multi_team_org_isolates_claimed_work_item_exception(self) -> None: