Gitinbox/OpenOPC
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(company): stop/resume identity truth, failure-path closure, quota park

Browse Source 
OBS-11 — stop/resume killed pure-native runs over a phantom external pin.
Role templates' preferred_external_agent leaked into execution identity even
when the user requested native and execution actually ran native; on resume
the availability gate trusted the pin and failed every non-terminal item.
Root fixes across the whole chain:
- Staffing card per-role defaults are now the RESOLVED backend (explicit
  session agent choice > runnable template preference > native), never a
  hardcoded external default; seat enrichment and the dispatch selector's
  locked branch downgrade provably unavailable externals to native and
  record the wish in execution_agent_unavailable.
- The resume availability gate fails closed only when a resumable external
  session actually exists; a bare pin heals to native (snapshot AND task
  durable identity) and the run resumes — mirroring dispatch fallback.
- Suspend-checkpoint replies: force_resume (chat/headless spelling) is
  recognized alongside ui_force_resume, and bare continuation tokens
  (English and Chinese spellings) take the plain-resume path instead of
  being routed to the final decider as content, which reopened the
  already-approved intake card.

OBS-5 — failed runs never closed and dropped new input. The dispatcher's
convergence exit now settles terminally-failed runs (status=failed,
lifecycle=closed_failed, run_failure metadata) and emits a
company_run_failure_review card whose replies never swallow messages:
dismiss acknowledges, content falls through so normal routing starts a
fresh run. _maybe_resume_existing_company_runtime no longer re-executes a
terminally-failed tree: control replies get an honest closed status,
content-bearing input starts a new run.

OBS-6 — provider quota exhaustion terminally failed work items. Rate-limit
rejections are classified (LLMProvider.is_rate_limit_error, covering
status codes, exception types, and English/Chinese provider error text),
the agent runtime raises typed ProviderQuotaExhaustedError instead of
burning conversation-feedback retries, and the company dispatcher parks:
the item returns to READY (attempt interrupted, no terminal failure), the
member session idles, and claiming backs off exponentially (60s doubling
to a 900s cap; a quiet 30min resets the streak) before resuming
automatically.

Verified end-to-end on the real minimax-m3 campaign: same goal, same 300s
stop point, same run shape that previously killed the whole tree within
90s now resumes cleanly and completes with all items approved; staffing
defaults native for all 11 roles.

Tests: test_stop_resume_native_pin (10), test_run_failure_settlement (6),
test_provider_quota_park (9); attempt-ledger, recruiter, and
suspend-resume suites updated to the new contracts (their old assertions
pinned the defective behaviors).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
LZH-YS1998
2026-07-28 16:59:56 +08:00
parent 14ee8806de
commit d14f3920e0
10 changed files with 1185 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
opc/engine.py
+210 -5
View File
@@ -1280,11 +1280,50 @@ class OPCEngine:
if str(item.get("template_id", "") or "").strip()
}
roles: list[dict[str, Any]] = []
default_agent = "codex"
# The card's per-role agent defaults become recruitment_role_agents on
# approve, which outrank the session-level agent choice as explicit
# per-role overrides. They must therefore be the RESOLVED effective
# backend, not a hardcoded external preference: an explicit
# preferred_agent from the user wins, a role template preference only
# applies when that adapter can actually run, everything else is
# native. (OBS-11: hardcoded "codex" defaults poisoned the execution
# identity of runs the user explicitly requested as native.)
explicit_session_agent = normalize_recruitment_agent_choice(
getattr(decision, "preferred_agent", None)
)
available_external_agents = set(self._available_external_agents())
def _staffing_agent_is_runnable(agent_name: str) -> bool:
# Unknown availability (no adapter registry) fails open, matching
# the dispatch selector; a registry that exists and excludes the
# agent is proof it cannot run.
if agent_name == "native":
return True
if self.adapter_registry is None:
return True
return agent_name in available_external_agents
for agent in self.org_engine.list_agents():
role_id = str(getattr(agent, "role_id", "") or "").strip()
if not role_id or role_id == "task_generalist":
continue
role_preferred_agent = normalize_recruitment_agent_choice(
getattr(agent, "preferred_external_agent", None)
)
if explicit_session_agent:
default_agent = (
explicit_session_agent
if _staffing_agent_is_runnable(explicit_session_agent)
else "native"
)
elif (
role_preferred_agent
and role_preferred_agent != "native"
and _staffing_agent_is_runnable(role_preferred_agent)
):
default_agent = role_preferred_agent
else:
default_agent = "native"
same_role_employees = employee_by_role.get(role_id, [])
default_selection: dict[str, Any] = {"kind": "fallback"}
default_source = "system"
@@ -1311,6 +1350,8 @@ class OPCEngine:
}
default_source = "org"
selected_agent = normalize_recruitment_agent_choice(saved_agents.get(role_id), default=default_agent) or default_agent
if not _staffing_agent_is_runnable(selected_agent):
selected_agent = default_agent
roles.append(
{
"role_id": role_id,
@@ -2917,6 +2958,7 @@ class OPCEngine:
else None
)
explicit_force_native = explicit_agent_choice == "native"
enrichment_available_external_agents = set(self._available_external_agents())
seats: list[dict[str, Any]] = []
for raw_seat in list(enriched.get("seats", []) or []):
seat = dict(raw_seat or {})
@@ -2972,6 +3014,21 @@ class OPCEngine:
or explicit_agent_choice
or ("native" if force_native_execution or not preferred_external_agent else preferred_external_agent)
)
execution_agent_unavailable = ""
if (
selected_execution_agent
and selected_execution_agent != "native"
and self.adapter_registry is not None
and selected_execution_agent not in enrichment_available_external_agents
):
# The chosen external backend cannot run. Dispatch would fall
# back to native while seat/task metadata kept claiming the
# external agent, and the resume gate trusts that identity
# (OBS-11). Record the truth up front; the wish stays visible
# via execution_agent_unavailable.
execution_agent_unavailable = selected_execution_agent
selected_execution_agent = "native"
preferred_external_agent = None
execution_agent_locked = bool(selected_role_agent or explicit_agent_choice)
selection_source = (
"recruitment_user_override"
@@ -2987,6 +3044,7 @@ class OPCEngine:
seat["execution_agent_locked"] = execution_agent_locked
seat["selected_execution_agent_source"] = selection_source
seat["force_native_execution"] = force_native_execution
seat["execution_agent_unavailable"] = execution_agent_unavailable
seat["metadata"] = {
**dict(seat.get("metadata", {}) or {}),
"employee_prompt_context": str((employee_assignment or {}).get("prompt_context", "")).strip(),
@@ -4740,6 +4798,28 @@ class OPCEngine:
default=("native" if not str(task.assigned_external_agent or "").strip() else str(task.assigned_external_agent or "").strip()),
)
if task.metadata.get("execution_agent_locked") and locked_agent:
if (
locked_agent != "native"
and self.adapter_registry is not None
and locked_agent not in self._available_external_agents()
):
# A locked external backend that cannot run must not be
# stamped as execution identity: this attempt actually runs
# native (the external candidate pool is empty) and resume
# trusts this metadata (OBS-11).
task.assigned_external_agent = None
task.metadata["selected_execution_agent"] = "native"
task.metadata["preferred_external_agent"] = None
task.metadata["execution_agent_unavailable"] = locked_agent
task.metadata["agent_selection"] = {
"selected": "native",
"strategy": WorkItemExecutionStrategy.NATIVE.value,
"role_id": task.assigned_to or task.metadata.get("work_item_role_id", ""),
"decision_reason": "locked_external_agent_unavailable_native_fallback",
"available_external_agents": self._available_external_agents(),
"selection_source": "availability_fallback",
}
return None
selected = None if locked_agent == "native" else locked_agent
task.assigned_external_agent = selected
task.metadata["preferred_external_agent"] = selected
@@ -6384,12 +6464,59 @@ class OPCEngine:
# not fully initialized / delegate context) — fail open and let the
# dispatch-time selector guard decide; only a registry that exists
# and excludes the pinned agent is proof of unavailability.
if (
pinned_agent_unavailable = bool(
pinned_agent != "native"
and not work_item_already_terminal
and self.adapter_registry is not None
and pinned_agent not in self._available_external_agents()
):
)
if pinned_agent_unavailable:
gate_external_session = dict(
(payload.get("external_sessions", {}) or {}).get(task.id) or {}
)
has_resumable_external_session = bool(
gate_external_session
) and external_session_status_allows_resume(
gate_external_session.get("status")
)
if not has_resumable_external_session:
# The pin is a preference, not a fact: this work item has
# no external session to revive, so dispatch would run it
# natively anyway (availability fallback). Heal the
# checkpointed identity to native and resume instead of
# failing the item — a run the user launched as native
# must survive stop/resume even when a disabled external
# agent leaked into its metadata (OBS-11).
healed_identity = dict(
task_snapshot.get("execution_identity", {}) or {}
)
healed_identity["selected_execution_agent"] = "native"
healed_identity["assigned_external_agent"] = ""
if not str(
healed_identity.get("preferred_external_agent", "") or ""
).strip():
healed_identity["preferred_external_agent"] = pinned_agent
task_snapshot["execution_identity"] = healed_identity
task_snapshot["selected_execution_agent"] = "native"
task_snapshot["assigned_external_agent"] = ""
# The task's durable identity must agree with the healed
# snapshot, or the restore validation below rejects the
# resume as an identity mismatch.
task.metadata = dict(task.metadata or {})
task.metadata["resume_execution_agent_healed_from"] = pinned_agent
task.metadata["selected_execution_agent"] = "native"
task.metadata.pop("agent_selection", None)
task.metadata.pop("preferred_external_agent", None)
task.assigned_external_agent = None
logger.info(
"company runtime resume: healed work item {} to native — pinned "
"external agent {!r} is unavailable and no resumable external "
"session exists",
work_item_id or task.id,
pinned_agent,
)
pinned_agent_unavailable = False
if pinned_agent_unavailable:
diagnostic = (
f"Cannot resume work item: its execution is pinned to external agent "
f"'{pinned_agent}', which is currently disabled or unavailable. "
@@ -8546,7 +8673,43 @@ class OPCEngine:
@staticmethod
def _reply_metadata_requests_force_resume(reply_metadata: dict[str, Any] | None) -> bool:
return bool(dict(reply_metadata or {}).get("ui_force_resume", False))
metadata = dict(reply_metadata or {})
# ui_force_resume is what the Office UI resume button sends;
# force_resume is the documented key for chat/headless callers.
# Both express the same control intent (OBS-11: only recognizing the
# UI spelling pushed headless resumes into the content-followup path).
return bool(
metadata.get("ui_force_resume", False)
or metadata.get("force_resume", False)
)
@staticmethod
def _is_plain_resume_control_reply(user_reply: str) -> bool:
"""A bare continuation acknowledgement carries no new instructions.
Such a reply to a suspended-runtime checkpoint means "resume the run",
never "route this text to the final decider as a follow-up" — routing
it as content reopens the already-approved intake card and churns the
run it was supposed to continue (OBS-11).
"""
normalized = " ".join(str(user_reply or "").strip().lower().split())
return normalized in {
"",
"continue",
"resume",
"proceed",
"go",
"go on",
"ok",
"okay",
"y",
"yes",
# Chinese spellings of the same continuation intent.
"继续",
"恢复",
"继续执行",
"继续跑",
}
async def _maybe_resume_existing_company_runtime(
self,
@@ -8579,6 +8742,28 @@ class OPCEngine:
blocked_tasks = [task for task in tasks if task.status == TaskStatus.BLOCKED]
no_active_runtime_work = not live_running_tasks and not waiting_tasks and not pending_tasks and not failed_tasks and not blocked_tasks
has_closed_delivery_review = any(self._is_closed_company_delivery_review_task(task) for task in tasks)
run_terminally_failed = (
bool(failed_tasks)
and not has_closed_delivery_review
and all(
t.status in {TaskStatus.DONE, TaskStatus.CANCELLED, TaskStatus.FAILED}
for t in tasks
)
)
if run_terminally_failed:
# The run is a corpse: re-executing it drops the user's new
# content on the floor (OBS-5). Content-bearing input falls
# through to start a fresh run; bare control replies get an
# honest status instead of a fake resume.
if force_resume or self._is_plain_resume_control_reply(user_reply):
snapshot_text = self._format_company_runtime_snapshot(tasks)
return (
"This company run ended with failed work items and is closed. "
"It cannot be resumed. Send a new request (for example the "
"original goal, optionally adjusted) to start a fresh run.\n\n"
f"{snapshot_text}"
)
return None
if not force_resume:
if not (no_active_runtime_work and has_closed_delivery_review):
followup_result = await self._resume_company_runtime_via_final_decider(
@@ -10997,9 +11182,29 @@ class OPCEngine:
if checkpoint.checkpoint_type == "company_work_item_gate":
return await self._resume_company_runtime_checkpoint(checkpoint, user_reply)
if self._is_company_runtime_suspend_checkpoint(checkpoint.checkpoint_type):
if self._reply_metadata_requests_force_resume(reply_metadata):
if self._reply_metadata_requests_force_resume(
reply_metadata
) or self._is_plain_resume_control_reply(user_reply):
return await self._resume_company_suspend_checkpoint(checkpoint, user_reply)
return await self._resume_company_suspend_checkpoint_via_final_decider(checkpoint, user_reply)
if checkpoint.checkpoint_type == "company_run_failure_review":
if not explicit_checkpoint_id:
# Closure cards must never swallow ordinary session messages.
return None
normalized_failure_reply = " ".join(
str(user_reply or "").strip().lower().split()
)
await self.store.resolve_execution_checkpoint(
checkpoint.checkpoint_id, status="resolved"
)
if normalized_failure_reply in {
# trailing entries are Chinese acknowledgement spellings
"", "dismiss", "ignore", "close", "ok", "okay", "知道了", "关闭",
}:
return "Company run closure acknowledged."
# Content-bearing reply: the failed run is closed, so let the
# message continue as a fresh request through normal routing.
return None
if checkpoint.checkpoint_type == "company_delivery_feedback":
if not explicit_checkpoint_id:
return None
opc/layer2_organization/company_mode.py
+218 -6
View File
@@ -8,6 +8,7 @@ import hashlib
import inspect
import json
import re
import time
import uuid
from contextvars import ContextVar, Token
from dataclasses import asdict, dataclass, field
@@ -156,6 +157,7 @@ from opc.layer2_organization.work_item_runtime_invariants import (
validate_work_item_runtime_projection,
)
from opc.layer4_tools.output_budget import clip_text
from opc.llm.provider import ProviderQuotaExhaustedError
from opc.llm.retry import LLMRetryError, call_llm_json_with_retry
@@ -1423,6 +1425,13 @@ class CompanyWorkItemExecutor:
# waits on this Event so children are claimed+spawned without
# waiting for the parent turn's gather batch to drain.
self._dispatcher_wake = asyncio.Event()
# Provider-quota park (OBS-6): while monotonic time is below
# _quota_park_until the dispatcher stops claiming new work instead of
# hammering an exhausted quota. The streak drives exponential backoff
# (60s → 900s cap); parks separated by more than 30 min restart it.
self._quota_park_until = 0.0
self._quota_park_streak = 0
self._quota_last_park_at = 0.0
# Single-dispatcher-per-run invariant: refcount of live
# _execute_multi_team_org loops keyed by delegation run id.
# Checkpoint answers consult this via wake_live_run_dispatcher —
@@ -4760,12 +4769,23 @@ class CompanyWorkItemExecutor:
self._rehydrate_parked_member_sessions(work_items)
# Claim whatever is immediately claimable and spawn each
# work item as an independent asyncio.Task so the loop no
# longer blocks on the slowest sibling.
claims = await self._claim_and_create_work_item_tasks(
tasks,
work_items,
active_work_item_tasks,
)
# longer blocks on the slowest sibling. While a provider
# quota park is active, claiming is skipped so an exhausted
# quota is not hammered with doomed dispatches (OBS-6).
if self._quota_park_until and time.monotonic() < self._quota_park_until:
claims = []
else:
if self._quota_park_until:
self._quota_park_until = 0.0
await self._emit_progress(
"[Company] provider quota backoff elapsed — resuming dispatch",
task_id=tasks[0].id if tasks else "",
)
claims = await self._claim_and_create_work_item_tasks(
tasks,
work_items,
active_work_item_tasks,
)
# Termination: only when nothing is in-flight AND nothing
# else is runnable. If work items are still running, even an
# "empty runnable" snapshot may become non-empty within
@@ -4972,6 +4992,10 @@ class CompanyWorkItemExecutor:
res,
)
active_work_item_tasks.clear()
try:
await self._settle_run_lifecycle_on_convergence(tasks)
except Exception:
logger.opt(exception=True).debug("run failure settlement skipped")
return self._summarize_multi_team_org_results(tasks)
@staticmethod
@@ -5136,12 +5160,80 @@ class CompanyWorkItemExecutor:
) == "running"
)
def _exception_is_provider_quota(self, exc: BaseException | None) -> bool:
seen: set[int] = set()
while exc is not None and id(exc) not in seen:
if isinstance(exc, ProviderQuotaExhaustedError):
return True
seen.add(id(exc))
exc = exc.__cause__ or exc.__context__
return False
async def _park_claimed_work_item_for_quota(
self,
member_session: CompanyMemberSession,
task: Task,
exc: Exception,
) -> None:
"""Return the item to READY and back off instead of failing it.
An exhausted provider quota is an environment outage, not a defect in
the work: failing the card (the previous behavior) terminally killed
intake and with it the whole run, with no way to continue after the
quota window reset (OBS-6). Parking keeps the run alive; dispatch
resumes automatically after the backoff, and stop/resume stays
available throughout.
"""
projection_id = self._projection_id_for_task(task)
now = time.monotonic()
if now - self._quota_last_park_at > 1800:
self._quota_park_streak = 0
self._quota_park_streak += 1
self._quota_last_park_at = now
backoff_sec = min(60 * (2 ** (self._quota_park_streak - 1)), 900)
self._quota_park_until = now + backoff_sec
summary = (
f"[Company:{projection_id}] provider quota/rate limit exhausted — "
f"work item returned to the queue; dispatch pauses for {backoff_sec}s "
f"(streak {self._quota_park_streak}). {str(exc)[:300]}"
)
logger.warning(summary)
if self._claimed_work_item_needs_cleanup(member_session, task):
try:
await transition_work_item_from_task(
self.store, task,
target_status_or_phase=Phase.READY,
reason="provider_quota_exhausted",
summary=summary or None,
release_claim=True,
attempt_outcome="interrupted",
)
except Exception:
logger.opt(exception=True).error(
f"[Company:{projection_id}] quota park: READY transition failed"
)
self.runtime._claimed_task_ids.discard(task.id)
work_item_id = linked_work_item_id_for_task(task)
if work_item_id:
self.runtime._claimed_work_item_ids.discard(work_item_id)
member_session.status = "idle"
member_session.resident_status = "idle"
member_session.current_task_id = ""
member_session.focused_work_item_id = ""
member_session.current_work_item = {}
member_session.current_assignment = {}
member_session.updated_at = datetime.now()
await self._emit_progress(summary, task_id=task.id)
async def _handle_claimed_work_item_exception(
self,
member_session: CompanyMemberSession,
task: Task,
exc: Exception,
) -> None:
if self._exception_is_provider_quota(exc):
await self._park_claimed_work_item_for_quota(member_session, task, exc)
return
projection_id = self._projection_id_for_task(task)
work_item_id = linked_work_item_id_for_task(task)
summary = (
@@ -13898,6 +13990,126 @@ class CompanyWorkItemExecutor:
except Exception:
logger.opt(exception=True).debug("Failed to save delegation run owner review lifecycle update")
async def _settle_run_lifecycle_on_convergence(self, tasks: list[Task]) -> None:
"""Close the delegation run when its tree converged in terminal failure.
The success path closes through delivery review (awaiting_owner plus
the feedback card). A run whose intake or delivery failed previously
stayed running/active forever: no closure signal, no card, and new
session input was routed into resuming the dead run (OBS-5).
"""
if not self.store or not hasattr(self.store, "get_delegation_run"):
return
run_id = self._delegation_run_id_for_tasks(tasks)
if not run_id:
return
list_work_items = getattr(self.store, "list_delegation_work_items", None)
if not callable(list_work_items):
return
try:
work_items = await list_work_items(run_id)
except Exception:
logger.opt(exception=True).debug("run failure settlement: work item load failed")
return
if not work_items:
return
if any(getattr(item, "phase", None) not in DONE_PHASES for item in work_items):
return
failed_core = [
item
for item in work_items
if str(getattr(item, "kind", "") or "").strip().lower() in {"intake", "delivery"}
and getattr(item, "phase", None) in {Phase.FAILED, Phase.CANCELLED}
]
if not failed_core:
return
try:
run = await self.store.get_delegation_run(run_id)
except Exception:
logger.opt(exception=True).debug("run failure settlement: run load failed")
return
if run is None:
return
lifecycle = str(getattr(run, "lifecycle_status", "") or "").strip()
if lifecycle in {"closed_failed", "awaiting_owner"}:
return
failed_items = [
{
"work_item_id": str(getattr(item, "work_item_id", "") or ""),
"kind": str(getattr(item, "kind", "") or ""),
"role_id": str(getattr(item, "role_id", "") or ""),
"phase": str(getattr(getattr(item, "phase", None), "value", "") or ""),
"blocked_reason": str(getattr(item, "blocked_reason", "") or "")[:300],
}
for item in work_items
if getattr(item, "phase", None) in {Phase.FAILED, Phase.CANCELLED}
]
closed_at = datetime.now().isoformat()
run.status = (
"failed"
if any(item["phase"] == Phase.FAILED.value for item in failed_items)
else "cancelled"
)
run.lifecycle_status = "closed_failed"
run.metadata = {
**dict(run.metadata or {}),
"run_failure": {
"closed_at": closed_at,
"failed_items": failed_items,
},
}
try:
await self.store.save_delegation_run(run)
except Exception:
logger.opt(exception=True).debug("run failure settlement: run save failed")
return
origin_task = tasks[0] if tasks else None
failure_lines = "\n".join(
f"- `{item['kind']}::{item['role_id']}` {item['phase']}"
+ (f" — {item['blocked_reason']}" if item["blocked_reason"] else "")
for item in failed_items
)
await self._emit_progress(
f"[Company] run closed after terminal failure — {len(failed_items)} "
f"failed/cancelled work item(s). Send a new request to start a fresh run.",
task_id=origin_task.id if origin_task else "",
)
if self.checkpoint_callback and origin_task is not None:
original_request = str(
(origin_task.metadata or {}).get("original_request", "")
or origin_task.description
or origin_task.title
or ""
).strip()
try:
await self.checkpoint_callback(
{
"checkpoint_type": "company_run_failure_review",
"project_id": origin_task.project_id,
"session_id": origin_task.session_id,
"task_id": origin_task.id,
"payload": {
"run_id": run_id,
"waiting_task_id": origin_task.id,
"session_id": origin_task.session_id,
"task_ids": [t.id for t in tasks],
"closed_at": closed_at,
"failed_items": failed_items,
"original_request": original_request,
"prompt": (
"This company run ended with failed work items and has "
"been closed.\n\n"
f"{failure_lines}\n\n"
"Reply `dismiss` to acknowledge, or send a new request "
"(for example the original goal with adjustments) to "
"start a fresh run."
),
},
}
)
except Exception:
logger.opt(exception=True).debug("run failure settlement: checkpoint save failed")
async def _finalize_completed_work_item(self, task: Task) -> None:
if self._is_authoritative_delivery_work_item(task):
plan = self._active_plan or CompanyWorkItemRuntimePlan(
opc/layer3_agent/runtime_v2/runtime.py
+18 -1
View File
@@ -39,7 +39,7 @@ from opc.layer4_tools.output_budget import clip_text
from opc.layer4_tools.registry import ToolDefinition
from opc.layer4_tools.registry import ToolRegistry
from opc.layer6_observability.cost_tracker import CostEntry
from opc.llm.provider import LLMProvider
from opc.llm.provider import LLMProvider, ProviderQuotaExhaustedError
ApprovalCallback = Callable[[ToolDefinition, dict[str, Any], Optional[Task], Any], Awaitable[tuple[bool, Any]]]
@@ -537,6 +537,23 @@ class NativeRuntimeV2:
},
)
except Exception as exc:
rate_limit_checker = getattr(self.llm, "is_rate_limit_error", None)
if callable(rate_limit_checker) and rate_limit_checker(exc):
# Quota/rate-limit rejections never reach the model, so
# conversation-feedback retries cannot help — surface a
# typed error for the dispatcher to park on instead of
# burning retries and failing the work item (OBS-6).
await self._cancel_early_tool_runs(early_tool_runs)
await self._emit_runtime_event(
runtime_session_id,
task,
"provider_quota_exhausted",
{
"iteration": iteration + 1,
"message": str(exc)[:600],
},
)
raise ProviderQuotaExhaustedError(str(exc)) from exc
if self.llm.is_context_overflow_error(exc) and overflow_retries < max_overflow_retries:
overflow_retries += 1
messages = await self._apply_context_pipeline(
opc/llm/provider.py
+48
View File
@@ -204,6 +204,16 @@ def _parse_tool_arguments(tool_name: str, arguments: Any) -> tuple[Any, str | No
return raw, raw, error
class ProviderQuotaExhaustedError(RuntimeError):
"""The provider rejected the request for quota/rate-limit reasons.
Raised by the agent runtime instead of retrying in place: replaying the
same payload against an exhausted quota can only fail, so the company
dispatcher parks the work (returns the item to READY and backs off)
rather than failing it terminally (OBS-6).
"""
class LLMProvider:
"""Unified LLM interface via LiteLLM supporting tool calls."""
@@ -401,6 +411,44 @@ class LLMProvider:
raw = json.dumps(payload, ensure_ascii=False, sort_keys=True, default=str)
return hashlib.sha256(raw.encode("utf-8")).hexdigest()
def is_rate_limit_error(self, error: Exception) -> bool:
"""Classify provider quota / rate-limit rejections.
These never produce model output and never get better by replaying
the identical payload, so callers must park/back off instead of
burning conversation-feedback retries (OBS-6). Classification is by
exception type when available and by error text otherwise — the
streaming path re-raises provider errors as plain RuntimeError with
only the message preserved.
"""
if isinstance(error, litellm.exceptions.RateLimitError):
return True
if "ratelimit" in type(error).__name__.lower():
return True
if getattr(error, "status_code", None) == 429:
return True
message = str(error).lower()
keywords = (
"rate limit",
"rate_limit",
"ratelimit",
"too many requests",
"insufficient_quota",
"quota exceeded",
"exceeded your quota",
"quota exhausted",
"error code: 429",
"status code: 429",
"http 429",
# Chinese-provider spellings of the same rejection (Volces/DeepSeek
# and other domestic endpoints return localized error text).
"请求过于频繁",
"配额已用完",
"配额耗尽",
"触发限流",
)
return any(keyword in message for keyword in keywords)
def is_context_overflow_error(self, error: Exception) -> bool:
if isinstance(error, litellm.exceptions.ContextWindowExceededError):
return True