fix(llm): resolve context window via max_input_tokens with 128k fallback for unmapped models
- get_context_window() now reads litellm.get_model_info().max_input_tokens instead of get_max_tokens(), which returns the output cap and severely under-reported the window for every mapped model (e.g. deepseek 8k vs 1M) - models litellm cannot map fall back to 128000 with a single warning per model instead of warning on every call and returning None - raise LLMConfig.max_tokens default 8192 -> 32768 to match the template - README: configure the API key directly in llm_config.yaml; document max_tokens / context_window in the example Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
+4
-3
@@ -272,12 +272,13 @@ class LLMConfig(BaseModel):
|
||||
routing: dict[str, str] = Field(default_factory=dict)
|
||||
fallback: dict[str, Any] = Field(default_factory=dict)
|
||||
temperature: float = 0.3
|
||||
max_tokens: int = 8192
|
||||
max_tokens: int = 32768
|
||||
# Total input context window (tokens) for the active model. Set this when
|
||||
# the model is not mapped in litellm (e.g. proxy/self-hosted models like
|
||||
# doubao/minimax/glm), so the context-usage ring and compaction thresholds
|
||||
# have a real denominator. 0 = auto-detect via litellm. Optional per-model
|
||||
# overrides keyed by model name take precedence over the scalar value.
|
||||
# have a real denominator. 0 = auto-detect via litellm; unmapped models
|
||||
# fall back to 128000. Optional per-model overrides keyed by model name
|
||||
# take precedence over the scalar value.
|
||||
context_window: int = 0
|
||||
context_window_overrides: dict[str, int] = Field(default_factory=dict)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user