fix(llm): resolve context window via max_input_tokens with 128k fallback for unmapped models

- get_context_window() now reads litellm.get_model_info().max_input_tokens
  instead of get_max_tokens(), which returns the output cap and severely
  under-reported the window for every mapped model (e.g. deepseek 8k vs 1M)
- models litellm cannot map fall back to 128000 with a single warning per
  model instead of warning on every call and returning None
- raise LLMConfig.max_tokens default 8192 -> 32768 to match the template
- README: configure the API key directly in llm_config.yaml; document
  max_tokens / context_window in the example

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
LZH-YS1998
2026-07-04 17:53:12 +08:00
parent 08e48c2f9c
commit 6c8d3f3dc9
5 changed files with 82 additions and 38 deletions
+4 -3
View File
@@ -272,12 +272,13 @@ class LLMConfig(BaseModel):
routing: dict[str, str] = Field(default_factory=dict)
fallback: dict[str, Any] = Field(default_factory=dict)
temperature: float = 0.3
max_tokens: int = 8192
max_tokens: int = 32768
# Total input context window (tokens) for the active model. Set this when
# the model is not mapped in litellm (e.g. proxy/self-hosted models like
# doubao/minimax/glm), so the context-usage ring and compaction thresholds
# have a real denominator. 0 = auto-detect via litellm. Optional per-model
# overrides keyed by model name take precedence over the scalar value.
# have a real denominator. 0 = auto-detect via litellm; unmapped models
# fall back to 128000. Optional per-model overrides keyed by model name
# take precedence over the scalar value.
context_window: int = 0
context_window_overrides: dict[str, int] = Field(default_factory=dict)