figmar
184 lines
6.5 KiB
Markdown
184 lines
6.5 KiB
Markdown
---
|
|
name: hermes-troubleshooting
|
|
description: "Use when Hermes gateway, auth, or platform issues arise."
|
|
version: 1.0.0
|
|
category: devops
|
|
metadata:
|
|
hermes:
|
|
tags: [hermes, troubleshooting, gateway, auth, profile, feishu]
|
|
---
|
|
|
|
# Hermes Agent Troubleshooting
|
|
|
|
Diagnose and fix common Hermes Agent issues: provider authentication failures, gateway crashes, profile isolation, and messaging platform problems.
|
|
|
|
## Profile Isolation — Always Check the Right Profile
|
|
|
|
Hermes profiles have **separate** configs, logs, and credentials. A critical first step:
|
|
|
|
```bash
|
|
hermes profile list # See all profiles and which gateway is running
|
|
hermes status # Check current profile's config
|
|
```
|
|
|
|
**Key insight:** When a messaging platform (Feishu, Telegram, etc.) reports an error, the gateway is running under a **different profile** than the TUI session you're currently in. The TUI session's logs won't contain the platform's errors.
|
|
|
|
### Log Locations Per Profile
|
|
|
|
| Profile | Logs Directory |
|
|
|---------|---------------|
|
|
| default | `/opt/data/logs/` |
|
|
| fey | `/opt/data/profiles/fey/logs/` |
|
|
| other | `/opt/data/profiles/<name>/logs/` |
|
|
|
|
Each profile has `agent.log`, `errors.log`, and `gateway.log` (if gateway is running).
|
|
|
|
## Provider Authentication Failed
|
|
|
|
**Symptom:** User on a messaging platform sees "Provider authentication failed. Check the configured credentials; raw provider details are in the gateway logs."
|
|
|
|
**Root cause:** This almost always means the **model API key** (in `config.yaml` → `model.api_key`) is invalid/expired, NOT the platform credentials (App ID, App Secret, etc.).
|
|
|
|
### Diagnostic Steps
|
|
|
|
1. **Find which profile the gateway runs under:**
|
|
```bash
|
|
hermes profile list
|
|
# Look for the profile with "running" gateway
|
|
```
|
|
|
|
2. **Check that profile's agent.log for the real error:**
|
|
```bash
|
|
grep -i "401\|auth.*fail\|invalid.*key\|user_key" /opt/data/logs/agent.log | tail -20
|
|
```
|
|
(Adjust path for non-default profiles.)
|
|
|
|
3. **Check the model config:**
|
|
```bash
|
|
grep "api_key\|base_url\|model:" /opt/data/config.yaml | head -10
|
|
```
|
|
|
|
4. **Check platform config (to confirm platform creds are fine):**
|
|
```bash
|
|
grep -i "feishu\|telegram\|discord" /opt/data/.env | head -10
|
|
```
|
|
|
|
### Common Causes
|
|
|
|
- **Custom proxy endpoint key expired:** If `model.base_url` points to a proxy (e.g., `http://113.249.102.8:18080/...`), the proxy's API key may have expired. The proxy returns `401: Authentication failed: invalid user_key`.
|
|
- **API key rotated elsewhere:** The key was changed in the provider's dashboard but not in Hermes `.env`/`config.yaml`.
|
|
- **Wrong profile's key:** The key in the gateway profile doesn't match the one you think is active.
|
|
- **Key format mismatch when switching providers:** Different providers use different key formats. A key valid for one provider (e.g., `sk-sp-...` for a custom proxy) will **not** work for another (e.g., DashScope expects `sk-ant-...`, DeepSeek expects `sk-...`). When changing `model.base_url`, you must also provide a key from the new provider. Verify the key with a quick curl test before restarting the gateway:
|
|
```bash
|
|
curl -s -o /dev/null -w "%{http_code}" https://api.deepseek.com/v1/chat/completions \
|
|
-H "Authorization: Bearer YOUR_KEY" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model":"deepseek-chat","messages":[{"role":"user","content":"hi"}],"max_tokens":1}'
|
|
# Expect 200, not 401
|
|
```
|
|
|
|
### Fix
|
|
|
|
```bash
|
|
# Update the API key in the correct profile's config
|
|
hermes -p <profile_name> config set model.api_key 'new-key'
|
|
|
|
# If also changing the provider endpoint:
|
|
hermes -p <profile_name> config set model.base_url 'https://api.provider.com/v1'
|
|
hermes -p <profile_name> config set model.default 'model-name'
|
|
|
|
# Restart gateway (may need s6 kill pattern — see references/s6-gateway-restart.md)
|
|
hermes gateway restart
|
|
```
|
|
|
|
### `hermes` CLI Not on PATH
|
|
|
|
In containerized setups, `hermes` may not be on the default PATH. Use the full path:
|
|
```bash
|
|
/opt/hermes/.venv/bin/hermes -p default config set model.api_key 'new-key'
|
|
```
|
|
|
|
### WebSocket 1011 and "session not found"
|
|
|
|
These are **symptoms**, not root causes. When the model API returns 401 repeatedly:
|
|
- Gateway's conversation loop fails → internal error → WebSocket close code 1011
|
|
- Session cache expires during the outage → "session not found"
|
|
- Gateway may disconnect/reconnect the platform adapter
|
|
|
|
Fix the auth issue first; these resolve automatically.
|
|
|
|
## Gateway Not Running
|
|
|
|
**Symptom:** `hermes gateway status` shows "stopped" or messaging platforms show "not configured".
|
|
|
|
### Check
|
|
|
|
```bash
|
|
hermes gateway status
|
|
hermes profile list # See which profiles have running gateways
|
|
```
|
|
|
|
### Fix
|
|
|
|
```bash
|
|
hermes gateway run # Foreground (for testing)
|
|
hermes gateway install # Install as background service
|
|
hermes gateway start # Start the service
|
|
```
|
|
|
|
If gateway is installed but stopped (e.g., in a container with s6):
|
|
```bash
|
|
# Check if it's a container environment
|
|
hermes status | grep "Manager"
|
|
# If s6: the service may need manual start or container restart
|
|
```
|
|
|
|
**Cannot restart from inside gateway?** See `references/s6-gateway-restart.md` for the kill + s6 auto-restart pattern.
|
|
|
|
## Gateway Crash Loop
|
|
|
|
**Symptom:** Gateway keeps dying and restarting.
|
|
|
|
### Check crash logs
|
|
|
|
```bash
|
|
# TUI gateway crash log (thread stacks at crash)
|
|
tail -100 ~/.hermes/logs/tui_gateway_crash.log
|
|
|
|
# Systemd service failures
|
|
systemctl --user status hermes-gateway
|
|
journalctl --user -u hermes-gateway --since "1 hour ago"
|
|
```
|
|
|
|
### Common causes
|
|
|
|
- **SSH logout kills gateway:** Enable linger: `sudo loginctl enable-linger $USER`
|
|
- **WSL2 close kills gateway:** Requires `systemd=true` in `/etc/wsl.conf`
|
|
- **Reset failed state:** `systemctl --user reset-failed hermes-gateway`
|
|
|
|
## Feishu-Specific Issues
|
|
|
|
### Feishu shows "not configured" in one profile but works in another
|
|
|
|
Feishu credentials are per-profile. Check the profile that runs the gateway:
|
|
|
|
```bash
|
|
# In the gateway's profile:
|
|
grep "FEISHU" ~/.hermes/.env
|
|
# Should have: FEISHU_APP_ID, FEISHU_APP_SECRET, FEISHU_CONNECTION_MODE
|
|
```
|
|
|
|
### Feishu WebSocket disconnects
|
|
|
|
Normal disconnects (code 1000 "bye") happen during gateway restart. Abnormal disconnects (code 1011) indicate internal errors — usually model auth failures.
|
|
|
|
## Quick Diagnostic Checklist
|
|
|
|
When something is broken:
|
|
|
|
1. `hermes profile list` — Which profile is running?
|
|
2. `hermes status` — Is gateway running? Are platforms configured?
|
|
3. `hermes doctor` — Detailed diagnostics
|
|
4. Check the **right profile's** logs — not the TUI session's logs
|
|
5. `grep "401\|auth\|fail" agent.log` — Find the real error behind generic messages
|