---
name: hermes-troubleshooting
description: "Use when Hermes gateway, auth, or platform issues arise."
version: 1.0.0
category: devops
metadata:
  hermes:
    tags: [hermes, troubleshooting, gateway, auth, profile, feishu]
---

# Hermes Agent Troubleshooting

Diagnose and fix common Hermes Agent issues: provider authentication failures, gateway crashes, profile isolation, and messaging platform problems.

## Profile Isolation — Always Check the Right Profile

Hermes profiles have **separate** configs, logs, and credentials. A critical first step:

```bash
hermes profile list    # See all profiles and which gateway is running
hermes status          # Check current profile's config
```

**Key insight:** When a messaging platform (Feishu, Telegram, etc.) reports an error, the gateway is running under a **different profile** than the TUI session you're currently in. The TUI session's logs won't contain the platform's errors.

### Log Locations Per Profile

| Profile | Logs Directory |
|---------|---------------|
| default | `/opt/data/logs/` |
| fey     | `/opt/data/profiles/fey/logs/` |
| other   | `/opt/data/profiles/<name>/logs/` |

Each profile has `agent.log`, `errors.log`, and `gateway.log` (if gateway is running).

## Provider Authentication Failed

**Symptom:** User on a messaging platform sees "Provider authentication failed. Check the configured credentials; raw provider details are in the gateway logs."

**Root cause:** This almost always means the **model API key** (in `config.yaml` → `model.api_key`) is invalid/expired, NOT the platform credentials (App ID, App Secret, etc.).

### Diagnostic Steps

1. **Find which profile the gateway runs under:**
   ```bash
   hermes profile list
   # Look for the profile with "running" gateway
   ```

2. **Check that profile's agent.log for the real error:**
   ```bash
   grep -i "401\|auth.*fail\|invalid.*key\|user_key" /opt/data/logs/agent.log | tail -20
   ```
   (Adjust path for non-default profiles.)

3. **Check the model config:**
   ```bash
   grep "api_key\|base_url\|model:" /opt/data/config.yaml | head -10
   ```

4. **Check platform config (to confirm platform creds are fine):**
   ```bash
   grep -i "feishu\|telegram\|discord" /opt/data/.env | head -10
   ```

### Common Causes

- **Custom proxy endpoint key expired:** If `model.base_url` points to a proxy (e.g., `http://113.249.102.8:18080/...`), the proxy's API key may have expired. The proxy returns `401: Authentication failed: invalid user_key`.
- **API key rotated elsewhere:** The key was changed in the provider's dashboard but not in Hermes `.env`/`config.yaml`.
- **Wrong profile's key:** The key in the gateway profile doesn't match the one you think is active.
- **Key format mismatch when switching providers:** Different providers use different key formats. A key valid for one provider (e.g., `sk-sp-...` for a custom proxy) will **not** work for another (e.g., DashScope expects `sk-ant-...`, DeepSeek expects `sk-...`). When changing `model.base_url`, you must also provide a key from the new provider. Verify the key with a quick curl test before restarting the gateway:
  ```bash
  curl -s -o /dev/null -w "%{http_code}" https://api.deepseek.com/v1/chat/completions \
    -H "Authorization: Bearer YOUR_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"hi"}],"max_tokens":1}'
  # Expect 200, not 401
  ```

### Fix

```bash
# Update the API key in the correct profile's config
hermes -p <profile_name> config set model.api_key 'new-key'

# If also changing the provider endpoint:
hermes -p <profile_name> config set model.base_url 'https://api.provider.com/v1'
hermes -p <profile_name> config set model.default 'model-name'

# Restart gateway (may need s6 kill pattern — see references/s6-gateway-restart.md)
hermes gateway restart
```

### `hermes` CLI Not on PATH

In containerized setups, `hermes` may not be on the default PATH. Use the full path:
```bash
/opt/hermes/.venv/bin/hermes -p default config set model.api_key 'new-key'
```

### WebSocket 1011 and "session not found"

These are **symptoms**, not root causes. When the model API returns 401 repeatedly:
- Gateway's conversation loop fails → internal error → WebSocket close code 1011
- Session cache expires during the outage → "session not found"
- Gateway may disconnect/reconnect the platform adapter

Fix the auth issue first; these resolve automatically.

## Gateway Not Running

**Symptom:** `hermes gateway status` shows "stopped" or messaging platforms show "not configured".

### Check

```bash
hermes gateway status
hermes profile list    # See which profiles have running gateways
```

### Fix

```bash
hermes gateway run     # Foreground (for testing)
hermes gateway install # Install as background service
hermes gateway start   # Start the service
```

If gateway is installed but stopped (e.g., in a container with s6):
```bash
# Check if it's a container environment
hermes status | grep "Manager"
# If s6: the service may need manual start or container restart
```

**Cannot restart from inside gateway?** See `references/s6-gateway-restart.md` for the kill + s6 auto-restart pattern.

## Gateway Crash Loop

**Symptom:** Gateway keeps dying and restarting.

### Check crash logs

```bash
# TUI gateway crash log (thread stacks at crash)
tail -100 ~/.hermes/logs/tui_gateway_crash.log

# Systemd service failures
systemctl --user status hermes-gateway
journalctl --user -u hermes-gateway --since "1 hour ago"
```

### Common causes

- **SSH logout kills gateway:** Enable linger: `sudo loginctl enable-linger $USER`
- **WSL2 close kills gateway:** Requires `systemd=true` in `/etc/wsl.conf`
- **Reset failed state:** `systemctl --user reset-failed hermes-gateway`

## Feishu-Specific Issues

### Feishu shows "not configured" in one profile but works in another

Feishu credentials are per-profile. Check the profile that runs the gateway:

```bash
# In the gateway's profile:
grep "FEISHU" ~/.hermes/.env
# Should have: FEISHU_APP_ID, FEISHU_APP_SECRET, FEISHU_CONNECTION_MODE
```

### Feishu WebSocket disconnects

Normal disconnects (code 1000 "bye") happen during gateway restart. Abnormal disconnects (code 1011) indicate internal errors — usually model auth failures.

## Quick Diagnostic Checklist

When something is broken:

1. `hermes profile list` — Which profile is running?
2. `hermes status` — Is gateway running? Are platforms configured?
3. `hermes doctor` — Detailed diagnostics
4. Check the **right profile's** logs — not the TUI session's logs
5. `grep "401\|auth\|fail" agent.log` — Find the real error behind generic messages