_parse_session_key() blindly assigned parts[5] as thread_id for all
chat types. For group sessions with per-user isolation, parts[5] is
a user_id, not a thread_id. This could cause shutdown notifications
to route with incorrect thread metadata.
Only return thread_id for chat types where the 6th element is
unambiguous: dm and thread. For group/channel sessions, omit
thread_id since the suffix may be a user_id.
Based on the approach from PR #9938 by @Ruzzgar.
Commentary messages (interim assistant status updates like "Using browser
tool...") are sent via _send_commentary(), which was incorrectly setting
_already_sent = True on success. This caused the final response to be
suppressed when there were multiple tool calls, because the gateway checks
already_sent to decide whether to skip re-sending the response.
The fix: commentary messages are interim status updates, not the final
response, so _already_sent should not be set when they succeed. This
ensures the final response is always delivered regardless of how many
commentary messages were sent during the turn.
Fixes: #10454
After clear_session_vars() reset contextvars to their default (''),
get_session_env() treated the empty string as falsy and fell through
to os.environ — resurrecting stale HERMES_SESSION_* values from CLI
startup, cron, or previous sessions. This broke session isolation
in the gateway where concurrent messages could see each other's
stale environment values.
Fix: use a sentinel (_UNSET) as the contextvar default instead of ''.
get_session_env() now checks 'value is not _UNSET' instead of
truthiness. Three states are cleanly distinguished:
- _UNSET (never set): fall back to os.environ (CLI/cron compat)
- '' (explicitly cleared): return '' — no os.environ fallback
- 'telegram' (actively set): return the value
clear_session_vars() now uses var.set('') instead of var.reset(token)
to mark vars as explicitly cleared rather than reverting to _UNSET.
Closes#10304
When a model (e.g. mimo-v2-pro) streams intermediate text alongside tool
calls ("Let me search for that") but then returns empty after processing
tool results, the stream consumer already_sent flag is True from the
earlier text delivery. The gateway suppression check
(already_sent=True, failed=False → return None) would swallow the final
response, leaving the user staring at silence after the search.
Two changes:
1. gateway/run.py return path: skip already_sent suppression when the
final_response is "(empty)" or empty — the user needs to know the
agent finished even if streaming sent partial content earlier.
2. gateway/run.py response handler: convert the internal "(empty)"
sentinel to a user-friendly warning instead of delivering the raw
sentinel string.
Tests added for all empty/None/sentinel cases plus preserved existing
suppression behavior for normal non-empty responses.
Discord's _register_slash_commands() had a hardcoded list of ~27 commands
while COMMAND_REGISTRY defines 34+ gateway-available commands. Missing
commands (debug, branch, rollback, snapshot, profile, yolo, fast, reload,
commands) were invisible in Discord's / autocomplete — users couldn't
discover them.
Add a dynamic catch-all loop after the explicit registrations that
iterates COMMAND_REGISTRY, skips already-registered commands, and
auto-registers the rest using discord.app_commands.Command(). Commands
with args_hint get an optional string parameter; parameterless commands
get a simple callback.
This ensures any future commands added to COMMAND_REGISTRY automatically
appear on Discord without needing a manual entry in discord.py.
Telegram and Slack already derive dynamically from COMMAND_REGISTRY
via telegram_bot_commands() and slack_subcommand_map() — no changes
needed there.
- Pastes uploaded by /debug now auto-delete after 1 hour via a detached
background process that sends DELETE to paste.rs
- CLI: shows privacy notice listing what data will be uploaded
- Gateway: only uploads summary report (system info + log tails), NOT
full log files containing conversation content
- Added 'hermes debug delete <url>' for immediate manual deletion
- 16 new tests covering auto-delete scheduling, paste deletion, privacy
notices, and the delete subcommand
Addresses user privacy concern where /debug uploaded full conversation
logs to a public paste service with no warning or expiry.
Two gateway fixes:
1. MessageDeduplicator.is_duplicate() now checks TTL at query time (#10306)
Previously, is_duplicate() returned True for any previously seen ID
without checking its age — expired entries were only purged when cache
size exceeded max_size. On normal workloads that never overflow, message
IDs stayed deduplicated forever instead of expiring after the TTL.
Fix: check `now - timestamp < ttl` before returning True. Expired
entries are removed and treated as new messages.
2. Gateway --config flag now uses yaml.safe_load() (#10216)
The --config CLI flag in gateway/run.py main() used json.load() to
parse config files. YAML is the only documented config format and
every other config loader uses yaml.safe_load(). A YAML config file
passed via --config would crash with json.JSONDecodeError.
Closes#10306Closes#10216
_parse_session_key() now extracts the optional 6th part (thread_id) from
session keys, and _notify_active_sessions_of_shutdown uses _parsed.get()
instead of the removed 'parts' variable. Without this, shutdown notifications
silently failed (NameError caught by try/except) and forum topic routing
was lost.
- Populate watcher_* routing fields for watch-only processes (not just
notify_on_complete), so watch-pattern events carry direct metadata
instead of relying solely on session_key parsing fallback
- Extract _parse_session_key() helper to dedupe session key parsing
at two call sites in gateway/run.py
- Add negative test proving cross-thread leakage doesn't happen
- Add edge-case tests for _build_process_event_source returning None
(empty evt, invalid platform, short session_key)
- Add unit tests for _parse_session_key helper
Three fixes for the duplicate reply bug affecting all gateway platforms:
1. base.py: Suppress stale response when the session was interrupted by a
new message that hasn't been consumed yet. Checks both interrupt_event
and _pending_messages to avoid false positives. (#8221, #2483)
2. run.py (return path): Remove response_previewed guard from already_sent
check. Stream consumer's already_sent alone is authoritative — if
content was delivered via streaming, the duplicate send must be
suppressed regardless of the agent's response_previewed flag. (#8375)
3. run.py (queued-message path): Same fix — already_sent without
response_previewed now correctly marks the first response as already
streamed, preventing re-send before processing the queued message.
The response_previewed field is still produced by the agent (run_agent.py)
but is no longer required as a gate for duplicate suppression. The stream
consumer's already_sent flag is the delivery-level truth about what the
user actually saw.
Concepts from PR #8380 (konsisumer). Closes#8375, #8221, #2483.
Three independent fixes:
1. Reset activity timestamp on cached agent reuse (#9051)
When the gateway reuses a cached AIAgent for a new turn, the
_last_activity_ts from the previous turn (possibly hours ago)
carried over. The inactivity timeout handler immediately saw
the agent as idle for hours and killed it.
Fix: reset _last_activity_ts, _last_activity_desc, and
_api_call_count when retrieving an agent from the cache.
2. Detect uv-managed virtual environments (#8620 sub-issue 1)
The systemd unit generator fell back to sys.executable (uv's
standalone Python) when running under 'uv run', because
sys.prefix == sys.base_prefix (uv doesn't set up traditional
venv activation). The generated ExecStart pointed to a Python
binary without site-packages, crashing the service on startup.
Fix: check VIRTUAL_ENV env var before falling back to
sys.executable. uv sets VIRTUAL_ENV even when sys.prefix
doesn't reflect the venv.
3. Nudge model to continue after empty post-tool response (#9400)
Weaker models (GLM-5, mimo-v2-pro) sometimes return empty
responses after tool calls instead of continuing to the next
step. The agent silently abandoned the remaining work with
'(empty)' or used prior-turn fallback text.
Fix: when the model returns empty after tool calls AND there's
no prior-turn content to fall back on, inject a one-time user
nudge message telling the model to process the tool results and
continue. The flag resets after each successful tool round so it
can fire again on later rounds.
Test plan: 97 gateway + CLI tests pass, 9 venv detection tests pass
When a user sends a message while the agent is executing a task on the
gateway, the agent is now interrupted immediately — not silently queued.
Previously, messages were stored in _pending_messages with zero feedback
to the user, potentially leaving them waiting 1+ hours.
Root cause: Level 1 guard (base.py) intercepted all messages for active
sessions and returned with no response. Level 2 (gateway/run.py) which
calls agent.interrupt() was never reached.
Fix: Expand _handle_active_session_busy_message to handle the normal
(non-draining) case:
1. Call running_agent.interrupt(text) to abort in-flight tool calls
and exit the agent loop at the next check point
2. Store the message as pending so it becomes the next turn once the
interrupted run returns
3. Send a brief ack: 'Interrupting current task (10 min elapsed,
iteration 21/60, running: terminal). I'll respond shortly.'
4. Debounce acks to once per 30s to avoid spam on rapid messages
Reported by @Lonely__MH.
When compression fails after max attempts, the agent returns
{completed: False, partial: True} but was missing the 'failed' flag.
The gateway's agent_failed_early guard checked for 'failed' AND
'not final_response', but _run_agent_blocking always converts errors
to final_response — making the guard dead code. This caused the
oversized session to persist, creating an infinite fail loop where
every subsequent message hits the same compression failure.
Changes:
- run_agent.py: add 'failed: True' and 'compression_exhausted: True'
to all 5 compression-exhaustion return paths
- gateway/run.py (_run_agent_blocking): forward 'failed' and
'compression_exhausted' flags through to the caller
- gateway/run.py (_handle_message_with_agent): fix agent_failed_early
to check bool(failed) without the broken 'not final_response' clause;
auto-reset the session when compression is exhausted so the next
message starts fresh
- Update tests to match new guard logic and add
TestCompressionExhaustedFlag test class
Closes#9893
The /v1/responses endpoint generated a new UUID session_id for every
request, even when previous_response_id was provided. This caused each
turn of a multi-turn conversation to appear as a separate session on the
web dashboard, despite the conversation history being correctly chained.
Fix: store session_id alongside the response in the ResponseStore, and
reuse it when a subsequent request chains via previous_response_id.
Applies to both the non-streaming /v1/responses path and the streaming
SSE path. The /v1/runs endpoint also gains session continuity from
stored responses (explicit body.session_id still takes priority).
Adds test verifying session_id is preserved across chained requests.
* fix: hermes gateway restart waits for service to come back up (#8260)
Previously, systemd_restart() sent SIGUSR1 to the gateway, printed
'restart requested', and returned immediately. The gateway still
needed to drain active agents, exit with code 75, wait for systemd's
RestartSec=30, and start the new process. The user saw 'success' but
the gateway was actually down for 30-60 seconds.
Now the SIGUSR1 path blocks with progress feedback:
Phase 1 — wait for old process to die:
⏳ User service draining active work...
Polls os.kill(pid, 0) until ProcessLookupError (up to 90s)
Phase 2 — wait for new process to become active:
⏳ Waiting for hermes-gateway to restart...
Polls systemctl is-active + verifies new PID (up to 60s)
Success:
✓ User service restarted (PID 12345)
Timeout:
⚠ User service did not become active within 60s.
Check status: hermes gateway status
Check logs: journalctl --user -u hermes-gateway --since '2 min ago'
The reload-or-restart fallback path (line 1189) already blocks because
systemctl reload-or-restart is synchronous.
Test plan:
- Updated test to verify wait-for-restart behavior
- All 118 gateway CLI tests pass
* fix: add 402 billing error hint to gateway error handler (#5220)
The gateway's exception handler for agent errors had specific hints for
HTTP 401, 429, 529, 400, 500 — but not 402 (Payment Required / quota
exhausted). Users hitting billing limits from custom proxy providers
got a generic error with no guidance.
Added: 'Your API balance or quota is exhausted. Check your provider
dashboard.'
The underlying billing classification (error_classifier.py) already
correctly handles 402 as FailoverReason.billing with credential
rotation and fallback. The original issue (#5220) where 402 killed
the entire gateway was from an older version — on current main, 402
is excluded from the is_client_error abort path (line 9460) and goes
through the proper retry/fallback/fail flow. Combined with PR #9875
(auto-recover from unexpected SIGTERM), even edge cases where the
gateway dies are now survivable.
The streaming path emits output as content-part arrays for Open WebUI
compatibility, but the batch (non-streaming) Responses API path must
return output as a plain string per the OpenAI Responses API spec.
Reverts the _extract_output_items change from the cherry-picked commits
while preserving the streaming path's array format.
When a session gets stuck (hung terminal, runaway tool loop) and the
user restarts the gateway, the same session history loads and puts the
agent right back in the stuck state. The user is trapped in a loop:
restart → stuck → restart → stuck.
Fix: track restart-failure counts per session using a simple JSON file
(.restart_failure_counts). On each shutdown with active agents, the
counter increments for those sessions. On startup, if any session has
been active across 3+ consecutive restarts, it's auto-suspended —
giving the user a clean slate on their next message.
The counter resets to 0 when a session completes a turn successfully
(response delivered), so normal sessions that happen to be active
during planned restarts (/restart, hermes update) won't accumulate
false counts.
Implementation:
- _increment_restart_failure_counts(): called during stop() when
agents are active. Writes {session_key: count} to JSON file.
Sessions NOT active are dropped (loop broken).
- _suspend_stuck_loop_sessions(): called on startup. Reads the file,
suspends sessions at threshold (3), clears the file.
- _clear_restart_failure_count(): called after successful response
delivery. Removes the session from the counter file.
No SessionEntry schema changes. No database migration. Pure file-based
tracking that naturally cleans up.
Test plan:
- 9 new stuck-loop tests (increment, accumulate, threshold, clear,
suspend, file cleanup, edge cases)
- All 28 gateway lifecycle tests pass (restart drain + auto-continue
+ stuck loop)
When the gateway restarts mid-agent-work, the session transcript ends
on a tool result the agent never processed. Previously, the user had
to type 'continue' or use /retry (which replays from scratch, losing
all prior work).
Now, when the next user message arrives and the loaded history ends
with role='tool', a system note is prepended:
[System note: Your previous turn was interrupted before you could
process the last tool result(s). Please finish processing those
results and summarize what was accomplished, then address the
user's new message below.]
This is injected in _run_agent()'s run_sync closure, right before
calling agent.run_conversation(). The agent sees the full history
(including the pending tool results) and the system note, so it can
summarize what was accomplished and then handle the user's new input.
Design decisions:
- No new session flags or schema changes — purely detects trailing
tool messages in the loaded history
- Works for any restart scenario (clean, crash, SIGTERM, drain timeout)
as long as the session wasn't suspended (suspended = fresh start)
- The user's actual message is preserved after the note
- If the session WAS suspended (unclean shutdown), the old history is
abandoned and the user starts fresh — no false auto-continue
Also updates the shutdown notification message from 'Use /retry after
restart to continue' to 'Send any message after restart to resume
where it left off' — which is now accurate.
Test plan:
- 6 new auto-continue tests (trailing tool detection, no false
positives for assistant/user/empty history, multi-tool, message
preservation)
- All 13 restart drain tests pass (updated /retry assertion)
Instead of consuming one top-level slash command slot per skill (hitting the
100-command limit with ~26 built-ins + 74 skills), skills are now organized
under a single /skill group command with category-based subcommand groups:
/skill creative ascii-art [args]
/skill media gif-search [args]
/skill mlops axolotl [args]
Discord supports 25 subcommand groups × 25 subcommands = 625 max skills,
well beyond the previous 74-slot ceiling.
Categories are derived from the skill directory structure:
- skills/creative/ascii-art/ → category 'creative'
- skills/mlops/training/axolotl/ → category 'mlops' (top-level parent)
- skills/dogfood/ → uncategorized (direct subcommand)
Changes:
- hermes_cli/commands.py: add discord_skill_commands_by_category() with
category grouping, hub/disabled filtering, Discord limit enforcement
- gateway/platforms/discord.py: replace top-level skill registration with
_register_skill_group() using app_commands.Group hierarchy
- tests: 7 new tests covering group creation, category grouping,
uncategorized skills, hub exclusion, deep nesting, empty skills,
and handler dispatch
Inspired by Discord community suggestion from bottium.
When the gateway receives SIGTERM/SIGINT, the shutdown handler now
runs 'ps aux' and logs every hermes/gateway-related process (excluding
itself). This will show in agent.log as:
WARNING: Shutdown diagnostic — other hermes processes running:
hermes 1234 ... hermes update --gateway
hermes 5678 ... hermes gateway restart
This is the missing diagnostic for #5646 / #6666 — we can prove
the restarts are from systemctl but can't determine WHO issues the
systemctl command. Next time it happens, the agent.log will contain
the evidence (the process that sent the signal or called systemctl
should still be alive when the handler fires).
The dashboard's gateway status detection relied solely on local PID checks
(os.kill + /proc), which fails when the gateway runs in a separate container.
Changes:
- web_server.py: Add _probe_gateway_health() that queries the gateway's HTTP
/health/detailed endpoint when the local PID check fails. Activated by
setting the GATEWAY_HEALTH_URL env var (e.g. http://gateway:8642/health).
Falls back to standard PID check when the env var is not set.
- api_server.py: Add GET /health/detailed endpoint that returns full gateway
state (platforms, gateway_state, active_agents, pid, etc.) without auth.
The existing GET /health remains unchanged for backwards compatibility.
- StatusPage.tsx: Handle the case where gateway_pid is null but the gateway
is running remotely, displaying 'Running (remote)' instead of 'PID null'.
Environment variables:
- GATEWAY_HEALTH_URL: URL of the gateway health endpoint (e.g.
http://gateway-container:8642/health). Unset = local PID check only.
- GATEWAY_HEALTH_TIMEOUT: Probe timeout in seconds (default: 3).
Root cause: when the gateway received SIGTERM (from hermes update,
external kill, WSL2 runtime, etc.), it exited with status 0. systemd's
Restart=on-failure only restarts on non-zero exit, so the gateway
stayed dead permanently. Users had to manually restart.
Fix 1: Signal-initiated shutdown exits non-zero
When SIGTERM/SIGINT is received and no restart was requested (via
/restart, /update, or SIGUSR1), start_gateway() returns False which
causes sys.exit(1). systemd sees a failure exit and auto-restarts
after RestartSec=30.
This is safe because systemctl stop tracks its own stop-requested
state independently of exit code — Restart= never fires for a
deliberate stop, regardless of exit code.
Also logs 'Received SIGTERM/SIGINT — initiating shutdown' so the
cause of unexpected shutdowns is visible in agent.log.
Fix 2: PID file ownership guard
remove_pid_file() now checks that the PID file belongs to the current
process before removing it. During --replace handoffs, the old
process's atexit handler could fire AFTER the new process wrote its
PID file, deleting the new record. This left the gateway running but
invisible to get_running_pid(), causing 'Another gateway already
running' errors on next restart.
Test plan:
- All restart drain tests pass (13)
- All gateway service tests pass (84)
- All update gateway restart tests pass (34)
Feishu approval clicks need the resolved card to come back from the
synchronous callback path itself. Leaving approval resolution to the
generic asynchronous card-action flow made button feedback depend on
later loop work instead of the callback response the client is waiting
for.
Change-Id: I574997cbbcaa097fdba759b47367e28d1b56b040
Constraint: Feishu card-action callbacks must acknowledge quickly and reflect final approval state from the callback response path
Rejected: Keep approval handling on the generic async card-action route | leaves card state synchronization vulnerable to callback timing and follow-up update ordering
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep approval callback response construction separate from async queue unblocking unless Feishu callback semantics change
Tested: pytest tests/gateway/test_feishu.py tests/gateway/test_feishu_approval_buttons.py tests/gateway/test_approve_deny_commands.py tests/gateway/test_slack_approval_buttons.py tests/gateway/test_telegram_approval_buttons.py -q
Not-tested: Live Feishu workspace end-to-end callback rendering
Three fixes for gateway lifecycle stability:
1. Notify active sessions before shutdown (#new)
When the gateway receives SIGTERM or /restart, it now sends a
notification to every chat with an active agent BEFORE starting
the drain. Users see:
- Shutdown: 'Gateway shutting down — your task will be interrupted.'
- Restart: 'Gateway restarting — use /retry after restart to continue.'
Deduplicates per-chat so group sessions with multiple users get
one notification. Best-effort: send failures are logged and swallowed.
2. Skip .clean_shutdown marker when drain timed out
Previously, a graceful SIGTERM always wrote .clean_shutdown, even if
agents were force-interrupted when the drain timed out. This meant
the next startup skipped session suspension, leaving interrupted
sessions in a broken state (trailing tool response, no final message).
Now the marker is only written if the drain completed without timeout,
so interrupted sessions get properly suspended on next startup.
3. Post-restart health check for hermes update (#6631)
cmd_update() now verifies the gateway actually survived after
systemctl restart (sleep 3s + is-active check). If the service
crashed immediately, it retries once. If still dead, prints
actionable diagnostics (journalctl command, manual restart hint).
Also closes#8104 — already fixed on main (the /restart handler
correctly detects systemd via INVOCATION_ID and uses via_service=True).
Test plan:
- 6 new tests for shutdown notifications (dedup, restart vs shutdown
messaging, sentinel filtering, send failure resilience)
- Existing restart drain + update tests pass (47 total)
When BlueBubbles posts webhook events to the adapter, it uses the exact
URL registered via /api/v1/webhook — and BB's registration API does not
support custom headers. The adapter currently registers the bare URL
(no credentials), but then requires password auth on inbound POSTs,
rejecting every webhook with HTTP 401.
This is masked on fresh BB installs by a race condition: the webhook
might register once with a prior (possibly patched) URL and keep working
until the first restart. On v0.9.0, _unregister_webhook runs on clean
shutdown, so the next startup re-registers with the bare URL and the
401s begin. Users see the bot go silent with no obvious cause.
Root cause: there's no way to pass auth credentials from BB to the
webhook handler except via the URL itself. BB accepts query params and
preserves them on outbound POSTs.
## Fix
Introduce `_webhook_register_url` — the URL handed to BB's registration
API, with the configured password appended as a `?password=<value>`
query param. The existing webhook auth handler already accepts this
form (it reads `request.query.get("password")`), so no change to the
receive side is needed.
The bare `_webhook_url` is still used for logging and for binding the
local listener, so credentials don't leak into log output. Only the
registration/find/unregister paths use the password-bearing form.
## Notes
- Password is URL-encoded via urllib.parse.quote, handling special
characters (&, *, @, etc.) that would otherwise break parsing.
- Storing the password in BB's webhook table is not a new disclosure:
anyone with access to that table already has the BB admin password
(same credential used for every other API call).
- If `self.password` is empty (no auth configured), the register URL
is the bare URL — preserves current behavior for unauthenticated
local-only setups.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
BlueBubbles v1.9+ webhook payloads for new-message events do not always
include a top-level chatGuid field on the message data object. Instead,
the chat GUID is nested under data.chats[0].guid.
The adapter currently checks five top-level fallback locations (record and
payload, snake_case and camelCase, plus payload.guid) but never looks
inside the chats array. When none of those top-level fields contain the
GUID, the adapter falls through to using the sender's phone/email as the
session chat ID.
This causes two observable bugs when a user is a participant in both a DM
and a group chat with the bot:
1. DM and group sessions merge. Every message from that user ends up with
the same session_chat_id (their own address), so the bot cannot
distinguish which thread the message came from.
2. Outbound routing becomes ambiguous. _resolve_chat_guid() iterates all
chats and returns the first one where the address appears as a
participant; group chats typically sort ahead of DMs by activity, so
replies and cron messages intended for the DM can land in a group.
This was observed in production: a user's morning brief cron delivered to
a group chat with his spouse instead of his DM thread.
The fix adds a single fallback that extracts chat_guid from
record["chats"][0]["guid"] when the top-level fields are empty. The chats
array is included in every new-message webhook payload in BB v1.9.9
(verified against a live server). It is backwards compatible: if a future
BB version starts including chatGuid at the top level, that still wins.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The BlueBubbles adapter registers its webhook with three events:
["new-message", "updated-message", "message"]. The third, "message",
is not a valid event type in the BlueBubbles server API — BB rejects
the registration payload with HTTP 400 Bad Request.
Currently this is masked by the "crash resilience" check in
_register_webhook, which reuses any existing registration matching the
webhook URL and short-circuits before reaching the API call. So an
already-registered webhook from a prior run keeps working. But any fresh
install, or any restart after _unregister_webhook has run during a clean
shutdown, fails to re-register and silently stops receiving messages.
Observed in production: after a gateway restart in v0.9.0 (which auto-
unregisters on shutdown), the next startup hit this 400 and the bot went
silent until the invalid event was removed.
BlueBubbles documents "new-message" and "updated-message" as the message
event types (see https://docs.bluebubbles.app/). There is no "message"
event, and no harm in dropping it — the two remaining events cover all
inbound message webhooks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When GATEWAY_PROXY_URL (or gateway.proxy_url in config.yaml) is set,
the gateway becomes a thin relay: it handles platform I/O (encryption,
threading, media) and delegates all agent work to a remote Hermes API
server via POST /v1/chat/completions with SSE streaming.
This enables the primary use case of running a Matrix E2EE gateway in
Docker on Linux while the actual agent runs on the host (e.g. macOS)
with full access to local files, memory, skills, and a unified session
store. Works for any platform adapter, not just Matrix.
Configuration:
- GATEWAY_PROXY_URL env var (Docker-friendly)
- gateway.proxy_url in config.yaml
- GATEWAY_PROXY_KEY env var for API auth (matches API_SERVER_KEY)
- X-Hermes-Session-Id header for session continuity
Architecture:
- _get_proxy_url() checks env var first, then config.yaml
- _run_agent_via_proxy() handles HTTP forwarding with SSE streaming
- _run_agent() delegates to proxy path when URL is configured
- Platform streaming (GatewayStreamConsumer) works through proxy
- Returns compatible result dict for session store recording
Files changed:
- gateway/run.py: proxy mode implementation (~250 lines)
- hermes_cli/config.py: GATEWAY_PROXY_URL + GATEWAY_PROXY_KEY env vars
- tests/gateway/test_proxy_mode.py: 17 tests covering config
resolution, dispatch, HTTP forwarding, error handling, message
filtering, and result shape validation
Closes discussion from Cars29 re: Matrix gateway mixed-mode issue.
The /new and /reset commands were not calling shutdown_memory_provider()
on the cached agent before eviction. This caused OpenViking (and any
memory provider that relies on session-end shutdown) to skip commit,
leaving memories un-indexed until idle timeout or gateway shutdown.
Add the missing shutdown_memory_provider() call in _handle_reset_command(),
matching the behavior already present in the session expiry watcher.
Fixes#7759
During rapid tool-calling, the model often emits 1-2 tokens before
switching to tool calls. The stream consumer would create a new message
with 'X ▉' (short text + cursor), and if the follow-up edit to strip
the cursor was rate-limited by the platform, the cursor remained as
a permanent standalone message — reported on Telegram as 'white box'
artifacts.
Add a minimum-content guard in _send_or_edit: when creating a new
standalone message (no existing message_id), require at least 4
visible characters alongside the cursor before sending. Shorter text
accumulates into the next streaming segment instead.
This prevents cursor-only 'tofu' messages across all platforms without
affecting normal streaming (edits to existing messages, final sends
without cursor, and messages with substantial text are all unaffected).
Reported by @michalkomar on X.
Production fixes:
- Add clear_session_context() to hermes_logging.py (fixes 48 teardown errors)
- Add clear_session() to tools/approval.py (fixes 9 setup errors)
- Add SyncError M_UNKNOWN_TOKEN check to Matrix _sync_loop (bug fix)
- Fall back to inline api_key in named custom providers when key_env
is absent (runtime_provider.py)
Test fixes:
- test_memory_user_id: use builtin+external provider pair, fix honcho
peer_name override test to match production behavior
- test_display_config: remove TestHelpers for non-existent functions
- test_auxiliary_client: fix OAuth tokens to match _is_oauth_token
patterns, replace get_vision_auxiliary_client with resolve_vision_provider_client
- test_cli_interrupt_subagent: add missing _execution_thread_id attr
- test_compress_focus: add model/provider/api_key/base_url/api_mode
to mock compressor
- test_auth_provider_gate: add autouse fixture to clean Anthropic env
vars that leak from CI secrets
- test_opencode_go_in_model_list: accept both 'built-in' and 'hermes'
source (models.dev API unavailable in CI)
- test_email: verify email Platform enum membership instead of source
inspection (build_channel_directory now uses dynamic enum loop)
- test_feishu: add bot_added/bot_deleted handler mocks to _Builder
- test_ws_auth_retry: add AsyncMock for sync_store.get_next_batch,
add _pending_megolm and _joined_rooms to Matrix adapter mocks
- test_restart_drain: monkeypatch-delete INVOCATION_ID (systemd sets
this in CI, changing the restart call signature)
- test_session_hygiene: add user_id to SessionSource
- test_session_env: use relative baseline for contextvar clear check
(pytest-xdist workers share context)
Improvements from our earlier #8269 salvage work applied to #7616:
- Platform token lock: acquire_scoped_lock/release_scoped_lock prevents
two profiles from double-connecting the same QQ bot simultaneously
- Send retry with exponential backoff (3 attempts, 1s/2s/4s) with
permanent vs transient error classification (matches Telegram pattern)
- Proper long-message splitting via truncate_message() instead of
hard-truncating at MAX_MESSAGE_LENGTH (preserves code blocks, adds 1/N)
- REST-based one-shot send in send_message_tool — uses QQ Bot REST API
directly with httpx instead of creating a full WebSocket adapter per
message (fixes the connect→send race condition)
- Use shared strip_markdown() from helpers.py instead of 15 lines of
inline regex with import-inside-method (DRY, same as BlueBubbles/SMS)
- format_message() now wired into send() pipeline
- Add Platform.QQBOT to _UPDATE_ALLOWED_PLATFORMS (enables /update command)
- Add 'qqbot' to webhook cross-platform delivery routing
- Add 'qqbot' to hermes dump platform detection
- Fix test_name_property casing: 'QQBot' not 'QQBOT'
- Add _parse_qq_timestamp() for ISO 8601 + integer ms compatibility
(QQ API changed timestamp format — from PR #2411 finding)
- Wire timestamp parsing into all 4 message handlers
- Rename platform from 'qq' to 'qqbot' across all integration points
(Platform enum, toolset, config keys, import paths, file rename qq.py → qqbot.py)
- Add PLATFORM_HINTS for QQBot in prompt_builder (QQ supports markdown)
- Set SUPPORTS_MESSAGE_EDITING = False to skip streaming on QQ
(prevents duplicate messages from non-editable partial + final sends)
- Add _send_qqbot() standalone send function for cron/send_message tool
- Add interactive _setup_qq() wizard in hermes_cli/setup.py
- Restore missing _setup_signal/email/sms/dingtalk/feishu/wecom/wecom_callback
functions that were lost during the original merge
Models like MiniMax emit inline <think>...</think> reasoning blocks in
their content field. The CLI already suppresses these via a state machine
in _stream_delta, but the gateway's GatewayStreamConsumer had no
equivalent filtering — raw think blocks were streamed directly to
Discord/Telegram/Slack.
The fix adds a _filter_and_accumulate() method that mirrors the CLI's
approach: a state machine tracks whether we're inside a reasoning block
and silently discards the content. Includes the same block-boundary
check (tag must appear at line start or after whitespace-only prefix)
to avoid false positives when models mention <think> in prose.
Handles all tag variants: <think>, <thinking>, <THINKING>, <thought>,
<reasoning>, <REASONING_SCRATCHPAD>.
Also handles edge cases:
- Tags split across streaming deltas (partial tag buffering)
- Unclosed blocks (content suppressed until stream ends)
- Multiple consecutive blocks
- _flush_think_buffer on stream end for held-back partial tags
Adds 22 unit tests + 1 integration test covering all scenarios.
When the 5-second stream_task timeout in gateway/run.py expires (due to
slow Telegram API calls from rate limiting after several messages), the
stream consumer is cancelled via asyncio.CancelledError. The
CancelledError handler did a best-effort final edit but never set
final_response_sent, so the gateway fell through to the normal send path
and delivered the full response again as a reply — causing a duplicate.
The fix: in the CancelledError handler, set final_response_sent = True
when already_sent is True (i.e., the stream consumer had already
delivered content to the user). This tells the gateway's already_sent
check that the response was delivered, preventing the duplicate send.
Adds two tests verifying the cancellation behavior:
- Cancelled with already_sent=True → final_response_sent=True (no dup)
- Cancelled with already_sent=False → final_response_sent=False (normal
send path proceeds)
Reported by community user hume on Discord.
/stop was calling suspend_session() which marked the session for auto-reset
on the next message. This meant users lost their conversation history every
time they stopped a running agent — especially painful for untitled sessions
that can't be resumed by name.
Now /stop just interrupts the agent and cleans the session lock. The session
stays intact so users can continue the conversation.
The suspend behavior was introduced in #7536 to break stuck session resume
loops on gateway restart. That case is already handled by
suspend_recently_active() which runs at gateway startup, so removing it from
/stop doesn't regress the original fix.
The v11→v12 migration converts custom_providers (list) into providers
(dict), then deletes the list. But all runtime resolvers read from
custom_providers — after migration, named custom endpoints silently stop
resolving and fallback chains fail with AuthError.
Add get_compatible_custom_providers() that reads from both config schemas
(legacy custom_providers list + v12+ providers dict), normalizes entries,
deduplicates, and returns a unified list. Update ALL consumers:
- hermes_cli/runtime_provider.py: _get_named_custom_provider() + key_env
- hermes_cli/auth_commands.py: credential pool provider names
- hermes_cli/main.py: model picker + _model_flow_named_custom()
- agent/auxiliary_client.py: key_env + custom_entry model fallback
- agent/credential_pool.py: _iter_custom_providers()
- cli.py + gateway/run.py: /model switch custom_providers passthrough
- run_agent.py + gateway/run.py: per-model context_length lookup
Also: use config.pop() instead of del for safer migration, fix stale
_config_version assertions in tests, add pool mock to codex test.
Co-authored-by: 墨綠BG <s5460703@gmail.com>
Closes#8776, salvaged from PR #8814
Updated the acquire_scoped_lock function to treat empty or corrupt lock files as stale. This change ensures that if a lock file exists but is invalid, it will be removed to prevent issues with stale locks. Added tests to verify recovery from both empty and corrupt lock files.
- Store source metadata on /voice channel join so voice input shares the
same session as the linked text channel conversation
- Treat voice-linked text channels as free-response (skip @mention and
auto-thread) while voice is active
- Scope the voice-linked exemption to the exact bound channel, not
sibling threads
- Guard signal handler registration in start_gateway() for non-main
threads (prevents RuntimeError when gateway runs in a daemon thread)
- Clean up _voice_sources on leave_voice_channel
Salvaged from PR #3475 by twilwa (Modal runtime portions excluded).
When HTTPS_PROXY / HTTP_PROXY / ALL_PROXY env vars are set (or macOS system proxy
is detected), pass the proxy URL explicitly via HTTPXRequest(proxy=proxy_url) instead
of relying on httpx's trust_env mechanism, which is unreliable for HTTP CONNECT
proxies (e.g. Clash / ClashMac in fake-ip mode).
Uses the shared resolve_proxy_url() from base.py (handles env vars + macOS system
proxy detection) instead of duplicating env var reading inline. Consolidates the
proxy_configured boolean into a single proxy_url = resolve_proxy_url() call that
serves as both the gate for skipping fallback-IP transport and the value passed
to HTTPXRequest.
Co-authored-by: Hermes Agent <hermes@nousresearch.com>
Salvaged from PR #8931 by MaybeRichard.
The detached bash subprocess spawned by /restart gets killed by
systemd's KillMode=mixed cgroup cleanup, leaving the gateway dead.
Under systemd (detected via INVOCATION_ID env var), /restart now uses
via_service=True which exits with code 75 — RestartForceExitStatus=75
in the unit file makes systemd auto-restart the service. The detached
subprocess approach is preserved as fallback for non-systemd
environments (Docker, tmux, foreground mode).
When a user sends /restart, the gateway now persists their routing info
(platform, chat_id, thread_id) to .restart_notify.json. After the new
gateway process starts and adapters connect, it reads the file, sends a
'Gateway restarted successfully' message to that specific chat, and
cleans up the file.
This follows the same pattern as _send_update_notification (used by
/update). Thread IDs are preserved so the notification lands in the
correct Telegram topic or Discord thread.
Previously, after /restart the user had no feedback that the gateway was
back — they had to send a message to find out. Now they get a proactive
notification and know their session continues.
When tool_preview_length is 0 (default for platforms without a tier
default, like Session), verbose mode was truncating args JSON to 200
characters. Since the user explicitly opted into verbose mode, they
expect full tool call detail — the 200-char cap defeated the purpose.
Now: tool_preview_length=0 means no truncation in verbose mode.
Positive values still cap as before. Platform message-length limits
handle overflow naturally.
Three changes that address the poor WhatsApp experience reported by users:
1. Reclassify WhatsApp from TIER_LOW to TIER_MEDIUM in display_config.py
— enables streaming and tool progress via the existing Baileys /edit
bridge endpoint. Users now see progressive responses instead of
minutes of silence followed by a wall of text.
2. Lower MAX_MESSAGE_LENGTH from 65536 to 4096 and add proper chunking
— send() now calls format_message() and truncate_message() before
sending, then loops through chunks with a small delay between them.
The base class truncate_message() already handles code block boundary
detection (closes/reopens fences at chunk boundaries). reply_to is
only set on the first chunk.
3. Override format_message() with WhatsApp-specific markdown conversion
— converts **bold** to *bold*, ~~strike~~ to ~strike~, headers to
bold text, and [links](url) to text (url). Code blocks and inline
code are protected from conversion via placeholder substitution.
Together these fix the two user complaints:
- 'sends the whole code all the time' → now chunked at 4K with proper
formatting
- 'terminal gets interrupted and gets cooked' → streaming + tool progress
give visual feedback so users don't accidentally interrupt with
follow-up messages
Port from nearai/ironclaw#2304: Telegram's 4096 character limit is
measured in UTF-16 code units, not Unicode codepoints. Characters
outside the Basic Multilingual Plane (emoji like 😀, CJK Extension B,
musical symbols) are surrogate pairs: 1 Python char but 2 UTF-16 units.
Previously, truncate_message() used Python's len() which counts
codepoints. This could produce chunks exceeding Telegram's actual limit
when messages contain many astral-plane characters.
Changes:
- Add utf16_len() helper and _prefix_within_utf16_limit() for
UTF-16-aware string measurement and truncation
- Add _custom_unit_to_cp() binary-search helper that maps a custom-unit
budget to the largest safe codepoint slice position
- Update truncate_message() to accept optional len_fn parameter
- Telegram adapter now passes len_fn=utf16_len when splitting messages
- Fix fallback truncation in Telegram error handler to use
_prefix_within_utf16_limit instead of codepoint slicing
- Update send_message_tool.py to use utf16_len for Telegram platform
- Add comprehensive tests: utf16_len, _prefix_within_utf16_limit,
truncate_message with len_fn (emoji splitting, content preservation,
code block handling)
- Update mock lambdas in reply_mode tests to accept **kw for len_fn
Adds /debug as a slash command available in CLI, Telegram, Discord,
Slack, and all other gateway platforms. Uploads debug report + full
logs to paste services and returns shareable URLs.
- commands.py: CommandDef in Info category (no cli_only/gateway_only)
- gateway/run.py: async handler with run_in_executor for blocking I/O
- cli.py: dispatch in process_command to run_debug_share
Port from openclaw/openclaw#64586: users who copy .env.example without
changing placeholder values now get a clear error at startup instead of
a confusing auth failure from the platform API. Also rejects placeholder
API_SERVER_KEY when binding to a network-accessible address.
Cherry-picked from PR #8677.
Port from openclaw/openclaw#64796: Per MSC3952 / Matrix v1.7, the
m.mentions.user_ids field is the authoritative mention signal. Clients
that populate m.mentions but don't duplicate @bot in the body text
were being silently dropped when MATRIX_REQUIRE_MENTION=true.
Cherry-picked from PR #8673.
Some OpenAI-compatible clients (Open WebUI, LobeChat, etc.) send
message content as an array of typed parts instead of a plain string:
[{"type": "text", "text": "hello"}]
The agent pipeline expects strings, so these array payloads caused
silent failures or empty messages.
Add _normalize_chat_content() with defensive limits (recursion depth,
list size, output length) and apply it to both the Chat Completions
and Responses API endpoints. The Responses path had inline
normalization that only handled input_text/output_text — the shared
function also handles the standard 'text' type.
Salvaged from PR #7980 (ikelvingo) — only the content normalization;
the SSE and Weixin changes in that PR were regressions and are not
included.
Co-authored-by: ikelvingo <ikelvingo@users.noreply.github.com>
aiohttp.ClientSession defaults to trust_env=False, ignoring HTTP_PROXY/
HTTPS_PROXY env vars. This causes QR login and all API calls to fail for
users behind a proxy (e.g. Clash in fake-ip mode), which is common in
China where Weixin and WeCom are primarily used.
Added trust_env=True to all aiohttp.ClientSession instantiations that
connect to external hosts (weixin: 3 places, wecom: 1, matrix: 1).
WhatsApp sessions are excluded as they only connect to localhost.
httpx-based adapters (dingtalk, signal, wecom_callback) are unaffected
as httpx defaults to trust_env=True.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Four fixes for the Weixin/WeChat adapter, synthesized from the best
aspects of community PRs #8407, #8521, #8360, #7695, #8308, #8525,
#7531, #8144, #8251.
1. Streaming cursor (▉) stuck permanently — WeChat doesn't support
message editing, so the cursor appended during streaming can never
be removed. Add SUPPORTS_MESSAGE_EDITING = False to WeixinAdapter
and check it in gateway/run.py to use an empty cursor for non-edit
platforms. (Fixes#8307, #8326)
2. Media upload failures — two bugs in _send_file():
a) upload_full_url path used PUT (404 on WeChat CDN); now uses POST.
b) aes_key was base64(raw_bytes) but the iLink API expects
base64(hex_string); images showed as grey boxes. (Fixes#8352, #7529)
Also: unified both upload paths into _upload_ciphertext(), preferring
upload_full_url. Added send_video/send_voice methods and voice_item
media builder for audio/.silk files. Added video_md5 field.
3. Markdown links stripped — WeChat can't render [text](url), so
format_message() now converts them to 'text (url)' plaintext.
Code blocks are preserved. (Fixes#7617)
4. Blank message prevention — three guards:
a) _split_text_for_weixin_delivery('') returns [] not ['']
b) send() filters empty/whitespace chunks before _send_text_chunk
c) _send_message() raises ValueError for empty text as safety net
Community credit: joei4cm (#8407), lyonDan (#8521), SKFDJKLDG (#8360),
tomqiaozc (#7695), joshleeeeee (#8308), luoxiao6645(#8525),
longsizhuo (#7531), Astral-Yang (#8144), QingWei-Li (#8251).
Add agent.gateway_notify_interval config option (default 600s).
Set to 0 to disable periodic 'still working' notifications.
Bridged to HERMES_AGENT_NOTIFY_INTERVAL env var (same pattern as
gateway_timeout and gateway_timeout_warning).
The inactivity warning (gateway_timeout_warning) was already
configurable; this makes the wall-clock ping configurable too.
The _watch_update_progress() poll loop never deleted .update_prompt.json
after forwarding the prompt to the user, causing the same prompt to be
re-sent every poll cycle (2s). Two fixes:
1. Delete .update_prompt.json after forwarding — the update process only
polls for .update_response, it doesn't need the prompt file to persist.
2. Guard re-sends with _update_prompt_pending check — belt-and-suspenders
to prevent duplicates even under race conditions.
Add regression test asserting the prompt is sent exactly once.
When a user configures a provider (e.g. `hermes auth add openai-codex`)
but never selects a model via `hermes model`, the gateway and CLI would
pass an empty model string to the API, causing:
'Codex Responses request model must be a non-empty string'
Now both gateway (_resolve_session_agent_runtime) and CLI
(_ensure_runtime_credentials) detect an empty model and fill it from
the provider's first catalog entry in _PROVIDER_MODELS. This covers
all providers that have a static model list (openai-codex, anthropic,
gemini, copilot, etc.).
The fix is conservative: it only triggers when model is truly empty
and a known provider was resolved. Explicit model choices are never
overridden.
When the gateway shuts down gracefully (hermes update, gateway restart,
/restart), it now writes a .clean_shutdown marker file. On the next
startup, if this marker exists, suspend_recently_active() is skipped
and the marker is cleaned up.
Previously, suspend_recently_active() fired on EVERY startup —
including planned restarts from hermes update or hermes gateway restart.
This caused users to lose their conversation history unexpectedly: the
session would be marked as suspended, and the next message would
trigger an auto-reset with a notification the user never asked for.
The original purpose of suspend_recently_active() is crash recovery —
preventing stuck sessions that were mid-processing when the gateway
died unexpectedly. Graceful shutdowns already drain active agents via
_drain_active_agents(), so there is no stuck-session risk. After a
crash (no marker written), suspension still fires as before.
Fixes the scenario where a user asks the agent to run hermes update,
the gateway restarts, and the user's next message gets an unwanted
'Session automatically reset' notification with their history cleared.
Follow-up for cherry-picked PR #8272:
- Add MATRIX_RECOVERY_KEY to module docstring header in matrix.py
- Register in OPTIONAL_ENV_VARS (config.py) with password=True, advanced=True
- Add to _NON_SETUP_ENV_VARS set
- Document cross-signing verification in matrix.md E2EE section
- Update migration guide with recovery key step (step 3)
- Add to environment-variables.md reference
After the PgCryptoStore migration in v0.8.0, the verify_with_recovery_key
call that previously ran after share_keys() was dropped. On any rotation
that uploads fresh device keys (fresh crypto.db, server had stale keys
from a prior install, etc.), the new device keys carry no valid self-
signing signature because the bot has no access to the self-signing
private key.
Peers like Element then refuse to share Megolm sessions with the
rotated device, so the bot silently stops decrypting incoming messages.
This restores the recovery-key bootstrap: on startup, if
MATRIX_RECOVERY_KEY is set, import the cross-signing private keys from
SSSS and sign_own_device(), producing a valid signature server-side.
Idempotent and gated on MATRIX_RECOVERY_KEY — no behavior change for
users who don't configure a recovery key.
Verified end-to-end by deleting crypto.db and restarting: the bot
rotates device identity keys, re-uploads, self-signs via recovery key,
and decrypts+replies to fresh messages from a paired Element client.
After /model switches the model (both picker and text paths), the cached
agent's config signature becomes stale — the agent was updated in-place
via switch_model() but the cache tuple's signature was never refreshed.
The next turn *should* detect the signature mismatch and create a fresh
agent, but this relies on the new model's signature differing from the
old one in _agent_config_signature().
Evicting the cached agent explicitly after storing the session override
is more defensive — the next turn is guaranteed to create a fresh agent
from the override without depending on signature mismatch detection.
Also adds debug logging at three key decision points so we can trace
exactly what happens when /model + /retry interact:
- _resolve_session_agent_runtime: which override path is taken (fast
with api_key vs fallback), or why no override was found
- _run_agent.run_sync: final resolved model/provider before agent
creation
Reported: /model switch to xiaomi/mimo-v2-pro followed by /retry still
used the old model (glm-5.1).
The monitor_for_interrupt() and backup interrupt checks were calling
get_pending_message() which pops the message from the adapter's queue.
This created a race condition: if the agent finished naturally before
checking _interrupt_requested, the pending message was permanently lost.
Timeline of the race:
1. Agent near completion, user sends message
2. Level 1 guard stores message in adapter._pending_messages, sets event
3. monitor_for_interrupt() detects event, POPS message, calls agent.interrupt()
4. Agent's run_conversation() was already returning (interrupted=False)
5. Post-run dequeue finds nothing (monitor already consumed it)
6. result.get('interrupted') is False so interrupt_message fallback doesn't fire
7. User message permanently lost — agent finishes without processing it
Fix: change all three interrupt detection sites (primary monitor + two
backup checks) from get_pending_message() (pop) to
_pending_messages.get() (peek). The message stays in the adapter's queue
until _dequeue_pending_event() consumes it in the post-run handler,
which runs regardless of whether the agent was interrupted or finished
naturally.
Reported by @_SushantSays — intermittent message loss during long
terminal command execution, persisting after the previous fix (73f970fa)
which addressed monitor task death but not this consumption race.
Add content-aware splitting to compact mode: short chat-like exchanges
(2-6 short lines without headings/lists/quotes) get separate message
bubbles for a natural chat feel, while structured content (tables,
headings with body, numbered lists) stays in a single message.
Cherry-picked from PR #7587 by bravohenry, adapted to the compact/legacy
split_per_line architecture from #7903.
When the agent calls process(action='wait') or process(action='poll')
and gets the exited status, the completion_queue notification is
redundant — the agent already has the output from the tool return.
Previously, the drain loops in CLI and gateway would still inject
the [SYSTEM: Background process completed] message, causing the
agent to receive the same information twice.
Fix: track session IDs in _completion_consumed set when wait/poll/log
returns an exited process. Drain loops in cli.py and gateway watcher
skip completion events for consumed sessions. Watch pattern events
are never suppressed (they have independent semantics).
Adds 4 tests covering wait/poll/log marking and running-process
negative case.
Add a 'tip of the day' feature that displays a random one-liner about
Hermes Agent features on every new session — CLI startup, /clear, /new,
and gateway /new across all messaging platforms.
- New hermes_cli/tips.py module with 210 curated tips covering slash
commands, keybindings, CLI flags, config options, tools, gateway
platforms, profiles, sessions, memory, skills, cron, voice, security,
and more
- CLI: tips display in skin-aware dim gold color after the welcome line
- Gateway: tips append to the /new and /reset response on all platforms
- Fully wrapped in try/except — tips are non-critical and never break
startup or reset
Display format (CLI):
✦ Tip: /btw <question> asks a quick side question without tools or history.
Display format (gateway):
✨ Session reset! Starting fresh.
✦ Tip: hermes -c resumes your most recent CLI session.
The interrupt mechanism for regular text messages (non-commands) during
active agent runs relied on a single async polling task
(monitor_for_interrupt) with no error handling. If this task died
silently due to an unhandled exception, stale adapter reference after
reconnect, or any other failure, user messages sent during agent
execution would be queued but never trigger an actual interrupt — the
agent would continue running until it finished naturally, then process
the queued message.
Three improvements:
1. Error handling in monitor_for_interrupt(): wrap the polling body in
try/except so transient errors are logged and retried instead of
silently killing the task.
2. Fresh adapter reference on each poll iteration: re-resolve
self.adapters.get(source.platform) every 200ms instead of capturing
the adapter once at task creation time. This prevents stale
references after adapter reconnects.
3. Backup interrupt check in the inactivity poll loop: both the
unlimited and timeout-enabled paths now check for pending interrupts
every 5 seconds (the existing poll interval). Uses a shared
_interrupt_detected asyncio.Event to avoid double-firing when the
primary monitor already handled the interrupt. Logs at INFO level
with monitor task state for debugging.
On servers with broken or unreachable IPv6, Python's socket.getaddrinfo
returns AAAA records first. urllib/httpx/requests all try IPv6 connections
first and hang for the full TCP timeout before falling back to IPv4. This
affects web_extract, web_search, the OpenAI SDK, and all HTTP tools.
Adds network.force_ipv4 config option (default: false) that monkey-patches
socket.getaddrinfo to resolve as AF_INET when the caller didn't specify a
family. Falls back to full resolution if no A record exists, so pure-IPv6
hosts still work.
Applied early at all three entry points (CLI, gateway, cron scheduler)
before any HTTP clients are created.
Reported by user @29n — Chinese Ubuntu server with unreachable IPv6 causing
timeouts on lobste.rs and other IPv6-enabled sites while Google/GitHub
worked fine (IPv4-only resolution).
The gateway startup path references RedactingFormatter without
importing it, causing a NameError crash when launched with a
verbosity flag (e.g. via launchd --replace).
Fixes#8044
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds an optional focus topic to /compress: `/compress database schema`
guides the summariser to preserve information related to the focus topic
(60-70% of summary budget) while compressing everything else more aggressively.
Inspired by Claude Code's /compact <focus>.
Changes:
- context_compressor.py: focus_topic parameter on _generate_summary() and
compress(); appends FOCUS TOPIC guidance block to the LLM prompt
- run_agent.py: focus_topic parameter on _compress_context(), passed through
to the compressor
- cli.py: _manual_compress() extracts focus topic from command string,
preserves existing manual_compression_feedback integration (no regression)
- gateway/run.py: _handle_compress_command() extracts focus from event args
and passes through — full gateway parity
- commands.py: args_hint="[focus topic]" on /compress CommandDef
Salvaged from PR #7459 (CLI /compress focus only — /context command deferred).
15 new tests across CLI, compressor, and gateway.
Fixes#7952 — Matrix E2EE completely broken after mautrix migration.
- Replace MemoryCryptoStore + pickle/HMAC persistence with mautrix's
PgCryptoStore backed by SQLite via aiosqlite. Crypto state now
persists reliably across restarts without fragile serialization.
- Add handle_sync() call on initial sync response so to-device events
(queued Megolm key shares) are dispatched to OlmMachine instead of
being silently dropped.
- Add _verify_device_keys_on_server() after loading crypto state.
Detects missing keys (re-uploads), stale keys from migration
(attempts re-upload), and corrupted state (refuses E2EE).
- Add _CryptoStateStore adapter wrapping MemoryStateStore to satisfy
mautrix crypto's StateStore interface (is_encrypted,
get_encryption_info, find_shared_rooms).
- Remove redundant share_keys() call from sync loop — OlmMachine
already handles this via DEVICE_OTK_COUNT event handler.
- Fix datetime vs float TypeError in session.py suspend_recently_active()
that crashed gateway startup.
- Add aiosqlite and asyncpg to [matrix] extra in pyproject.toml.
- Update test mocks for PgCryptoStore/Database and add query_keys mock
for key verification. 174 tests pass.
- Add E2EE upgrade/migration docs to Matrix user guide.
* feat: component-separated logging with session context and filtering
Phase 1 — Gateway log isolation:
- gateway.log now only receives records from gateway.* loggers
(platform adapters, session management, slash commands, delivery)
- agent.log remains the catch-all (all components)
- errors.log remains WARNING+ catch-all
- Moved gateway.log handler creation from gateway/run.py into
hermes_logging.setup_logging(mode='gateway') with _ComponentFilter
Phase 2 — Session ID injection:
- Added set_session_context(session_id) / clear_session_context() API
using threading.local() for per-thread session tracking
- _SessionFilter enriches every log record with session_tag attribute
- Log format: '2026-04-11 10:23:45 INFO [session_id] logger.name: msg'
- Session context set at start of run_conversation() in run_agent.py
- Thread-isolated: gateway conversations on different threads don't leak
Phase 3 — Component filtering in hermes logs:
- Added --component flag: hermes logs --component gateway|agent|tools|cli|cron
- COMPONENT_PREFIXES maps component names to logger name prefixes
- Works with all existing filters (--level, --session, --since, -f)
- Logger name extraction handles both old and new log formats
Files changed:
- hermes_logging.py: _SessionFilter, _ComponentFilter, COMPONENT_PREFIXES,
set/clear_session_context(), gateway.log creation in setup_logging()
- gateway/run.py: removed redundant gateway.log handler (now in hermes_logging)
- run_agent.py: set_session_context() at start of run_conversation()
- hermes_cli/logs.py: --component filter, logger name extraction
- hermes_cli/main.py: --component argument on logs subparser
Addresses community request for component-separated, filterable logging.
Zero changes to existing logger names — __name__ already provides hierarchy.
* fix: use LogRecord factory instead of per-handler _SessionFilter
The _SessionFilter approach required attaching a filter to every handler
we create. Any handler created outside our _add_rotating_handler (like
the gateway stderr handler, or third-party handlers) would crash with
KeyError: 'session_tag' if it used our format string.
Replace with logging.setLogRecordFactory() which injects session_tag
into every LogRecord at creation time — process-global, zero per-handler
wiring needed. The factory is installed at import time (before
setup_logging) so session_tag is available from the moment hermes_logging
is imported.
- Idempotent: marker attribute prevents double-wrapping on module reload
- Chains with existing factory: won't break third-party record factories
- Removes _SessionFilter from _add_rotating_handler and setup_verbose_logging
- Adds tests: record factory injection, idempotency, arbitrary handler compat
Add display.platforms section to config.yaml for per-platform overrides of
display settings (tool_progress, show_reasoning, streaming, tool_preview_length).
Each platform gets sensible built-in defaults based on capability tier:
- High (telegram, discord): tool_progress=all, streaming follows global
- Medium (slack, mattermost, matrix, feishu): tool_progress=new
- Low (signal, whatsapp, bluebubbles, wecom, etc.): tool_progress=off, streaming=false
- Minimal (email, sms, webhook, homeassistant): tool_progress=off, streaming=false
Example config:
display:
platforms:
telegram:
tool_progress: all
show_reasoning: true
slack:
tool_progress: off
Resolution order: platform override > global setting > built-in platform default.
Changes:
- New gateway/display_config.py: resolver module with tier-based platform defaults
- gateway/run.py: tool_progress, tool_preview_length, streaming, show_reasoning
all resolve per-platform via the new resolver
- /verbose command: now cycles tool_progress per-platform (saves to
display.platforms.<platform>.tool_progress instead of global)
- /reasoning show|hide: now saves show_reasoning per-platform
- Config version 15 -> 16: migrates tool_progress_overrides into display.platforms
- Backward compat: legacy tool_progress_overrides still read as fallback
- 27 new tests for resolver, normalization, migration, backward compat
- Updated verbose command tests for per-platform behavior
Addresses community request for per-channel verbosity control (Guillaume Meyer,
Nathan Danielsen) — high verbosity on backchannel Telegram, low on customer-facing
Slack, none on email.
Add display.interim_assistant_messages config (enabled by default) that
forwards completed assistant commentary between tool calls to the user
as separate chat messages. Models already emit useful status text like
'I'll inspect the repo first.' — this surfaces it on Telegram, Discord,
and other messaging platforms instead of swallowing it.
Independent from tool_progress and gateway streaming. Disabled for
webhooks. Uses GatewayStreamConsumer when available, falls back to
direct adapter send. Tracks response_previewed to prevent double-delivery
when interim message matches the final response.
Also fixes: cursor not stripped from fallback prefix in stream consumer
(affected continuation calculation on no-edit platforms like Signal).
Cherry-picked from PR #7885 by asheriif, default changed to enabled.
Fixes#5016
Complete the contextvars migration by adding HERMES_SESSION_KEY to the
unified _VAR_MAP in session_context.py. Without this, concurrent gateway
handlers race on os.environ["HERMES_SESSION_KEY"].
- Add _SESSION_KEY ContextVar to _VAR_MAP, set_session_vars(), clear_session_vars()
- Wire session_key through _set_session_env() from SessionContext
- Replace os.getenv fallback in tools/approval.py with get_session_env()
(function-level import to avoid cross-layer coupling)
- Keep os.environ set as CLI/cron fallback
Cherry-picked from PR #7878 by 0xbyt4.
Add a second WeCom integration mode for regular enterprise self-built
applications. Unlike the existing bot/websocket adapter (wecom.py),
this handles WeCom's standard callback flow: WeCom POSTs encrypted XML
to an HTTP endpoint, the adapter decrypts, queues for the agent, and
immediately acknowledges. The agent's reply is delivered proactively
via the message/send API.
Key design choice: always acknowledge immediately and use proactive
send — agent sessions take 3-30 minutes, so the 5-second inline reply
window is never useful. The original PR's Future/pending-reply
machinery was removed in favour of this simpler architecture.
Features:
- AES-CBC encrypt/decrypt (BizMsgCrypt-compatible)
- Multi-app routing scoped by corp_id:user_id
- Legacy bare user_id fallback for backward compat
- Access-token management with auto-refresh
- WECOM_CALLBACK_* env var overrides
- Port-in-use pre-check before binding
- Health endpoint at /health
Salvaged from PR #7774 by @chqchshj. Simplified by removing the
inline reply Future system and fixing: secrets.choice for nonce
generation, immediate plain-text acknowledgment (not encrypted XML
containing 'success'), and initial token refresh error handling.
When 'hermes claw migrate' copies Telegram/Discord/Slack bot tokens from
OpenClaw while the Hermes gateway is already polling with those same tokens,
the platforms conflict (e.g. Telegram 409). Add a pre-flight check that reads
gateway_state.json via get_running_pid() + read_runtime_status(), warns the
user, and lets them cancel or continue.
Also improve the Telegram polling conflict error message to mention OpenClaw
as a common cause and give the 'hermes start' restart command.
Refs #7907
In _run_agent(), the pending message handler references 'event' which
is not defined in that scope — it only exists in the caller. This
causes a NameError when sending the first response before processing a
queued follow-up message.
Replace getattr(event, 'metadata', None) with the established pattern
using source.thread_id, consistent with lines 2625, 2810, 3678, 4410, 4566
in the same file.
When sending multi-chunk responses, individual chunks can fail due to
transient iLink API errors. Previously a single failure would abort the
entire message. Now each chunk is retried with linear backoff before
giving up, and the same client_id is reused across retries for
server-side deduplication.
Configurable via config.yaml (platforms.weixin.extra) or env vars:
- send_chunk_delay_seconds (default 0.35s) — pacing between chunks
- send_chunk_retries (default 2) — max retry attempts per chunk
- send_chunk_retry_delay_seconds (default 1.0s) — base retry delay
Replaces the hardcoded 0.3s inter-chunk delay from #7903.
Salvaged from PR #7899 by @corazzione. Fixes#7836.
WeCom AI Bot sends file attachments with msgtype="appmsg", not
msgtype="file". Previously only file content was discarded while
the text title reached the agent.
Changes:
- _extract_text(): Extract appmsg title (filename) for display
- _extract_media(): Handle appmsg type with file/image content
Fixes#7750
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cherry-picked from PR #7747 with follow-up fixes:
- Narrowed suspend_all_active() to suspend_recently_active() — only
suspends sessions updated within the last 2 minutes (likely in-flight),
not all sessions which would unnecessarily reset idle users
- /stop with no running agent no longer suspends the session; only
actual force-stops mark the session for reset
Background process watchers (notify_on_complete, check_interval) created
synthetic SessionSource objects without user_id/user_name. While the
internal=True bypass (1d8d4f28) prevented false pairing for agent-
generated notifications, the missing identity caused:
- Garbage entries in pairing rate limiters (discord:None, telegram:None)
- 'User None' in approval messages and logs
- No user identity available for future code paths that need it
Additionally, platform messages arriving without from_user (Telegram
service messages, channel forwards, anonymous admin actions) could still
trigger false pairing because they are not internal events.
Fix:
1. Propagate user_id/user_name through the full watcher chain:
session_context.py → gateway/run.py → terminal_tool.py →
process_registry.py (including checkpoint persistence/recovery)
2. Add None user_id guard in _handle_message() — silently drop
non-internal messages with no user identity instead of triggering
the pairing flow.
Salvaged from PRs #7664 (kagura-agent, ContextVar approach),
#6540 (MestreY0d4-Uninter, tests), and #7709 (guang384, None guard).
Closes#6341, #6485, #7643
Relates to #6516, #7392
The Weixin adapter was splitting responses at every top-level newline,
causing notification spam (up to 70 API calls for a single long markdown
response). This salvages the best aspects of six contributor PRs:
Compact mode (new default):
- Messages under the 4000-char limit stay as a single bubble even with
multiple lines, paragraphs, and code blocks
- Only oversized messages get split at logical markdown boundaries
- Inter-chunk delay (0.3s) between chunks prevents WeChat rate-limit drops
Legacy mode (opt-in):
- Set split_multiline_messages: true in platforms.weixin.extra config
- Or set WEIXIN_SPLIT_MULTILINE_MESSAGES=true env var
- Restores the old per-line splitting behavior
Salvaged from PRs #7797 (guantoubaozi), #7792 (luoxiao6645),
#7838 (qyx596), #7825 (weedge), #7784 (sherunlock03), #7773 (JnyRoad).
Core fix unanimous across all six; config toggle from #7838; inter-chunk
delay from #7825.
Matrix gateway: fix sync loop never dispatching events (#5819)
- _sync_loop() called client.sync() but never called handle_sync()
to dispatch events to registered callbacks — _on_room_message was
registered but never fired for new messages
- Store next_batch token from initial sync and pass as since= to
subsequent incremental syncs (was doing full initial sync every time)
- 17 comments, confirmed by multiple users on matrix.org
Feishu docs: add interactive card configuration for approvals (#6893)
- Error 200340 is a Feishu Developer Console configuration issue,
not a code bug — users need to enable Interactive Card capability
and configure Card Request URL
- Added required 3-step setup instructions to feishu.md
- Added troubleshooting entry for error 200340
- 17 comments from Feishu users
Copilot provider drift: detect GPT-5.x Responses API requirement (#3388)
- GPT-5.x models are rejected on /v1/chat/completions by both OpenAI
and OpenRouter (unsupported_api_for_model error)
- Added _model_requires_responses_api() to detect models needing
Responses API regardless of provider
- Applied in __init__ (covers OpenRouter primary users) and in
_try_activate_fallback() (covers Copilot->OpenRouter drift)
- Fixed stale comment claiming gateway creates fresh agents per message
(it caches them via _agent_cache since the caching was added)
- 7 comments, reported on Copilot+Telegram gateway
* fix(matrix): pass required args to MemoryCryptoStore for mautrix ≥0.21
MemoryCryptoStore.__init__() now requires account_id and pickle_key
positional arguments as of mautrix 0.21. The migration from matrix-nio
(commit 1850747) didn't account for this, causing E2EE initialization
to fail with:
MemoryCryptoStore.__init__() missing 2 required positional arguments:
'account_id' and 'pickle_key'
Pass self._user_id as account_id and derive pickle_key from the same
user_id:device_id pair already used for the on-disk HMAC signature.
Update the test stub to accept the new parameters.
Fixes#7803
* fix: use consistent fallback for pickle_key derivation
Address review: _pickle_key now uses _acct_id (which has the 'hermes'
fallback) instead of raw self._user_id, so both values stay consistent
when user_id is empty.
---------
Co-authored-by: Hermes Agent <hermes@nousresearch.com>
Telegram flood control during streaming caused messages to be cut off
mid-response. The old behavior permanently disabled edits after a single
flood-control failure, losing the remainder of the response.
Changes:
- Adaptive backoff: on flood-control edit failures, double the edit interval
instead of immediately disabling edits. Only permanently disable after 3
consecutive failures (_MAX_FLOOD_STRIKES).
- Cursor strip: when entering fallback mode, best-effort edit to remove the
cursor (▉) from the last visible message so it doesn't appear stuck.
- Fallback send retry: _send_fallback_final retries each chunk once on
flood-control failures (3s delay) before giving up.
- Default edit_interval increased from 0.3s to 1.0s. Telegram rate-limits
edits at ~1/s per message; 0.3s was virtually guaranteed to trigger flood
control on any non-trivial response.
- _send_or_edit returns bool so the overflow split loop knows not to
truncate accumulated text when an edit fails (prevents content loss).
Fixes: messages cutting/stopping mid-response on Telegram, especially
with streaming enabled.
* feat: add watch_patterns to background processes for output monitoring
Adds a new 'watch_patterns' parameter to terminal(background=true) that
lets the agent specify strings to watch for in process output. When a
matching line appears, a notification is queued and injected as a
synthetic message — triggering a new agent turn, similar to
notify_on_complete but mid-process.
Implementation:
- ProcessSession gets watch_patterns field + rate-limit state
- _check_watch_patterns() in ProcessRegistry scans new output chunks
from all three reader threads (local, PTY, env-poller)
- Rate limited: max 8 notifications per 10s window
- Sustained overload (45s) permanently disables watching for that process
- watch_queue alongside completion_queue, same consumption pattern
- CLI drains watch_queue in both idle loop and post-turn drain
- Gateway drains after agent runs via _inject_watch_notification()
- Checkpoint persistence + crash recovery includes watch_patterns
- Blocked in execute_code sandbox (like other bg params)
- 20 new tests covering matching, rate limiting, overload kill,
checkpoint persistence, schema, and handler passthrough
Usage:
terminal(
command='npm run dev',
background=true,
watch_patterns=['ERROR', 'WARN', 'listening on port']
)
* refactor: merge watch_queue into completion_queue
Unified queue with 'type' field distinguishing 'completion',
'watch_match', and 'watch_disabled' events. Extracted
_format_process_notification() in CLI and gateway to handle
all event types in a single drain loop. Removes duplication
across both CLI drain sites and the gateway.
When the stream consumer has sent at least one message (already_sent=True),
the gateway skips sending the final response to avoid duplicates. But this
also suppressed error messages when the agent failed mid-loop — rate limit
exhaustion, context overflow, compression failure, etc.
The user would see the last streamed content and then nothing: no error
message, no explanation. The agent appeared to 'stop responding.'
Fix: check the 'failed' flag at both the producer (_run_agent marks
already_sent) and consumer (_handle_message_with_agent checks it) sites.
Error messages are always delivered regardless of streaming state.
- Add agent.close() call to _finalize_shutdown_agents() to prevent
zombie processes (terminal sandboxes, browser daemons, httpx clients)
- Global cleanup (process_registry, environments, browsers) preserved
in _stop_impl() during conflict resolution
- Move /restart CommandDef from 'Info' to 'Session' category to match
/stop and /status
* fix: circuit breaker stops CPU-burning restart loops on persistent errors
When a gateway session hits a non-retryable error (e.g. invalid model
ID → HTTP 400), the agent fails and returns. But if the session keeps
receiving messages (or something periodically recreates agents), each
attempt spawns a new AIAgent — reinitializing MCP server connections,
burning CPU — only to hit the same 400 error again. On a 4-core server,
this pegs an entire core per stuck session and accumulates 300+ minutes
of CPU time over hours.
Fix: add a per-session consecutive failure counter in the gateway runner.
- Track consecutive non-retryable failures per session key
- After 3 consecutive failures (_MAX_CONSECUTIVE_FAILURES), block
further agent creation for that session and notify the user:
'⚠️ This session has failed N times in a row with a non-retryable
error. Use /reset to start a new session.'
- Evict the cached agent when the circuit breaker engages to prevent
stale state from accumulating
- Reset the counter on successful agent runs
- Clear the counter on /reset and /new so users can recover
- Uses getattr() pattern so bare GatewayRunner instances (common in
tests using object.__new__) don't crash
Tests:
- 8 new tests in test_circuit_breaker.py covering counter behavior,
threshold, reset, session isolation, and bare-runner safety
Addresses #7130.
* Revert "fix: circuit breaker stops CPU-burning restart loops on persistent errors"
This reverts commit d848ea7109d62a2fc4ba6da36fc4f0366b5ded94.
* fix: don't evict cached agent on failed runs — prevents MCP restart loop
When a run fails (e.g. invalid model ID → 400) and fallback activated,
the gateway was evicting the cached agent to 'retry primary next time.'
But evicting a failed agent forces a full AIAgent recreation on the next
message — reinitializing MCP server connections, spawning stdio
processes — only to hit the same 400 again. This created a CPU-burning
loop (91%+ for hours, #7130).
The fix: add `and not _run_failed` to the fallback-eviction check.
Failed runs keep the cached agent. The next message reuses it (no MCP
reinit), hits the same error, returns it to the user quickly. The user
can /reset or /model to fix their config.
Successful fallback runs still evict as before so the next message
retries the primary model.
Addresses #7130.
Follow-up fixes for the matrix-nio → mautrix migration:
1. Module-level mautrix.types import now wrapped in try/except with
proper stub classes. Without this, importing gateway.platforms.matrix
crashes the entire gateway when mautrix isn't installed — even for
users who don't use Matrix. The stubs mirror mautrix's real attribute
names so tests that exercise adapter methods (send, reactions, etc.)
work without the real SDK.
2. Removed _ensure_mautrix_mock() from test_matrix_mention.py — it
permanently installed MagicMock modules in sys.modules via setdefault(),
polluting later tests in the suite. No longer needed since the module
imports cleanly without mautrix.
3. Fixed thread persistence tests to use direct class reference in
monkeypatch.setattr() instead of string-based paths, which broke
when the module was reimported by other tests.
4. Moved the module-importability test to a subprocess to prevent it
from polluting sys.modules (reimporting creates a second module object
with different __dict__, breaking patch.object in subsequent tests).
The old nio code only handled RoomMessageText (m.text). The mautrix
rewrite dispatched both m.text and m.notice, which would cause infinite
loops between bots since m.notice is the conventional msgtype for bot
responses in the Matrix ecosystem.
- Add api.session.close() on E2EE dep check and E2EE setup failure
paths (two missing cleanup points from the mautrix migration)
- Replace raw pickle.load/dump with HMAC-SHA256 signed payloads to
prevent arbitrary code execution from a tampered store file
- Extract _resolve_message_context() to deduplicate ~40 lines of
mention/thread/DM gating logic between text and media handlers
- Move mautrix.types imports to module level (16 scattered local
imports consolidated)
- Parse mention/thread env vars once in __init__ instead of per-message
- Cache _is_bot_mentioned() result instead of calling 3x per event
- Consolidate send_emote/send_notice into shared _send_simple_message()
- Use _is_dm_room() in get_chat_info() instead of inline duplication
- Add _CRYPTO_PICKLE_PATH constant (was duplicated in 2 locations)
- Fix fragile event_ts extraction (double getattr, None safety)
- Clean up leaked aiohttp session on auth failure paths
- Remove redundant trailing _track_thread() calls
Address two bugs found by code review:
1. MemoryCryptoStore loses all E2EE keys on restart — now pickle the
store to disk on disconnect and restore on connect, preserving
Megolm sessions across restarts.
2. Encrypted events buffered for retry were silently dropped after
decryption because _on_encrypted_event registered the event ID
in the dedup set, then _on_room_message rejected it as a
duplicate. Now clear the dedup entry before routing decrypted
events.
Translate all nio SDK calls to mautrix equivalents while preserving the
adapter structure, business logic, and all features (E2EE, reactions,
threading, mention gating, text batching, media caching, voice MSC3245).
Key changes:
- nio.AsyncClient -> mautrix.client.Client + HTTPAPI + MemoryStateStore
- Manual E2EE key management -> OlmMachine with auto key lifecycle
- isinstance(resp, nio.XxxResponse) -> mautrix returns values directly
- add_event_callback per type -> single ROOM_MESSAGE handler with
msgtype dispatch
- Room state (member_count, display_name) via async state store lookups
- Upload/download return ContentURI/bytes directly (no wrapper objects)
Tool progress markers (e.g. `⏰ list`) were injected directly into
SSE delta.content chunks. OpenAI-compatible frontends (Open WebUI,
LobeChat, etc.) store delta.content verbatim as the assistant message
and send it back on subsequent requests. After enough turns, the model
learns to emit these markers as plain text instead of issuing real tool
calls — silently hallucinating tool results without ever running them.
Fix: Send tool progress as a custom `event: hermes.tool.progress` SSE
event instead of mixing it into delta.content. Per the SSE spec, clients
that don't understand a custom event type silently ignore it, so this is
backward-compatible. Frontends that want to render progress indicators
can listen for the custom event without persisting it to conversation
history.
The /v1/runs endpoint already uses structured events — this aligns the
/v1/chat/completions streaming path with the same principle.
Closes#6972
Six platforms (matrix, mattermost, dingtalk, feishu, wecom, homeassistant)
were missing from the session-based discovery loop, causing /channels and
send_message to return empty results on those platforms.
Instead of adding them to the hardcoded tuple (which would break again when
new platforms are added), derive the list dynamically from the Platform enum.
Only infrastructure entries (local, api_server, webhook) are excluded;
Discord and Slack are skipped automatically because their direct builders
already populate the platforms dict.
Reported by sprmn24 in PR #7416.
When two gateway messages arrived concurrently, _set_session_env wrote
HERMES_SESSION_PLATFORM/CHAT_ID/CHAT_NAME/THREAD_ID into the process-global
os.environ. Because asyncio tasks share the same process, Message B would
overwrite Message A's values mid-flight, causing background-task notifications
and tool calls to route to the wrong thread/chat.
Replace os.environ with Python's contextvars.ContextVar. Each asyncio task
(and any run_in_executor thread it spawns) gets its own copy, so concurrent
messages never interfere.
Changes:
- New gateway/session_context.py with ContextVar definitions, set/clear/get
helpers, and os.environ fallback for CLI/cron/test backward compatibility
- gateway/run.py: _set_session_env returns reset tokens, _clear_session_env
accepts them for proper cleanup in finally blocks
- All tool consumers updated: cronjob_tools, send_message_tool, skills_tool,
terminal_tool (both notify_on_complete AND check_interval blocks), tts_tool,
agent/skill_utils, agent/prompt_builder
- Tests updated for new contextvar-based API
Fixes#7358
Co-authored-by: teknium1 <127238744+teknium1@users.noreply.github.com>
Two fixes from PR review:
1. Session expiry was looking in _running_agents for the cached agent,
but idle expired sessions live in _agent_cache. Now checks
_agent_cache first, falls back to _running_agents.
2. Global cleanup in stop() was missing process_registry.kill_all(),
so background processes from agents evicted without close() (branch,
fallback) survived shutdown.
Use getattr guard for _agent_cache_lock in _handle_reset_command
because test fixtures may create GatewayRunner without calling
__init__, leaving the attribute unset.
Fixes e2e test failure: test_new_resets_session,
test_new_then_status_reflects_reset, test_new_is_idempotent.
Add 9 tests covering the full zombie process prevention chain:
- TestZombieReproduction: demonstrates that processes survive when
references are dropped without explicit cleanup (the original bug)
- TestAgentCloseMethod: verifies close() calls all cleanup functions,
is idempotent, propagates to children, and continues cleanup even
when individual steps fail
- TestGatewayCleanupWiring: verifies stop() calls close() and that
_evict_cached_agent() does NOT call close() (since it's also used
for non-destructive cache refreshes)
- TestDelegationCleanup: calls the real _run_single_child function and
verifies close() is called on the child agent
Ref: #7131
Wire AIAgent.close() into every gateway code path where an agent's
session is actually ending:
- stop(): close all running agents after interrupt + memory shutdown,
then call cleanup_all_environments() and cleanup_all_browsers() as
a global catch-all
- _session_expiry_watcher(): close agents when sessions expire after
the 5-minute idle timeout
- _handle_reset_command(): close the old agent before evicting it from
cache on /new or /reset
Note: _evict_cached_agent() intentionally does NOT call close() because
it is also used for non-destructive cache refreshes (model switch,
branch, fallback) where tool resources should persist.
Ref: #7131
Add is_network_accessible() helper using Python's ipaddress module to
robustly classify bind addresses (IPv4/IPv6 loopback, wildcards,
mapped addresses, hostname resolution with DNS-failure-fails-closed).
The API server connect() now refuses to start when the bind address is
network-accessible and no API_SERVER_KEY is set, preventing RCE from
other machines on the network.
Co-authored-by: entropidelic <entropidelic@users.noreply.github.com>
When enabled, @mentioning the bot in a DM creates a thread (default:
false). Supports both env var and YAML config (matrix.dm_mention_threads).
6 new tests, docs updated.
From #6957
Bot-added and bot-removed events were silently dropped because
_on_bot_added_to_chat and _on_bot_removed_from_chat were not
registered in _build_event_handler().
From #6975
Replace the simple DISCORD_IGNORE_NO_MENTION check with bot-aware
multi-agent filtering. When multiple agents share a channel:
- If other bots are @mentioned but this bot is not → stay silent
- If only humans are mentioned but not this bot → stay silent
- Messages with no mentions still flow to _handle_message for the
existing DISCORD_REQUIRE_MENTION check
- DMs are unaffected (always handled)
This prevents both agents from responding when only one is addressed.
- Remove sys.path.insert hack (leftover from standalone dev)
- Add token lock (acquire_scoped_lock/release_scoped_lock) in
connect()/disconnect() to prevent duplicate pollers across profiles
- Fix get_connected_platforms: WEIXIN check must precede generic
token/api_key check (requires both token AND account_id)
- Add WEIXIN_HOME_CHANNEL_NAME to _EXTRA_ENV_KEYS
- Add gateway setup wizard with QR login flow
- Add platform status check for partially configured state
- Add weixin.md docs page with full adapter documentation
- Update environment-variables.md reference with all 11 env vars
- Update sidebars.ts to include weixin docs page
- Wire all gateway integration points onto current main
Salvaged from PR #6747 by Zihan Huang.
Telegram's Bot API only allows a specific set of emoji for bot reactions
(the ReactionEmoji enum). ✅ (U+2705) and ❌ (U+274C) are not in that
set, causing on_processing_complete reactions to silently fail with
REACTION_INVALID (caught at debug log level).
Replace with 👍 (U+1F44D) / 👎 (U+1F44E) which are always available in
Telegram's allowed reaction list. The 👀 (eyes) reaction used by
on_processing_start was already valid.
Based on the fix by @ppdng in PR #6685.
Fixes#6068
Simplified implementation of the feature from PR #6842 (RunzhouLi).
Allows Discord channels/forum threads to auto-bind skills via config:
discord:
channel_skill_bindings:
- id: "123456"
skills: ["skill-a", "skill-b"]
The run.py auto-skill loader now handles both str and list[str],
loading multiple skills in order and concatenating their payloads.
Forum threads inherit their parent channel's bindings.
Co-authored-by: RunzhouLi <RunzhouLi@users.noreply.github.com>
Add debug logging when eyes reaction redaction fails, and add tests
for the success=False path and the no-pending-reaction edge case.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The on_processing_complete handler was never removing the eyes reaction because
_send_reaction didn't return the reaction event_id.
Fix:
- _send_reaction returns Optional[str] event_id
- on_processing_start stores it in _pending_reactions dict
- on_processing_complete redacts the eyes reaction before adding completion emoji
When extra.base_url is set in the Telegram platform config, use it as
the base URL for all Telegram API requests instead of api.telegram.org.
This allows agents to route Telegram traffic through the credential
proxy, which injects the real bot token — the VM never sees it.
Also supports extra.base_file_url for file downloads (defaults to
base_url if not set separately).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Follow-up to Dusk1e's PR #7120 (Slack send_image redirect guard):
- Rename _safe_url_for_log -> safe_url_for_log (drop underscore) since
it is now imported cross-module by the Slack adapter
- Add _ssrf_redirect_guard httpx event hook to cache_image_from_url()
and cache_audio_from_url() in base.py — same pattern as vision_tools
and the Slack adapter fix
- Update url_safety.py docstring to reflect broader coverage
- Add regression tests for image/audio redirect blocking + safe passthrough
The API server's _run_agent() was not passing task_id to
run_conversation(), causing a fresh random UUID per request. This meant
every Open WebUI message spun up a new Docker container and tore it down
afterward — making persistent filesystem state impossible.
Two fixes:
1. Pass task_id="default" so all API server conversations share the same
Docker container (matching the design intent: one configured Docker
environment, always the same container).
2. Derive a stable session_id from the system prompt + first user message
hash instead of uuid4(). This stops hermes sessions list from being
polluted with single-message throwaway sessions.
Fixes#3438.
Slack may return an HTML sign-in/redirect page instead of actual media
bytes (e.g. expired token, restricted file access). This adds two layers
of defense:
1. Content-Type check in slack.py rejects text/html responses early
2. Magic-byte validation in base.py's cache_image_from_bytes() rejects
non-image data regardless of source platform
Also adds ValueError guards in wecom.py and email.py so the new
validation doesn't crash those adapters.
Closes#6829
Platforms that don't return a message_id after the first send (Signal,
GitHub webhooks) were causing GatewayStreamConsumer to re-enter the
"first send" path on every tool boundary, posting one platform message
per tool call (observed as 155 PR comments on a single response).
Fix: treat _message_id == "__no_edit__" as a sentinel meaning "platform
accepted the send but cannot be edited". When a tool boundary arrives
in that state, skip the message_id/accumulated/last_sent_text reset so
all continuation text is delivered once via _send_fallback_final rather
than re-posted per segment.
Also make prompt_toolkit imports in hermes_cli/commands.py optional so
gateway and test environments that lack the package can still import
resolve_command, gateway_help_lines, and COMMAND_REGISTRY.
- configure Telegram HTTPXRequest pool/timeouts with env-overridable defaults\n- use separate request/get_updates request objects to reduce pool contention\n- skip fallback-IP transport when proxy is configured (or explicitly disabled)\n\nThis mitigates recurrent pool-timeout failures during polling reconnect/bootstrap (delete_webhook).
Python assertions are stripped when running with `python -O` (optimized
mode), making them unsuitable for runtime error handling.
1. `telegram_network.py:113` — After exhausting all fallback IPs, the code
uses `assert last_error is not None` before `raise last_error`. In
optimized mode, the assert is skipped; if `last_error` is unexpectedly
None, `raise None` produces a confusing `TypeError` instead of a
meaningful error. Replace with an explicit `if` check that raises
`RuntimeError` with a descriptive message.
2. `feishu.py:975` — The `_configure_with_overrides` closure uses
`assert original_configure is not None` as a guard. While the outer
scope only installs this closure when `original_configure` is not None,
the assert would silently disappear in optimized mode. Replace with an
explicit `if` check for defensive safety.
When /background was sent during an active run, it was not in the
platform adapter's bypass list and fell through to the interrupt path
instead of spawning a parallel background task.
Add "background" to the active-session command bypass in the platform
adapter, and add an early return in the gateway runner's running-agent
guard to route /background to _handle_background_command() before it
reaches the default interrupt logic.
Fixes#6827
Automated dead code audit using vulture + coverage.py + ast-grep intersection,
confirmed by Opus deep verification pass. Every symbol verified to have zero
production callers (test imports excluded from reachability analysis).
Removes ~1,534 lines of dead production code across 46 files and ~1,382 lines
of stale test code. 3 entire files deleted (agent/builtin_memory_provider.py,
hermes_cli/checklist.py, tests/hermes_cli/test_setup_model_selection.py).
Co-authored-by: alt-glitch <balyan.sid@gmail.com>
Legacy flat stt.model config key (from cli-config.yaml.example and older
versions) was passed as a model override to transcribe_audio() by the
gateway, bypassing provider-specific model resolution. When the provider
was 'local' (faster-whisper), this caused:
ValueError: Invalid model size 'whisper-1'
Changes:
- gateway/run.py, discord.py: stop passing model override — let
transcribe_audio() handle provider-specific model resolution internally
- get_stt_model_from_config(): now provider-aware, reads from the correct
nested section (stt.local.model, stt.openai.model, etc.); ignores
legacy flat key for local provider to prevent model name mismatch
- cli-config.yaml.example: updated STT section to show nested provider
config structure instead of legacy flat key
- config migration v13→v14: moves legacy stt.model to the correct
provider section and removes the flat key
Reported by community user on Discord.
Follow-up to cherry-picked PR #6592:
- Extract _webhook_url property to deduplicate URL construction
- Add _find_registered_webhooks() helper for reuse
- Crash resilience: check for existing registration before POSTing
(handles restart after unclean shutdown without creating duplicates)
- Accept 200-299 status range (not just 200) for webhook creation
- Unregister removes ALL matching registrations (cleans up orphaned dupes)
- Add 17 tests covering register/unregister/find/edge cases
**Problem:**
The BlueBubbles iMessage gateway was not receiving incoming messages even though:
1. BlueBubbles Server was properly configured and running
2. Hermes gateway started without errors
3. Webhook listener was started on the configured port
The root cause was that the BlueBubbles adapter only started a local webhook
listener but never registered the webhook URL with the BlueBubbles server via
the API. Without registration, the server doesn't know where to send events.
**Fix:**
1. Added _register_webhook() method that POSTs to /api/v1/webhook with the
listener URL and event types (new-message, updated-message, message)
2. Added _unregister_webhook() method for clean shutdown
3. Both methods handle the case where webhook listens on 0.0.0.0/127.0.0.1
by using 'localhost' as the external hostname
4. Fixed documentation: 'hermes gateway logs' → 'hermes logs gateway'
**API Reference:**
https://docs.bluebubbles.app/server/developer-guides/rest-api-and-webhooks
**Testing:**
- Webhook registration is now automatic when gateway starts
- Failed registration logs a warning but doesn't prevent startup
- Clean shutdown unregisters the webhook
Closes: iMessage gateway not working issue
Add Discord thread support to cron delivery and send_message_tool.
- _parse_target_ref: handle discord platform with chat_id:thread_id format
- _send_discord: add thread_id param, route to /channels/{thread_id}/messages
- _send_to_platform: pass thread_id through for Discord
- Discord adapter send(): read thread_id from metadata for gateway path
- Update tool schema description to document Discord thread targets
Cherry-picked from PR #7046 by pandacooming (maxyangcn).
Follow-up fixes:
- Restore proxy support (resolve_proxy_url/proxy_kwargs_for_aiohttp) that was
accidentally deleted — would have caused NameError at runtime
- Remove duplicate _DISCORD_TARGET_RE regex; reuse existing _TELEGRAM_TOPIC_TARGET_RE
via _NUMERIC_TOPIC_RE alias (identical pattern)
- Fix misleading test comments about Discord negative snowflake IDs
(Discord uses positive snowflakes; negative IDs are a Telegram convention)
- Rewrite misleading scheduler test that claimed to exercise home channel
fallback but actually tested the explicit platform:chat_id parsing path
The delivery tuple in webhook.py only had 5 of 14 platforms with
gateway adapters. Adds whatsapp, matrix, mattermost, homeassistant,
email, dingtalk, feishu, wecom, and bluebubbles so webhooks can
deliver to any connected platform.
Updates docs delivery options table to list all platforms.
Follow-up to cherry-picked fix from olafthiele (PR #7035).
Custom providers defined in config.yaml under were
completely invisible to the /model command in both gateway (Telegram,
Discord, etc.) and CLI. The provider listing skipped them and explicit
switching via --provider failed with "Unknown provider".
Root cause: gateway/run.py, cli.py, and model_switch.py only read the
dict from config, ignoring entirely.
Changes:
- providers.py: add resolve_custom_provider() and extend
resolve_provider_full() to check custom_providers after user_providers
- model_switch.py: propagate custom_providers through switch_model(),
list_authenticated_providers(), and get_authenticated_provider_slugs();
add custom provider section to provider listings
- gateway/run.py: read custom_providers from config, pass to all
model-switch calls
- cli.py: hoist config loading, pass custom_providers to listing and
switch calls
Tests: 4 new regression tests covering listing, resolution, and gateway
command handler. All 71 tests pass.
prompt_builder.py: The `hidden_div` detection pattern uses `.*` which does not
match newlines in Python regex (re.DOTALL is not passed). An attacker can bypass
detection by splitting the style attribute across lines:
`<div style="color:red;\ndisplay: none">injected content</div>`
Replace `.*` with `[\s\S]*?` to match across line boundaries.
credential_files.py: `_load_config_files()` catches all exceptions at DEBUG level
(line 171), making YAML parse failures invisible in production logs. Users whose
credential files silently fail to mount into sandboxes have no diagnostic clue.
Promote to WARNING to match the severity pattern used by the path validation
warnings at lines 150 and 158 in the same function.
webhook.py: `_reload_dynamic_routes()` logs JSON parse failures at WARNING (line
265) but the impact — stale/corrupted dynamic routes persisting silently — warrants
ERROR level to ensure operator visibility in alerting pipelines.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
dingtalk.py: The session_webhook URL from incoming DingTalk messages is POSTed to
without any origin validation (line 290), enabling SSRF attacks via crafted webhook
URLs (e.g. http://169.254.169.254/ to reach cloud metadata). Add a regex check
that only accepts the official DingTalk API origin (https://api.dingtalk.com/).
Also cap _session_webhooks dict at 500 entries with FIFO eviction to prevent
unbounded memory growth from long-running gateway instances.
api_server.py: The X-Hermes-Session-Id request header is accepted and echoed back
into response headers (lines 675, 697) without sanitization. A session ID
containing \r\n enables HTTP response splitting / header injection. Add a check
that rejects session IDs containing control characters (\r, \n, \x00).
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Add DISCORD_ALLOWED_CHANNELS (env var) / discord.allowed_channels (config.yaml)
support to restrict the bot to only respond in specified channels.
When set, messages from any channel NOT in the allowed list are silently
ignored — even if the bot is @mentioned. This provides a secure default-
deny posture vs the existing ignored_channels which is default-allow.
This is especially useful when bots in other channels may create new
channels dynamically (e.g., project bots) — a blacklist requires constant
maintenance while a whitelist is set-and-forget.
Follows the same config pattern as ignored_channels and free_response_channels:
- Env var: DISCORD_ALLOWED_CHANNELS (comma-separated channel IDs)
- Config: discord.allowed_channels (string or list of channel IDs)
- Env var takes precedence over config.yaml
- Empty/unset = no restriction (backward compatible)
Files changed:
- gateway/platforms/discord.py: check allowed_channels before ignored_channels
- gateway/config.py: map discord.allowed_channels → DISCORD_ALLOWED_CHANNELS
- hermes_cli/config.py: add allowed_channels to DEFAULT_CONFIG
The session store was copying the ENTIRE parent DM transcript into new
thread sessions. This caused unrelated conversations to bleed across
threads in Slack DMs.
The Slack adapter already handles thread context correctly via
_fetch_thread_context() (conversations.replies API), which fetches
only the actual thread messages. The session-level seeding was both
redundant and harmful.
No other platform (Telegram, Discord) uses DM threads, so the seeding
code path was only triggered by Slack — where it conflicted with the
adapter-level context.
Tests updated to assert thread isolation: all thread sessions start
empty, platform adapters are responsible for injecting thread context.
Salvage of PR #5868 (jarvisxyz). Reported by norbert on Discord.
The gateway /model command stored session overrides in
_session_model_overrides but run_sync() never consulted them when
resolving the model and runtime for the next message. It always read
from config.yaml, so the switch was lost as soon as a new agent was
created.
Two fixes:
1. In run_sync(), apply _session_model_overrides after resolving from
config.yaml/env — the override takes precedence for model, provider,
api_key, base_url, and api_mode.
2. In post-run fallback detection, check whether the model mismatch
(agent.model != config_model) is due to an intentional /model switch
before evicting the cached agent. Without this, the first message
after /model would work (cached agent reused) but the fallback
detector would evict it, causing the next message to revert.
Affects all gateway platforms (Telegram, Discord, Slack, WhatsApp,
Signal, Matrix, BlueBubbles, HomeAssistant) since they all share
GatewayRunner._run_agent().
Fixes#6213
Two security hardening changes for the API server:
1. **Startup warning when no API key is configured.**
When `API_SERVER_KEY` is not set, all endpoints accept unauthenticated
requests. This is the default configuration, but operators may not
realize the security implications. A prominent warning at startup
makes the risk visible.
2. **Require authentication for session continuation.**
The `X-Hermes-Session-Id` header allows callers to load and continue
any session stored in state.db. Without authentication, an attacker
who can reach the API server (e.g. via CORS from a malicious page,
or on a shared host) could enumerate session IDs and read conversation
history — which may contain API keys, passwords, code, or other
sensitive data shared with the agent.
Session continuation now returns 403 when no API key is configured,
with a clear error message explaining how to enable the feature.
When a key IS configured, the existing Bearer token check already
gates access.
This is defense-in-depth: the API server is intended for local use,
but defense against cross-origin and shared-host attacks is important
since the default binding is 127.0.0.1 which is reachable from
browsers via DNS rebinding or localhost CORS.
The gateway /usage handler only looked in _running_agents for the agent
object, which is only populated while the agent is actively processing a
message. Between turns (when users actually type /usage), the dict is
empty and the handler fell through to a rough message-count estimate.
The agent object actually lives in _agent_cache between turns (kept for
prompt caching). This fix checks both dicts, with _running_agents taking
priority (mid-turn) and _agent_cache as the between-turns fallback.
Also brings the gateway output to parity with the CLI /usage:
- Model name
- Detailed token breakdown (input, output, cache read, cache write)
- Cost estimation (estimated amount or 'included' for subscriptions)
- Cache token lines hidden when zero (cleaner output)
This fixes Nous Portal rate limit headers not showing up for gateway
users — the data was being captured correctly but the handler could
never see it.
The text batching feature routes TEXT messages through
asyncio.create_task() + asyncio.sleep(delay). Even with delay=0,
the task fires asynchronously and won't complete before synchronous
test assertions. This broke 33 tests across Discord, Matrix, and
WeCom adapters.
When _text_batch_delay_seconds is 0 (the test fixture setting),
dispatch directly to handle_message() instead of going through
the async batching path. This preserves the pre-batching behavior
for tests while keeping batching active in production (default
delay 0.6s).
Feishu already had text batching with a static 0.6s delay. This adds
adaptive delay: waits 2.0s when a chunk is near the ~4096-char split
point since a continuation is almost certain.
Tracks _last_chunk_len on each queued event to determine the delay.
Configurable via HERMES_FEISHU_TEXT_BATCH_SPLIT_DELAY_SECONDS (default 2.0).
Ref #6892
Ports the adaptive batching pattern from the Telegram adapter.
WeCom clients split messages around 4000 chars. Adaptive delay waits
2.0s when a chunk is near the limit, 0.6s otherwise. Only text messages
are batched; commands/media dispatch immediately.
Ref #6892
Ports the adaptive batching pattern from the Telegram adapter.
Matrix clients split messages around 4000 chars. Adaptive delay waits
2.0s when a chunk is near the limit, 0.6s otherwise. Only text messages
are batched; commands dispatch immediately.
Ref #6892
Cherry-picked from PR #6894 by SHL0MS with fixes:
- Only batch TEXT messages; commands/media dispatch immediately
- Use build_session_key() for proper session-scoped batch keys
- Consistent naming (_text_batch_delay_seconds)
- Proper Dict[str, MessageEvent] typing
Discord splits at 2000 chars (lowest of all platforms). Adaptive delay
waits 2.0s when a chunk is near the limit, 0.6s otherwise.
Cherry-picked from PR #6891 by SHL0MS.
When a chunk is near the 4096-char split point, wait 2.0s instead of 0.6s
since a continuation is almost certain.
