Hermes Agent
Hermes can misclassify provider exhaustion as a permanent Kanban protocol failure
A Hermes Agent report traces a provider-wall shutdown through exit code 0 into auto-blocking as protocol_violation—the classification that explicitly forbids retry. A sibling gateway report says transient provider errors can also discard orchestration turns rather than resume them.
NousResearch/hermes-agent issue #73747 is the inspected primary source: “Kanban worker stopped by a provider wall exits 0 and is auto-blocked as a protocol violation, the one classification that forbids a retry.” The two related records below were inspected as supporting context, not treated as independent confirmation.
The facts
- NousResearch/hermes-agent issue #73747 was created 2026-07-29T00:51:30Z and was open at inspection; its title is “Kanban worker stopped by a provider wall exits 0 and is auto-blocked as a protocol violation, the one classification that forbids a retry.” - NousResearch/hermes-agent issue #73748 was created 2026-07-29T00:51:31Z and was open at inspection; its title is “Gateway orchestration turn aborted by a transient provider error (rate limit) is lost, not retried.” - NousResearch/hermes-agent PR #73726 was created 2026-07-29T00:30:01Z and was open at inspection; its title is “fix(cron): repair kanban goal_mode judge loop that never executes.” - The repository reported 221904 stars, 26179 open issues, default branch main, and last push 2026-07-29T00:42:37Z. - The latest tagged-release baseline checked was v2026.7.20 published 2026-07-20T18:35:55Z.
What changed
NousResearch/hermes-agent issue #73747 reports: Related to #46593 and #27924, which report the same clean_exit classification from different causes: a crash before the first API call, and a worker completing silently. This one is a third cause with a different remedy, and the remedy already exists in the codebase. Closing as a duplicate is fine if you disagree, but the discriminator below is not covered by either. v0.18.0 already distinguishes a transient provider failure from a task failure, and does it well. hermes_cli/kanban_db.py::_classify_worker_exit maps KANBAN_RATE_LIMIT_EXIT_CODE (75) to a rate_limited exit, and detect_crashed_workers then releases the card back to ready **without** incrementing the failure counter, so a long quota window cannot trip the circuit breaker. That is exactly the right behaviour. The problem is which runs can reach it. In cli.py the worker's exit code is decided by: failure_reason is only set where agent/conversation_loop.py returns failed: True after exhausting API retries. When the provider walls the run but the turn still ends normally — the model produces a final assistant message
NousResearch/hermes-agent issue #73748 reports: Related to #68771 (treating provider 5xx as a fallback-chain trigger) and #58061 (provider-level throttling), but distinct: this is about what happens to the inbound message *after* every model in the chain has already failed. When an inbound message triggers an agent turn on the gateway and every configured model (primary plus fallback_model) returns a transient error (e.g. HTTP 429 rate limiting) for that turn, the turn aborts and the agent posts a user-facing "the model provider is rate-limiting requests, please wait and try again" message. Expected (or at least desirable): the failed turn is retried after a backoff, or the inbound message is re-queued so the orchestration resumes on its own once the provider recovers. Actual: the inbound message is treated as processed. There is no automatic retry of the turn and no re-queue. If that turn was the one meant to create a kanban task (e.g. an orchestrator turning a chat request into a dispatched task), the task is never created, and nothing downstream
NousResearch/hermes-agent PR #73726 reports: Problem On a board with 97 goal_mode=1 tasks over ~7 weeks, the goal loop has executed **zero** times. The judge itself is healthy — the loop is structurally unreachable. Root cause run_kanban_goal_loop (hermes_cli/goals.py:1717-1734) checks task_status_fn() **before** calling judge_goal(). Since compliant workers always call kanban_complete or kanban_block during their first turn, the status is already terminal on the first iteration — so the loop returns immediately with completed_by_worker / blocked_by_worker and judge_goal is **never called**. Fix Swap the ordering: run judge_goal **first**, then check task status. Only honor worker self-termination when the judge also agrees the work is done. A continue verdict overrides the worker's termination, giving the judge authority to decide when more work is needed. Also adds a quick anomalous-status check on every iteration (reclaimed/archived/etc.) so those edge cases still stop the loop regardless of verdict. Testing All 8 goal loop tests pass, including 2 new tests: test_loop_judge_overrides_worker_completion — worker self-terminates but judge says continue test_loop_judge_overrides_worker_blocked — worker blocks but judge says continue Closes
Why it matters
Provider exhaustion is temporary capacity state. Translating it into successful process exit plus permanent workflow blame loses both the work and the true cause, while teaching automation not to retry. This is an operator boundary: persisted state, execution ownership and visible controls must describe the same event. A workaround that broadens access, deletes state or hides the symptom would trade a visible defect for an unmeasured one.
Current
The primary record was open when captured on 2026-07-29. The release baseline was v2026.7.20 published 2026-07-20T18:35:55Z. Current repository metadata, pull requests, issues, release records, Hacker News discovery results and the rendered ClawCharts row were checked. ClawCharts selected the subject; it did not prove the claim.
Evidence
The primary URL is https://github.com/NousResearch/hermes-agent/issues/73747. Supporting records are https://github.com/NousResearch/hermes-agent/issues/73748 and https://github.com/NousResearch/hermes-agent/pull/73726. Their source bodies, timestamps and states were captured in the daily evidence bundle. Test counts, reproductions and deployment observations remain attributed to their authors unless explicitly identified as independently rerun.
Source boundary
Open work is described as open, closed work as closed, and operator reports as reports. A pull request is evidence of proposed or reviewed direction, not proof of a shipped release. Issue closure would not by itself prove a deployed fix. Search residue, package mirrors and historically published source spines were excluded.
Operator take
Preserve a typed provider-wall receipt across worker, gateway and Kanban boundaries; successful shutdown is not successful task completion. Preserve a before-state receipt, make the smallest reversible change, and verify the original failure independently after intervention.
Caveat
Public project records are mutable. Status, scope and evidence can change after publication. This brief records the inspected state and does not authorize changes to a reader’s deployment.
Source inspected; source state, environment and release boundary remain explicit.