Hermes Agent
Hermes Agent wires fast decision layer into Playwright computer use
Hermes Agent proposed factorized decision loops for browser computer use, pairing Playwright execution with an optional Jev classifier to skip full frontier-model round trips. Replayable decision packets replace raw screenshots, while Daytona snapshot loading speeds sandboxed tool execution.
Hermes Agent contributors submitted PR #114532 on September 18, 2026, introducing a factorized decision architecture for browser-based computer use. Instead of streaming full-page screenshots back to a frontier language model on every single click, scroll, and keystroke, the new computer_use(action=decide) workflow evaluates structured actions, targets, vision needs, and completion status through a fast local pipeline of rules, rerankers, and optional auxiliary models like Jev. The change hooks into a Playwright browser backend with native type_text support and produces deterministic decision_packet logs that can be audited or replayed without storing sensitive user screenshots. Related work in PR #114536 enables Daytona environments to boot from pre-built snapshots instead of pulling raw container images on demand, while PR #114416 hardens gateway webhook admission against non-durable cron states.
The facts
- PR #114532 implements computer_use(action=decide) and run_goal loops without requiring frontier model invocations on routine interaction steps. - Factorizes UI control into discrete classifications: ACTION, TARGET, NEEDS_VISION, NEEDS_GENERATION, and DONE. - Supports Playwright as the execution backend via HERMES_COMPUTER_USE_BACKEND=browser with working text typing. - Emits structured decision_packet records containing candidates, latency, verifier scores, and selected actions without capturing or saving private screenshot images. - PR #114536 enables Daytona sandboxes to instantiate from pre-baked snapshots via CreateSandboxFromSnapshotParams, avoiding registry pull delays.
Why it matters
Computer use has historically been throttled by two bottlenecks: massive token consumption from sending high-resolution images on every turn, and high latency from multi-second frontier model inferences for basic UI navigations. By separating high-level planning from mechanical UI decision loops, Hermes Agent cuts step latency and operating cost dramatically. Furthermore, eliminating screenshot persistence in execution logs removes a major privacy and secret-leakage hazard for enterprise operators who need verifiable audit trails without exposing rendered credentials.
Current
Inspected on 2026-09-18. The Hermes Agent stable-channel baseline is v2026.9.14 published 2026-09-14T16:04:14Z. The primary source was open (not merged) when captured. Mainline merges, open proposals and packaged releases are distinct availability states; the release baseline does not establish that a proposal has shipped.
Evidence
Primary evidence: NousResearch/hermes-agent PR #114532 (https://github.com/NousResearch/hermes-agent/pull/114532). Supporting context: NousResearch/hermes-agent PR #114536 (https://github.com/NousResearch/hermes-agent/pull/114536); NousResearch/hermes-agent PR #114416 (https://github.com/NousResearch/hermes-agent/pull/114416). The source bodies and linked context were inspected. Test results quoted from contributors remain attributed reports, not independently reproduced experiments.
Operator take
Treating browser control as a localized classification problem rather than an open-ended vision-language conversation is the natural evolution of agentic automation. Operators should look closely at the decision_packet contract: being able to A/B test UI rules against auxiliary models without rerunning live web tasks makes browser agent debugging tractable for the first time. The addition of Playwright alongside OS-level drivers provides flexible deployment across headless cloud VMs.
Caveat
The fast decision loop fails open to the full frontier planner whenever generation text is missing or ambiguity thresholds are exceeded. The Jev decision provider is optional and requires external API keys; environments without keys fall back to heuristic rules and auxiliary rerankers. System-level permissions still require explicit human approval.
The fast decision loop fails open to the full frontier planner whenever generation text is missing or ambiguity thresholds are exceeded. The Jev decision provider is optional and requires external API keys; environments without keys fall back to heuristic rules and auxiliary rerankers. System-level permissions still require explicit human approval.