IronClaw
IronClaw’s benchmark audit says 71 failures belong to the model, but three belong to its leak guard
IronClaw’s July 25 failure taxonomy separates 85 non-passing ClawBench tasks into 71 model-quality misses, three local-write blocks caused by secret-shaped text, and a small evaluation tail. It proposes moving hard blocking to the real egress boundary.
nearai/ironclaw issue #6676 is the inspected primary source: “Daily ironclaw failure taxonomy — 2026-07-25.” The generated taxonomy analyzes a ClawBench run at revision 346db3cc6a6f487f9ca3f111997abfe5a3bdd37c. It says earlier byte-cap and crude secret-marker defects are fixed, reducing non-passes from 115 to 85. Seventy-one remaining tasks are categorized as partial completion where the model missed rubric checks. Three tasks share a harness defect: the provider argument scanner hard-blocks local write_file calls when an incident report contains secret-shaped strings. The audit traces that rejection through validate_provider_tool_call and argues the actual exfiltration boundary is the separately guarded HTTP path. It keeps two bad-evaluation examples separate rather than charging them to IronClaw.
The facts
- nearai/ironclaw issue #6676 was created 2026-07-25T11:03:41Z and updated 2026-07-25T11:03:41Z; state is open. - nearai/ironclaw returned 12557 GitHub stars, 1405 open issues, default branch main, and last push 2026-07-26T00:12:52Z. - Latest tagged-release baseline checked: ironclaw-v1.0.0-rc.1 published 2026-07-21T03:57:55Z. - The generated taxonomy analyzes a ClawBench run at revision 346db3cc6a6f487f9ca3f111997abfe5a3bdd37c. - It says earlier byte-cap and crude secret-marker defects are fixed, reducing non-passes from 115 to 85. - Seventy-one remaining tasks are categorized as partial completion where the model missed rubric checks. - Three tasks share a harness defect: the provider argument scanner hard-blocks local write_file calls when an incident report contains secret-shaped strings. - The audit traces that rejection through validate_provider_tool_call and argues the actual exfiltration boundary is the separately guarded HTTP path.
What changed
The generated taxonomy analyzes a ClawBench run at revision 346db3cc6a6f487f9ca3f111997abfe5a3bdd37c. It says earlier byte-cap and crude secret-marker defects are fixed, reducing non-passes from 115 to 85. Seventy-one remaining tasks are categorized as partial completion where the model missed rubric checks. Three tasks share a harness defect: the provider argument scanner hard-blocks local write_file calls when an incident report contains secret-shaped strings. The audit traces that rejection through validate_provider_tool_call and argues the actual exfiltration boundary is the separately guarded HTTP path. It keeps two bad-evaluation examples separate rather than charging them to IronClaw. The primary record was inspected with its timestamps, state, body, and adjacent project records. ClawCharts supplied the assignment list; it does not establish the claim.
Why it matters
Benchmark repair becomes dishonest when every failure is blamed on the harness, but it is equally wasteful to train around a guard that blocks legitimate local artifacts. The useful work is classification: model quality, harness policy, and evaluator defects need different owners and different fixes. Moving hard enforcement to egress preserves security while allowing incident-response text to exist locally. The operational test is whether persisted state, visible control, and runtime behavior agree at the handoff where authority changes.
Current
Issue #6676 was open and links every categorized task to its trajectory. Its confidence for the three-task leak-detector cluster is 0.60, so the proposed boundary change still needs adversarial tests. IronClaw’s latest tagged baseline was 1.0.0-rc.1. A separate mutation-audit PR found and repaired a decorative ordering test, reinforcing the project’s current emphasis on evidence quality. Repository metadata, current issues and pull requests, release baseline, Hacker News discovery, and the rendered ranking row were checked. Weak search residue and historically used source spines were not promoted.
Evidence
The primary URL is https://github.com/nearai/ironclaw/issues/6676. Supporting links are inspected current project records that establish implementation, adjacent direction, or boundary conditions. Dates, reproductions, test totals, and deployment observations remain attributed to their authors unless this brief explicitly says they were independently rerun.
Source boundary
Open work is described as open, merged work as merged, and operator reports as reports. A closed issue or passing branch test is not silently translated into a released fix. Ranking explains why the project was scanned; it does not prove correctness, adoption, or package behavior.
Operator take
Reproduce one local write containing synthetic secret-shaped strings and one HTTP egress attempt with the same payload. The local artifact should be admitted or safely redacted according to policy; the network attempt must remain blocked. Score only viable benchmark failures against the correct owner. Preserve a before-state receipt and verify the narrowest invariant where state changes. Do not broaden permissions or delete state merely because the visible symptom is inconvenient.
Caveat
Public project records are mutable. Source state was captured at publication time, and later revisions may change status, scope, or evidence. No source here authorizes automatic mutation of a reader’s deployment.
Source inspected; source state, environment, and release boundary remain explicit.