ZeroClaw
ZeroClaw builds a live evaluation lane with repeat statistics
ZeroClaw proposes sandboxed live evaluation, repeated runs with pass@k and pass^k, and a replay regression gate in CI. The live path isolates each case, caps repeats at 50, reports error bars, and treats consistency across repeats as the gating standard.
zeroclaw-labs/zeroclaw PR #9224 is the inspected primary source: “feat(eval): repeated live runs with pass@k, pass^k, and error bars.” The related records below were inspected as supporting context rather than independent confirmation.
The facts
- zeroclaw-labs/zeroclaw PR #9224 was created 2026-07-21T00:38:17Z and was open at inspection; its title is “feat(eval): repeated live runs with pass@k, pass^k, and error bars.” - zeroclaw-labs/zeroclaw PR #9214 was created 2026-07-20T21:41:33Z and was open at inspection; its title is “feat(eval): live execution mode with sandboxed tool surface.” - zeroclaw-labs/zeroclaw PR #9212 was created 2026-07-20T20:52:28Z and was open at inspection; its title is “feat(eval): gate CI on the replay regression suite.” - The repository reported 32506 stars, 721 open issues, default branch master, and last push 2026-08-04T23:25:55Z. - The latest tagged-release baseline checked was v0.8.4 published 2026-08-02T21:00:00Z.
What changed
zeroclaw-labs/zeroclaw PR #9224 reports: Summary **Base branch:** master (all contributions) **What changed and why:** Adds **repeated live runs** with pass@k / pass^k and suite **error bars**, so a flaky live case is measured rather than judged on a single sample. This is #7065's repeat-statistics dimension. Case fields repeat (clamped 1..=50; replay runs once with a warning) and an optional cluster label. run_case_repeated runs a live case k fully isolated times and aggregates RepeatStats (passes/k, pass@k, pass^k, per-check flip counts, mean +/- sample stddev of tokens and duration). **A case passes for gating/baselines iff pass^k** (the consistency standard): the representative outcome is a failing run when any exists, so CaseReport::passed() equals pass^k. Suite error bar: per-case success proportions are collapsed first (one value per case), then cluster-averaged, and the report prints pass rate p-bar +/-t*SEM (95% CI). With fewer than two independent units it reports the observed pass rate and explicitly marks the interval unavailable
zeroclaw-labs/zeroclaw PR #9214 reports: Summary **Base branch:** master (all contributions) **What changed and why:** Adds zeroclaw eval run --mode live: run a case against a real configured provider inside a per-case sandbox, alongside the existing deterministic replay. This is the "live" half of #7065's replay + live design. Restructures the runner around a RunDeps provider-factory closure so replay, live, and deterministic tests share one code path; the runner never constructs a provider itself. Extends the case format (optional steps, id/display_id(), setup.workspace_files, tools) and adds validate_workspace_rel_path so setup and (later) workspace grading cannot escape the sandbox. Adds [eval].live_provider, live_allowed_tools, and case_timeout_secs with Config::validate wiring (dotted ref, dangling-alias, empty opt-out). **Scope boundary:** No graders/receipts/baselines/judge (milestones D–K). No new tool-order or memory graders. Replay grading is unchanged. Does not touch [agent.eval]. **Blast radius:** zeroclaw eval run (new mode + threaded &Config); the eval crate's provider construction; the shared tests/support/mock_model_provider.rs (mechanical: iterate Option steps). **Linked issue(s):** Related #7065
zeroclaw-labs/zeroclaw PR #9212 reports: Summary **Base branch:** master **What changed and why:** Splits replay fixtures into evals/regression/ and makes that suite a hard CI gate through crates/zeroclaw-eval/tests/regression_suite.rs. Adds a negative regression case (no_tools_on_greeting.json) so the gate covers a must-not-happen behavior. Changes the default [eval].suite_dir from evals to evals/regression and keeps the CLI help, config rustdoc, and crate README examples aligned with that source of truth. Factors the process exit decision into the pure SuiteReport::exit_code() boundary with unit coverage. Documents the suite taxonomy and exit-code contract in evals/README.md and the operations book. **Scope boundary:** Does not touch [agent.eval] (EvalConfig), make load_suite recursive, add a separate CI workflow step, or add live-mode/new-grader behavior. **Blast radius:** The default suite used by zeroclaw eval run and the eval crate test target. **Linked issue(s):** Related #7065. #9225 is the fixture-seeding child. **Labels:** enhancement, docs, core, config, principal contributor, risk:medium, size:S Testing (required) How you can test **Reviewer testing requested?**
Why it matters
Agent evaluations fail when a single lucky sample becomes a release claim. Repeated live runs and replay gates make variance visible, but provider calls and open dependency stacks still carry cost and integration risk. The operator test is whether persisted state, execution authority and the visible outcome describe the same event after retries, restarts or delegation.
Current
The primary record was open when captured on 2026-08-05. The release baseline was v0.8.4 published 2026-08-02T21:00:00Z. Repository metadata, pull requests, issues, release records, Google News discovery, Hacker News discovery, Lobsters, Metamesh and the rendered ClawCharts row were checked. ClawCharts selected the subject; it did not prove the claim.
Evidence
The primary URL is https://github.com/zeroclaw-labs/zeroclaw/pull/9224. Supporting records are https://github.com/zeroclaw-labs/zeroclaw/pull/9214 and https://github.com/zeroclaw-labs/zeroclaw/pull/9212. Source bodies, timestamps and states are preserved in the daily evidence bundle. Test counts, reproductions and deployment observations remain attributed to their authors unless explicitly identified as independently rerun.
Source boundary
Open work is described as open, closed work as closed, and operator reports as reports. A pull request is evidence of proposed or reviewed direction, not proof of a shipped release. Search residue, package mirrors and historically published source spines were excluded.
Operator take
Separate replay from live budgets, require fresh landing-head CI, preserve per-case workspace isolation, and publish uncertainty instead of a single pass percentage. Preserve a before-state receipt, make the smallest reversible change, and verify the original failure independently after intervention.
Caveat
Public project records are mutable. Status, scope and evidence can change after publication. This brief records the inspected state and does not authorize changes to a reader’s deployment.
Source inspected; source state, environment and release boundary remain explicit.