IronClaw

IronClaw claws back a twelve-second Postgres regression

IronClaw traces a hosted PostgreSQL capacity regression to its row-native process journal: global commit serialization, observer write amplification and per-thread index triggers pushed p95 to 12 seconds. A forward-recovery branch reports local p95 back near 4.15 seconds while also documenting a separate libSQL writer pathology.

← Back to homeOriginal source ↗

nearai/ironclaw PR #6973 is the inspected primary source: “perf: recover hosted Postgres API capacity regressed by the row-native process journal (#6696).” The two related records below were inspected as supporting context, not treated as independent confirmation.

The facts

- nearai/ironclaw PR #6973 was created 2026-07-31T19:35:13Z and was open at inspection; its title is “perf: recover hosted Postgres API capacity regressed by the row-native process journal (#6696).” - nearai/ironclaw issue #6974 was created 2026-07-31T19:35:37Z and was open at inspection; its title is “libSQL thread_store_writes pathology: tool-heavy stress cases at p95 37-135s post-#6696.” - nearai/ironclaw PR #6696 was created 2026-07-27T07:22:11Z and was closed at inspection; its title is “Collapse lifecycle state into the row-native process journal.” - The repository reported 12587 stars, 1433 open issues, default branch main, and last push 2026-08-01T21:44:19Z. - The latest tagged-release baseline checked was ironclaw-v1.0.0 published 2026-07-27T19:32:01Z.

What changed

nearai/ironclaw PR #6973 reports: Problem The scheduled hosted-single-tenant Postgres API capacity gate regressed from p95 **3.74s / 6.86 ops/sec** (last passing nightly, b60038b) to p95 **12.0s / 2.57 ops/sec** (4c075913, run 30604660405) — send_message p95 went 275ms → 4.78s while mock-LLM latency stayed flat at 3.0s. The libSQL bottleneck/soak nightlies also began timing out. Bisection attributes the regression to #6696 (*Collapse lifecycle state into the row-native process journal*); a revert is infeasible (92 conflicting files, the replaced stores were deleted), so this recovers forward. Root causes (verified by instrumented benchmarks) 1. **Globally serialized commits.** Every lifecycle command (submit/claim/heartbeat/transition) ran its own transaction that locked the single /processes/materialized/journal-sequence row until

nearai/ironclaw issue #6974 reports: Split out of #6973 (Postgres capacity recovery). With all of #6973's fixes, the nightly-shape libSQL bottleneck suite (ops 20 / c4 / u50) finally **completes** — main cannot get past the large-context prefill inside the 20-minute CI timeout — but the tool-heavy cases remain far over the 2.5s p95 gate: | case | p95 | dominant op group | |---|---|---| | chat-baseline | 8.18s (10.46s on failing main nightly) | turn_store | | large-context | 3.67s | thread_store_writes | | tool-heavy | 87.75s | thread_store_writes (p95 63.5s) | | tool-wait | 73.30s | thread_store_writes (p95 51.9s) | | tool-failure | 135.57s | thread_store_writes (p95 95.9s)

nearai/ironclaw PR #6696 reports: Summary Make ironclaw_processes the lifecycle authority and keep turn state as process-journal projections. Use row-native journal/materialized records, transactional cursor allocation, durable observer cursors, bounded claim pagination, and explicit lease-recovery policy. Import deployed lifecycle state from /turns/rows/v1, /turns/state.json, and /run-state/.../runs before row-native initialization, preserving blocked/resumable state, checkpoints, leases, lineage, idempotency, and capability invocations. Maintain filesystem ordered projections transactionally across in-memory, libSQL, and PostgreSQL, including prefix-scoped declarations and descending keyset parity. Prevent idle supervisor polls from reserving unused global journal cursor ranges; reservations now match the exact entries a command will emit. Add deterministic supervisor fault coverage and split the turn process projection into focused runtime, metadata,

Why it matters

A cleaner lifecycle authority can still make the service slower if every transition contends on one sequence row and every observer writes its cursor separately. The repair is credible because it names round trips, trigger growth and before-and-after workloads rather than merely promising optimisation. This is an operator boundary: persisted state, execution ownership and visible controls must describe the same event. A workaround that broadens access, deletes state or hides the symptom would trade a visible defect for an unmeasured one.

Current

The primary record was open when captured on 2026-08-02. The release baseline was ironclaw-v1.0.0 published 2026-07-27T19:32:01Z. Current repository metadata, pull requests, issues, release records, Google News discovery, Hacker News discovery, Lobsters, Metamesh and the rendered ClawCharts row were checked. ClawCharts selected the subject; it did not prove the claim.

Evidence

The primary URL is https://github.com/nearai/ironclaw/pull/6973. Supporting records are https://github.com/nearai/ironclaw/issues/6974 and https://github.com/nearai/ironclaw/pull/6696. Their source bodies, timestamps and states were captured in the daily evidence bundle. Test counts, reproductions and deployment observations remain attributed to their authors unless explicitly identified as independently rerun.

Source boundary

Open work is described as open, closed work as closed, and operator reports as reports. A pull request is evidence of proposed or reviewed direction, not proof of a shipped release. Issue closure would not by itself prove a deployed fix. Search residue, package mirrors and historically published source spines were excluded.

Operator take

Wait for the scheduled CI verdict, monitor retryable group-commit failures, retain downgrade migration instructions, and keep the libSQL single-writer problem separate from the recovered PostgreSQL capacity claim. Preserve a before-state receipt, make the smallest reversible change, and verify the original failure independently after intervention.

Caveat

Public project records are mutable. Status, scope and evidence can change after publication. This brief records the inspected state and does not authorize changes to a reader’s deployment.

Source inspected; source state, environment and release boundary remain explicit.