IronClaw reports an upgrade crash loop and a starved authority journal

nearai/ironclaw issue #7720 is the inspected primary source: “1.3.0-rc.1 crash-loops on boot after 1.2.x upgrade: unknown field `activation_state` in v2 extension installation row.” The related records below were inspected as supporting context rather than independent confirmation.

The facts

- nearai/ironclaw issue #7720 was created 2026-08-18T00:53:05Z and was open at inspection; its title is “1.3.0-rc.1 crash-loops on boot after 1.2.x upgrade: unknown field `activation_state` in v2 extension installation row.” - nearai/ironclaw PR #7717 was created 2026-08-17T20:55:37Z and was open at inspection; its title is “fix(resources): stop libSQL write-lane starvation from cascading through the resource governor (#7714).” - nearai/ironclaw issue #7714 was created 2026-08-17T19:42:31Z and was open at inspection; its title is “libSQL: single shared write connection starves the resource-governor journal under bench load (cascading authority invalidation, permanent reservation leaks).” - The repository reported 12607 stars, 1514 open issues, default branch main, and last push 2026-08-18T00:33:37Z. - The latest tagged-release baseline checked was ironclaw-v1.3.0-rc.1 published 2026-08-17T23:16:03Z.

What changed

nearai/ironclaw issue #7720 reports: 1.3.0-rc.1 fails to boot on any deployment upgraded from 1.2.x. The process exits 1 during composition and crash-loops until the restart policy gives up, so the worker's HTTP and SSH ports go dead. Symptom Observed on staging-01, container eager-lion-dulap-worker, image nearaidev/ironclaw:1.3.0-rc.1 (ironclaw.git.sha=5d71d1c25), Exited (1), RestartCount 5. Root cause Commit bdf8aef02 ("fix(migration): forward-port legacy state upgrade to 1.2") added a field to V2InstallationRecord in crates/extensions/ironclaw_extension_registry/src/installations.rs: It has serde(default) but **no skip_serializing_if**, so every installation row written by a 1.2.x build serializes the key. bdf8aef02 exists only on origin/release/2026-08-11. It was never forward-ported: It is not on main either. The reader struct is #[serde(deny_unknown_fields)], so on 1.3.0-rc.1 the extra key is a fail-closed startup error rather than an ignorable unknown field. The two field sets are otherwise identical — 1.2's record is exactly 1.3's plus activation_state. Ground truth from the affected box's state DB, path /system/extensions/.installations/v2/installations/.json: activation_state is the first key, which is exactly line 1 column 19. The downgrade direction is safe (serde(default) absorbs the missing field). Only the forward direction breaks. A note on parse_v2_entry (not causal) parse_v2_entry deserializes into the typed struct **before** it checks schema_version, while its sibling helper ~30 lines above does the opposite (parse to Value, check schema_version, then from_value). That ordering is **not** what

nearai/ironclaw PR #7717 reports: Summary Fixes #7714 — during a PinchBench run on libSQL, the resource governor's delta journal stalled ~40s behind the process-wide single write connection, then cascaded: authority invalidation every ~40s, permanently leaked reservations, and capability calls dying with a mislabeled process invocation not found. Three commits, one per subsystem, each independently verified: 1. fix(resources) — batch governor deltas and stop congestion poisoning the authority (f3a49a7) **Batching now engages.** The journal flusher's greedy drain meant every caller's blocking ack kept the channel empty — batch_size=1 on every line of the bench log. Replaced with a bounded ~2ms collection window (still capped at 256). Discriminating test: reverting the loop shape yields batches [1, 7]; the fix yields [8]. **Transient BackendBusy no longer poisons the authority.** Two coupled bugs: eager invalidate_authority on congestion, and the writer thread breaking on any batch error (so every later enqueue failed and replacement became genuinely necessary). Busy-exhaustion now fails the caller retryably, keeps the writer alive, and discards the authority for replay (no poison, no journal replacement); infrastructure errors still invalidate. In-memory state stays consistent with the append-only log via replay, not in-place rollback (later stacked operations make rollback unsafe). **Stale-Active reservation sweeper.** sweep_stale_active_reservations(max_age) releases aged Active holds through the normal release path (append-only Release delta — nothing

nearai/ironclaw issue #7714 reports: Summary During a PinchBench run (147 tasks, separate server, libSQL backend), the resource governor's delta journal repeatedly stalled ~40s waiting for a write connection, then cascaded: authority invalidated → journal replacement → durable-state reload every ~40s, failed reservation releases, and capability calls ending in host runtime unavailable: process invocation not found. Root cause investigation (read-only, source-cited below) shows this is **libSQL write-lane starvation under per-turn write pressure**, not disk or a regression — the #5081 pool-exhaustion pattern reappearing on the backend the #7471 journal split skipped. Log signature: Root cause On libSQL, every durable writer in the process shares one write connection.** LIBSQL_WRITER_POOL_MAX_CONNECTIONS = 1 with a 10s checkout timeout (crates/substrates/ironclaw_libsql_runtime/src/lib.rs:21-22); composition hands the same Arc to the governor, event store, and process journal (crates/app/ironclaw_composition/src/factory/production_backend_assembly.rs:19-26, 202-203). The elapsed_ms=10000/20000/30000 in the log is exactly 1/2/3 × the checkout timeout: the governor's write spent ~40s queued for the connection and never reached SQL. The backend is provable from the log shape alone: only the libSQL lane maps checkout timeout to the retryable BackendBusy the governor loops on (crates/substrates/ironclaw_filesystem/src/libsql.rs:203-220); Postgres pool failure maps to BackendInfrastructure and fails with zero retries (crates/substrates/ironclaw_filesystem/src/postgres.rs:188-201; terminal arm crates/kernel/ironclaw_resources/src/filesystem_governor/journal.rs:365). The #7471 journal pool split is Postgres-only.** process_journal_pool: Option (production_backend_assembly.rs:1578, used:1615-1617) is never populated on libSQL; backend_store_assembly.rs:192 clones

Why it matters

Schema compatibility and authority accounting meet at startup: if migration rejects state or the journal cannot write, the runtime can lose both availability and budget truth. The operator test is whether the system remains bounded and its receipts still describe the action after failure, retry or restart.

Current

The primary record was open when captured on 2026-08-18. The tagged-release baseline was ironclaw-v1.3.0-rc.1 published 2026-08-17T23:16:03Z. Repository metadata, full source bodies, current pull requests and issues, releases, Google News, Hacker News, Lobsters, Metamesh and the rendered ClawCharts row were inspected. ClawCharts selected the subject; it did not prove the claim.

Evidence

The primary URL is https://github.com/nearai/ironclaw/issues/7720. Supporting records are https://github.com/nearai/ironclaw/pull/7717 and https://github.com/nearai/ironclaw/issues/7714. Source bodies, timestamps and states are preserved in the daily evidence bundle. Test counts and reproductions remain attributed to their authors unless identified as independently rerun.

Source boundary

Open work is described as open, closed work as closed, and operator reports as reports. A pull request is evidence of proposed or reviewed direction, not proof of a shipped release. Search residue, package mirrors and historically published source spines were excluded.

Operator take

Make extension migrations backward-readable, reserve a bounded journal write path, and fail startup with a repairable migration receipt rather than an opaque restart loop. Preserve a before-state receipt, make the smallest reversible change, and verify the original failure independently.

Caveat

Public project records are mutable. Status, scope and evidence can change after publication. This brief records the inspected state and does not authorize changes to a reader’s deployment.

Source inspected; source state, environment and release boundary remain explicit.