Paperclip
Paperclip operators trace a database lock convoy that can blind health checks
A Paperclip operator says database pressure wedged ten connections behind one idle transaction, leaving `/api/health` hanging. A second report says process IDs are usually absent, while a patch bounds queued-start storms to one waiter and one takeover.
paperclipai/paperclip issue #10867 is the inspected primary source: “agents-row FOR UPDATE convoy under DB I/O pressure consumes the pool; /api/health hangs with no 503.” The related records below were inspected as supporting context rather than independent confirmation.
The facts
- paperclipai/paperclip issue #10867 was created 2026-08-05T00:36:49Z and was open at inspection; its title is “agents-row FOR UPDATE convoy under DB I/O pressure consumes the pool; /api/health hangs with no 503.” - paperclipai/paperclip issue #10866 was created 2026-08-05T00:36:48Z and was open at inspection; its title is “heartbeat_runs.process_pid is almost never persisted — running rows have no usable liveness signal.” - paperclipai/paperclip PR #10868 was created 2026-08-05T00:44:02Z and was open at inspection; its title is “fix(heartbeat): coalesce queued-run start attempts instead of stacking them behind a stale lock.” - The repository reported 75620 stars, 5033 open issues, default branch master, and last push 2026-08-05T01:05:40Z. - The latest tagged-release baseline checked was v2026.722.0 published 2026-07-22T23:05:41Z.
What changed
paperclipai/paperclip issue #10867 reports: Pre-submission checklist [x] I have searched existing open and closed issues and this is not a duplicate. [x] I am on the latest released version of Paperclip (or can reproduce on master). [x] I have confirmed the error originates in Paperclip itself — not in my agent adapter, API provider, or local configuration. What happened? Under database I/O pressure, the whole API (including GET /api/health) hangs indefinitely. pg_stat_activity shows the pool's 10 connections in a lock convoy: One connection holds a transaction that ran SELECT id FROM agents WHERE id=$1 AND company_id=$2 FOR UPDATE, then INSERT INTO agent_wakeup_requests..., and is now idle in transaction on Client/ClientRead — the Node side never sends the next statement and never commits. Every other connection is active, blocked on Lock/tuple or Lock/transactionid behind that row lock, running the same agents... FOR UPDATE statement. With all pool connections consumed, /api/health's SELECT 1 can
paperclipai/paperclip issue #10866 reports: Pre-submission checklist [x] I have searched existing open and closed issues and this is not a duplicate. [x] I am on the latest released version of Paperclip (or can reproduce on master). [x] I have confirmed the error originates in Paperclip itself — not in my agent adapter, API provider, or local configuration. What happened? heartbeat_runs.process_pid and process_started_at are NULL for essentially every run, including healthy runs that complete successfully. Measured on our deployment (2026.722.0, claude-local adapter, ~30 agents, ~2,000 runs/day): consecutive one-hour windows show 0 of 12, 0 of 30, 0 of 34, 0 of 30 succeeded runs with a recorded PID; the best hour we ever measured was 4 of 52. Under load the write appears to be deferred and then lost. Expected behavior The child PID is persisted at (or immediately after) fork, so process_pid is a usable liveness signal for a run in status='running'. Why it
paperclipai/paperclip PR #10868 reports: Thinking Path - Paperclip is the open source app people use to manage AI agents for work. - The heartbeat subsystem starts queued runs per agent under withAgentStartLock, which serialises start attempts for one agent. - The lock has a liveness escape: an attempt proceeds after a 30 s stale window even if the previous holder has not finished. For a single hung start, that is correct. - But every attempt gets the same escape. When start attempts arrive every few seconds against a degraded process (post-restart recovery burst, event-loop starvation), each attempt fires after its own window — unbounded concurrent claim+spawn sequences stack up per agent. - Each stacked claim marks a run running and re-stamps state; the spawn side then stalls. The result is running rows with no process, no PID, and no run log, each holding its issue's execution lock, plus a new execution_workspaces row per attempt.
Why it matters
The failure combines three weak signals: a health probe sharing the exhausted pool, running rows without process identity, and stale-lock recovery that can mint more phantom work. The operator test is whether persisted state, execution authority and the visible outcome describe the same event after retries, restarts or delegation.
Current
The primary record was open when captured on 2026-08-05. The release baseline was v2026.722.0 published 2026-07-22T23:05:41Z. Repository metadata, pull requests, issues, release records, Google News discovery, Hacker News discovery, Lobsters, Metamesh and the rendered ClawCharts row were checked. ClawCharts selected the subject; it did not prove the claim.
Evidence
The primary URL is https://github.com/paperclipai/paperclip/issues/10867. Supporting records are https://github.com/paperclipai/paperclip/issues/10866 and https://github.com/paperclipai/paperclip/pull/10868. Source bodies, timestamps and states are preserved in the daily evidence bundle. Test counts, reproductions and deployment observations remain attributed to their authors unless explicitly identified as independently rerun.
Source boundary
Open work is described as open, closed work as closed, and operator reports as reports. A pull request is evidence of proposed or reviewed direction, not proof of a shipped release. Search residue, package mirrors and historically published source spines were excluded.
Operator take
Give health a bounded probe, persist process identity at spawn, narrow lock scope, and verify degraded startup never creates an unbounded queue of running rows. Preserve a before-state receipt, make the smallest reversible change, and verify the original failure independently after intervention.
Caveat
Public project records are mutable. Status, scope and evidence can change after publication. This brief records the inspected state and does not authorize changes to a reader’s deployment.
Source inspected; source state, environment and release boundary remain explicit.