Deployment 018ab08e: database connection crash

Evidence

• Commit 938cf240e started on two Railway replicas around 06:40 UTC and passed both deployment health checks. The public health endpoint and deployed memory retry asset were also verified.
• At 06:46:30 UTC one replica logged an uncaught PostgreSQL error: server conn crashed?, severity FATAL, SQLSTATE 08P01. The existing last-resort exception handler exited the process. Railway then reported one crashed and one running replica; the public health endpoint continued to pass through the surviving replica.
• The same error was reported by POST /api/check-ins/claim, which uses a database transaction. Adjacent logs recorded an 82,410 ms event-loop delay and many similarly delayed queries. The logs do not identify the synchronous work responsible for that stall, or establish why the backend connection failed.

Confirmed application defect and fix

The application listened for pool-level errors, but not client-level errors. In the installed pg-pool implementation, checking out a client removes its idle error listener. A fatal backend error between transaction queries then emits an unhandled client error event. This is reproducible with the installed pg driver's error path without opening a network connection.

Both pools now install a persistent error listener on each client. Pending and future queries still fail; errors remain logged and reported. The client is marked unusable and is destroyed when its transaction owner releases it. The listener does not release someone else's transaction, retry writes, or disable the process's last-resort exception handler. Existing pool-level handling remains in place for idle clients.

References: node-postgres client events, pool error events. Runtime behavior was checked against the locally installed pg 8.18.0 and pg-pool 3.11.0 source, not inferred from a newer API.

Additional memory load reduction

The legacy repair path, now reachable from automatic updates, fetched up to 400 historical state snapshots while using only the last eligible reply's snapshot. It now loads transcript text first and fetches only that endpoint snapshot. This removes unnecessary JSON transfer and parsing; it is a precaution against a concrete source of avoidable work, not proof that those snapshots caused the observed 82-second stall.

Verification and rollout

Regression tests exercise fatal errors between queries on both pools, propagation of pending query failures, destruction/replacement of a failed checked-out connection using the real pool lifecycle, and preservation of idle-pool error handling. The memory integration test also verifies legacy repair uses the eligible endpoint's state and excludes earlier/later snapshots.

Pre-release verification passed: 68 focused tests, 33 related memory integration tests, pnpm build (5 tasks), and pnpm typecheck (8 tasks). Integration tests used an isolated PGlite database and scripted provider output, with no production model calls. No schema change is required.

After rollout, verify both replica states and observe beyond the previous six-minute failure interval; one successful public health request cannot establish that every replica is healthy. Event-loop stalls remain a separate observation to investigate if they recur.