September 7: unreachable replicas and database deadline hardening

Incident evidence

Between approximately 09:28 and 09:34 UTC, Railway's proxy repeatedly failed to connect to both Yumina application replicas in us-west2. Each failed request attempted one replica for five seconds, then the other for five seconds, and returned 502. There were 6,228 5xx responses in the 09:28–09:33 minute buckets, including retries. The 09:32 bucket was 914/915 failures. Deployment cddb134ee had been live since 08:55; there was no replacement deploy during the incident.

PostHog recorded database calls taking approximately 205, 345 and 197 seconds, followed by dropped primary connections. The production Postgres compute did not restart. Railway resource summaries showed memory and aggregate CPU below their configured limits, and the inspected runtime logs did not show a restart or OOM. Aggregate CPU headroom cannot exclude Node main-thread saturation. The originating host/network/application event remains unconfirmed; these changes address reproduced reliability gaps, not a proven fix for that original trigger.

Database and service boundaries

• Production runs Hono/Node with Drizzle and node-postgres; the main service serves APIs and the application shell. Two app replicas are configured in us-west2.
• Neon production primary stores accounts, worlds, conversations, billing state and game profiles. The optional separate read endpoint handles public browsing; private account state and read-before-write guards stay on primary. The local development Neon endpoint is separate from production.
• Each process has a primary pool capped at 100 and a read pool capped at 80. Keepalive, connection establishment deadlines and existing read-only/connection-error eviction guards remain in place.
• Redis coordinates shared caching, rate limiting, notifications and background-job leases. Redis commands already have a three-second deadline and fail without queuing commands while disconnected.
• PvZ's HTML/JS/WASM are bucket-hosted and served through the PvZ delivery routes. Its realtime relay is a separate service. Dave/account save operations use the platform API/database; the game does not write every simulation frame to Postgres.

Read-only inspection of production confirmed statement_timeout=5min, lock_timeout=20s and idle_in_transaction_session_timeout=1min. The current activity sample contained no lock waiters. No schema or production role setting was changed.

Confirmed gaps and changes

Safe query deadlines

DATABASE_QUERY_TIMEOUT_MS defaults to 60,000 milliseconds for ordinary primary and read queries. This bounds client query execution, while the existing pool acquisition/connect timeout separately bounds waiting for a connection. Explicit schema self-heal and leader-coordinated scheduled maintenance use a bounded 300,000-millisecond scope. Health SQL probes use 4,000 milliseconds.

The installed pg 8.18 driver's query_timeout alone rejects the caller without cancelling an active query. The new guard intercepts the driver's timeout callback before pool release or Drizzle rollback, marks the client unusable and force-closes its local socket. Subsequent transaction commands fail with the original DB_QUERY_TIMEOUT, preserving the error through Drizzle rollback cleanup. The transaction owner still releases its own client. Healthy connections and ordinary SQL errors retain normal rollback/reuse behavior.

The error carries outcome: unknown: losing a COMMIT response does not prove that the write failed. The guard never retries a statement. Closing the socket is not a guarantee of immediate cancellation at Neon/PgBouncer; server-side statement limits remain necessary. Routes reaching the global error handler return 503 with instructions to refresh and check the operation's result.

Timeout scopes use async-local storage. A regression test exposed that pg-pool can serve a waiting query inside the previous borrower's async context, so pooled queries capture their own deadline at submission. This prevents a health request from inheriting a maintenance deadline, or vice versa. Await Drizzle's lazy thenables inside withDatabaseQueryTimeout, not after leaving its callback. The protected SQL pools do not support cursor/Submittable query lifecycles.

Health checks that do not amplify an outage

GET /health remains the dependency readiness endpoint used by Railway deployment checks. It retains the existing response fields and a five-second response limit. Concurrent callers share one probe per dependency, with a one-second cache. A timed-out probe stays shared until its underlying operation settles; another health request cannot abandon it and start more SQL. Successful probes clear their timeout handles. Responses have Cache-Control: no-store.

GET /health/live answers without querying Postgres, Redis, auth or an AI provider. Comparing it with readiness distinguishes a responding app with dependency trouble from a process/network reachability failure. It is a diagnostic endpoint, not a new automatic restart policy or an externally hosted monitor.

Per-replica diagnostics

A local 1,200-millisecond main-thread freeze produced p95=32ms and max=1,201ms. The former p95-only threshold suppressed the event. Runtime sampling now reports if p95 reaches 50ms, max reaches 1,000ms, or the sampling timer is delayed by at least 1,000ms. Healthy samples are also recorded every 30 seconds so a missing heartbeat is distinguishable from a healthy quiet period.

Runtime samples include per-process CPU usage over the measured interval, event-loop utilization, RSS/heap use, uptime, region, replica, deployment and boot identity. Query timing, connection counts, pool pressure, request timing and captured errors carry the same identity. Runtime reporting cannot throw into the application and stops on graceful shutdown. A blocked process can only report its stall after execution resumes; these metrics are not a live stack trace or CPU profile.

Validation and remaining work

Regression tests use a local PostgreSQL wire peer with the real pg driver, pool and Drizzle transaction code. They cover a hung pooled write, transaction cleanup, an uncertain COMMIT, queued commands, replacement connections, SQL-error rollback/reuse, scoped deadlines and cross-request pool handoffs. Health tests cover unavailable dependencies, concurrent/repeated timeouts, recovery and liveness. Runtime tests cover isolated freezes, delayed timers and failed telemetry sends. Existing connection safety, read routing, boot safety and Dave notebook tests are included in the focused run.

The two app replicas remain in one region. Regional failover is not solved by changing a timeout or moving one replica blindly: it needs a deliberate plan for primary-database access, replica freshness, Redis and job leadership. Railway host/network records for the two affected instances are still needed to attribute the original outage. No provider support message was sent from this task.

Rollout checks: confirm both replica states; verify /health/live, /health, guest auth/session and /pvz/; check runtime records from both new boot identities and look for query timeout/connection errors. No production fault injection or load flood is part of verification.

Follow-up findings during rollout

Commit cce676df3 deployed successfully at 10:31 UTC, with both replicas online and all public health, guest-session and PvZ delivery checks passing. The live PvZ 3928a638e3 WASM matched the latest local build by SHA-256. Its game assets were already uploaded; this reliability work does not replace them.

The new runtime diagnostics immediately exposed isolated 3.9-second and 3.0-second main-thread delays despite p95 remaining 22–27ms. Multiple unrelated query callbacks completed together after one pause. A bounded inspection of the recent message pages involved found approximately 4.1 MB of combined text, snapshots and swipes across eight sessions, not an unbounded history fetch. There is no CPU profile proving which function caused those particular live pauses.

A separate local reproduction confirmed a serious synchronous tokenizer weakness: js-tiktoken's adjacent-pair merge is quadratic for an individual large regex piece. Counting only 20,000 repeated A characters took approximately 14 seconds on the main thread. Budget estimation now uses the tokenizer's own piece boundaries and a conservative UTF-8 byte upper bound for pieces exceeding 256 bytes. Ordinary text keeps its cl100k counts, and Gemini/Claude/DeepSeek's existing CJK estimates remain unchanged. Oversized pieces may therefore consume more estimated context budget; the prompt text itself is not rewritten. This removes the reproduced pathological merge work without asserting it caused the earlier outage.

Live logs also showed openrouter/free rejecting story-summary requests because a routed model requires reasoning while summaries explicitly disable it. The existing one-step DeepSeek summary fallback now also recognizes this specific 400 compatibility error. It keeps the shared attempt allowance, provider/key selection, output cap and existing usage/billing rules. A successful fallback is recorded as the effective summary model, as with the existing content-refusal fallback; unrelated authentication and malformed-request errors are preserved.

Follow-up validation includes all 920 engine tests and 96 targeted server tests covering summary recovery, provider compatibility, shared attempt limits, database deadlines, runtime health and Dave regression checks. The original 20 KB repeated-letter benchmark fell from about 14 seconds to less than 1 millisecond after the guard; the ordinary 92 KB English and 84 KB Chinese samples retained their exact counts.

The follow-up release still observed an isolated 2.9-second pause, so the tokenizer vulnerability does not explain every stall. An opt-in RUNTIME_CPU_PROFILE_SECONDS diagnostic (default 0, maximum 180) samples once per process at 10ms intervals, then disconnects. It logs only the 25 heaviest sampled call stacks with function/file locations, not source text, request bodies or heap objects. It does not open a debugger port. A temporary 120-second capture is used to identify the remaining synchronous work, then the setting is returned to zero. Two additional tests exercise the real bounded profiler and dynamic-source redaction.

Production CPU profile and history counting

The bounded capture on deployment 265fcff9a completed on both replicas. The busier replica's heaviest application stacks were tokenization under selectCompactionWindow, estimateMessagesTokens, compactTurnOverflowIfNeeded and buildMessageHistory. Several individual sampled stacks accounted for roughly 1.2–3.0 seconds each over the two-minute capture. These are sampled cumulative times, not timings of individual pauses, and they do not retrospectively prove the cause of the 09:32 outage. They do establish repeated synchronous history tokenization as a live CPU bottleneck.

Summary preparation now yields to Node's sockets/timers between message batches and reuses exact per-row counts throughout window selection and chunking. The weak-reference cache checks content, model, role, metrics and attachment count, and does not retain startup heuristic counts across tokenizer initialization. Prompt overflow checks stop once the budget is exceeded instead of tokenizing every older row; under-budget sums remain exact. A separate bounded LRU retains at most 1,024 cl100k counts and two million string characters, so repeated prompt passes reuse counts without unbounded text retention. CJK model estimation stays unchanged.

The cooperative scheduler checks an eight-millisecond budget between messages, not within each message. This reduces monopolization by large histories but is not a hard per-message CPU deadline or a capacity guarantee. The profiling setting was returned to zero before the follow-up deployment. Regression coverage verifies timer responsiveness, early overflow exit, identical summary selection and cache invalidation; the expanded focused server suite contains 116 tests.

Deployment b334ab218 succeeded at 11:07:46 UTC. Both new replicas emitted healthy runtime samples and public health, guest and game checks passed. PostHog subsequently caught a 2,084ms pause on one process, with unrelated auth/notification queries completing together. This means the remaining stall work cannot be declared resolved merely because the first samples looked healthy.

The original CPU profile also identified final buildMessageHistory trimming. Unlike the overflow pass, this path runs for conversations without an enabled summarization handler. Send, regenerate and continue now use a cooperative version of that method. Synchronous and asynchronous callers share one generator implementing the same pinning, newest-message retention and token-budget decisions; the server supplies setImmediate, keeping the engine independent of Node. Regression tests cover preserved pinned blocks, CJK model behavior, oversized newest messages, empty/unlimited histories and real timer progress without a summary pass. The final test totals are 921 engine tests and 117 targeted server tests.

Railway's connector and CLI both failed OAuth refresh during this rollout (invalid_scope and invalid_grant). Deployment success was checked through Railway's GitHub commit status, and distinct new replicas were verified through PostHog runtime events. The provider authentication issue does not change application credentials or deployment configuration.