Studio recovery during deployment

Failure and correction

The web deployment had no explicit draining time while the server waited 50 seconds before aborting streams. A container could disappear before its shutdown handler persisted recovery state. In addition, funded model calls had no durable preflight checkpoint, and completing one iteration cleared the checkpoint before the next.

• Journal each funded preflight, including messages, iteration, world revision and ownership claim, before the provider starts.
• Consume a completed result and journal the next step in one transaction.
• Clear the journal and transition to completed/user-input/image-approval together.
• Keep partial, unfinished generations at preflight; do not mistake missing final usage for a completed result with unavailable billing details.
• Preserve existing transactional write progress and settlement references so replay skips applied mutations and settled charges.
• Wait for aborted handlers to unregister after their final database writes, bounded at 10 seconds. Configure Railway web draining at 90 seconds.

Regression evidence

Tests use the real streamed agent loop, checkpoint transactions, world writes and credit ledger against isolated PGlite. Provider responses and failure boundaries are synthetic. No production user data is changed by these tests.

New coverage:

1. Deployment before the first provider chunk: preflight already exists; continuation finishes and produces one usage charge.
2. Deployment during the second generation: the first applied step and its conversation context survive; continuation adds the second step without repeating the first.
3. Lost worker without shutdown cleanup: age both heartbeat and claim, reclaim the persisted preflight, and reject the old worker's late output and charge.
4. Database trigger rejects any attempt to erase a journal while the run remains running, verifying atomic completion.
5. Shutdown waits for asynchronous persistence after abort, but remains bounded for an unresponsive handler.

Existing regression cases also cover generated-result replay, interruption between two writes, duplicate charge prevention, explicit user stop, stale-world conflicts, concurrent claims, and frontend recovery/reconnect state.

Validation on 2026-09-20:

• pnpm build: passed (existing bundle-size warnings).
• pnpm typecheck: passed.
• pnpm --filter @yumina/server test: 2,374 passed, 0 failed, 9 skipped (2,383 total; isolated launcher, no forced process exit).
• App studio-credit-recovery.test.ts: 27 passed, 0 failed.
• The first two new interruption regressions failed on the unchanged baseline before the fix, demonstrating the missing preflight/next-step journal.
• Railway 5.42.1 read-only plan: one safe change, web draining null to 90; no variable deletion or changes to other services.

The local verification above did not deploy code or configuration.

Rollout update on 2026-09-20: the reviewed Railway setting was applied, changing only the web service's draining time from null to 90 seconds. PR #184 subsequently deployed as f538fd2491d2f558a74fd5ba8c353890f23c41f2, Railway deployment b1191a2c-5365-4375-a805-a0855a9afa22; both replicas and primary/read DB plus Redis passed health checks.

A separate Railway project with an independent PostgreSQL database, two replicas and mocked provider transport passed a real deployment interruption test. The unchanged compiled server received SIGTERM during the second generation, retained the first applied write/charge, saved resumable preflight, and exited cleanly. The authenticated continuation API resumed the same run and completed it: both writes appeared once, three completed results had three unique usage entries, and duplicate continuation returned 409 without another charge. Test deployments were stopped. No deliberate production shutdown test was performed.

The necessary production rollout itself was not interruption-free: the old version drained 12 active streams, with 11 ending inside the drain window and one aborted at its deadline. A matching ordinary-chat regeneration recorded partial estimated usage with no linked credit transaction; its previous reply remained complete. Recovery must not be described as guaranteed zero interruption.

Deployment checks and limits

The reviewed Railway plan changes only yumina.deploy.drainingSeconds from null to 90. Live analysis-service variable names are preserved; Redis networking is left as imported. Pull live configuration again before applying if deployment happens later. Applying deploy settings itself redeploys the web service.

After deployment, verify the running commit and draining value, then inspect a controlled deployment's shutdown logs and a synthetic Studio task's continuation. Local tests do not substitute for that live rollout verification.

This guarantees a recoverable step boundary, not uninterrupted provider execution: an incomplete model call can need regeneration. The existing Continue action is used; there is no automatic cross-process continuation. Unexpected worker loss waits for the five-minute stale-worker threshold. Recovery uses the existing paid Studio credit-recovery feature; BYOK and disabled-feature behavior are unchanged. Existing historical runs that never stored a checkpoint cannot gain one retroactively. World edits made after a checkpoint still block replay to prevent overwriting newer work.