Image generation: scale foundation (testing only)

Scope and capacity target

Planning scenario: 20,000 online users, each generating one image every ten minutes. That is 33.3 images/second. At 30 seconds per image, approximately 1,000 concurrent provider requests are needed before spare capacity. This is a workload assumption, not measured user behavior or a provider capacity commitment. At the measured Seedream rate of $0.035/image it would cost approximately $4,200/hour of continuous traffic, excluding our own hosting and storage.

This release builds the independent execution and notification layers. It does not certify the whole application, payment flow, database or supplier at 20,000 users.

Flow

1. The authenticated website validates image-only input, ownership, safety, storage and credits. A transaction reserves cost and creates a durable PostgreSQL job. Request IDs prevent a lost response from creating another paid job.
2. Independent image-worker replicas claim queued OpenRouter jobs. PostgreSQL enforces the shared concurrency ceiling. A provider HTTP request with an unknown result is never automatically replayed. Each purchased batch image gets its own provider request; batch requests are sequential within a job.
3. The worker saves outputs to the shared object store, commits asset receipts and completion, and refunds missing outputs or failures through the credit ledger.
4. Redis broadcasts a user-scoped invalidation. The website pushes only change events over authenticated SSE; the browser then reads its own current jobs. There is one subscriber connection per web process, not one Redis/SQL connection per browser. The browser shares one SSE stream across generation panels, and stops it for hidden tabs or when there are no pending jobs/refunds.
5. Lost events are repaired by jittered reads every 60–90 seconds while connected, or 30–45 seconds while disconnected. Reconnects use backoff and jitter. Streams reauthenticate after at most five minutes. Each connection buffers at most one pending invalidation and is subject to per-user and per-process caps.

Custom RunPod images remain a second channel. They still submit through the web API and run on RunPod; their reconciliation/refund recovery moves to the worker. Video submissions remain disabled. Model synchronization remains on the web service in this phase.

Reliability changes

• Cloud timeout recovery selects only expired jobs, in bounded batches. A large healthy queue cannot hide a timed-out running job. Refund recovery runs before slow custom-provider status requests.
• Contention on the global admission lock returns a retriable busy response after rollback instead of retaining unlimited waiting database sessions. User/wallet lock waits are bounded. No charge is retained for a rejected admission.
• Worker startup requires image credentials, database, Redis and object storage. Each replica records a heartbeat with a 90-second expiry. Job parameters retain the claiming worker ID for operational verification.
• GET /api/generation/admin/operations is admin-only and exposes cached queue, reservation, fleet heartbeat and per-web-replica event health information.
• Worker shutdown stops claiming and waits up to 270 seconds for paid work. Railway's supported drainingSeconds ceiling is 300 seconds. Web shutdown keeps its own shorter drain. Unexpected kills still recover through durable timeouts; they cannot reconstruct an upstream response that was lost.

Deployment and controls

YUMINA_PROCESS_ROLE=image-worker selects the independent Docker entry point. IMAGE_EXECUTION_MODE=external disables web dispatch/reconciliation; the default embedded mode preserves local development and rollback compatibility.

Testing rollout uses a separate image-worker service with two replicas, local concurrency 2 each and database pool limit 12 each. They share the web service's database, Redis, object storage, supplier credentials and budget through Railway variable references. Worker source watches server/shared/engine/build files so an app-only change does not restart paid generation. Root railway.toml is shared; the worker also has explicit health/predeploy/draining service settings because Railway now rejects new custom config-file path overrides.

Keep testing global concurrency 10, queue limit 100 and rolling 24-hour supplier budget $25. Effective execution capacity with two workers is four concurrent jobs. The budget alone allows roughly 714 single images per rolling day at $0.035 each. Higher validated configuration ceilings do not enable additional paid work by themselves. Roll back by setting web execution to embedded, then stopping worker replicas after their drain; shared database limits remain in effect during overlap.

What still limits a large launch

• Admission and dispatch still share one database advisory lock. Daily-budget admission still aggregates reservations from the last 24 hours. This remains a throughput limit; large bursts can receive 429. Next measure regional transaction latency and replace this with sharded durable admission accounting/worker slots.
• Full authenticated HTTP submission, wallet/ledger transactions, SQL pool pressure, Redis across web replicas, CDN delivery and reconnect storms require a distributed load generator. A transport benchmark cannot certify these paths.
• Obtain real supplier throughput/error-rate evidence for the chosen image model; OpenRouter does not imply unlimited model capacity. Increasing keys or worker counts alone does not secure upstream capacity.
• Decide acceptable queue wait and per-user usage before increasing the budget. A 30-minute queued-job expiry deliberately refunds excess waiting work. Memory must be sized for bounded image response buffers, not just HTTP connections.

Verification

The generation regression suite adds user-isolated fanout, bounded slow-client coalescing, actual SSE authentication/caps/disconnect cleanup and cloud timeout recovery behind 80 healthy queued jobs. Run:

pnpm --filter @yumina/server exec tsx --test src/lib/generation/*.test.ts
pnpm --filter @yumina/server exec tsx scripts/benchmark-generation-events.ts 2000
pnpm build
pnpm typecheck

The local benchmark uses the actual Hono SSE handler and real loopback sockets, with synthetic authentication and Redis and no SQL or paid AI requests. The final 2,000-connection run delivered 2,000/2,000 invalidations, with burst p95 76 ms and 137 MiB RSS growth for the combined client/server process, with zero remaining subscriptions after disconnect. All 55 related tests, the full build and all eight typecheck tasks passed. A 20,000-connection attempt exhausted the Windows load generator's socket resources (ENOBUFS); it is not a successful 20,000-connection result. The separate 20,000-subscriber unit test validates routing/cleanup logic only. Final testing deployment and smoke results are recorded after rollout below.

Testing rollout verified — 2026-09-07 05:01 UTC

• Deployed code: 392eabcdc254cd143cc3cbd31811e5b91b8f6335 on testing, including the concurrent blueprint updates already on that branch. Build/typecheck passed again after integration; the learning guard and four catalog tests also passed.
• Web deployment: 0c71689b-20cc-4816-a92c-6a46acfc1879, healthy. Its admin operations response confirmed executionMode: external.
• Worker deployment: bea0e5d9-7e98-4158-8b07-0436ef656dc6, two healthy replicas. It remained on this same deployment while the web was redeployed to external execution. Explicit service draining is 300 seconds; storage/credential variable references match the testing web service. No worker exists in production.
• Two real single-image jobs succeeded: f4c71ae6-0955-49ec-b094-c97f44b4fa51 and 2d81c796-eb12-49f2-ba6a-42f6d5a39a55. Both recorded a worker ID from the independent fleet; both happened to run on the same replica, so this smoke is not evidence of balanced scheduling. Each image was served successfully by the CDN. Supplier cost was $0.035 each, $0.07 total.
• Each request was repeated with the same request ID and retained exactly one usage charge. Real Redis-to-web SSE changes arrived. Unauthenticated SSE returned 401 and ordinary-user operations access returned 403. The ledger hash chain remained valid; dedicated fixture sessions were revoked and credits cleared.
• Testing limits remain global 10, queue 100, budget $25/rolling day, and two replicas × local concurrency 2. Production was not deployed.

This is a verified independent-worker rollout and a 2,000-connection local transport result, not certification of a 20,000-user generation service.