PvZ public scale — candidate and remaining release gates

September 20, 2026. This is a release-readiness record, not a certification that production supports 100,000 concurrent players.

Planning target

Pending the owner's answer, use 100,000 total players and a 10,000 concurrent player peak. Distinguish players from simultaneous AI requests and Versus rooms. The proposed release must pass representative staging traffic before promotion.

Implemented in the isolated release branch

• Campaign notebook reads now use one direct read for an existing campaign, avoiding a transaction, advisory lock and insert attempt on every visit. Creation remains locked and limited to sixteen campaigns per account. Tests exercise forty simultaneous reads and racing campaign creation beyond the cap.
• Content-addressed game resources under the exact packaged-art namespace get one-year immutable caching. Mutable HTML and user-owned media retain their prior policies. No Cloudflare configuration was changed.
• A per-process AI admission limit bounds active requests, including notebook, provider and settlement work. Default: 64; PVZ_DAVE_MAX_INFLIGHT accepts 1–4096. This is a conservative protective default, not a throughput promise. Distributed per-account limits remain in place. Excess work returns promptly with a retry indication before notebook/provider work; there is no unbounded queue or automatic paid retry. Metrics include active requests, capacity and the distinct capacity_limited outcome. Leases release exactly once.
• Existing account integration, save isolation, model controls, bounded native AI decisions and transactional official billing remain in the candidate.
• Dave requests participate in Yumina's deployment drain through notebook and billing settlement. Forced shutdown returns a normal recoverable error and records unheard provider usage without committing the interrupted reply. Cancellation releases the handler even if a provider ignores its signal; stopping upstream computation still depends on the provider honoring abort.
• The S3 page uploader accepts the manifest-based launch package, verifies every local file, refuses to overwrite immutable engines, and reads stored engine bytes back before publishing HTML. Dry runs do not load deployment credentials.

Measured locally

node scripts/test-local.mjs src/lib/game-npc/capacity.test.ts runs a real loopback HTTP listener and the production NPC route with synthetic identities, PGlite and a synthetic 30 ms provider. It sends both JSON and SSE requests.

Full suite sample: 512 requests, 64 concurrent clients, admission limit 8. Peak active requests: 8. Accepted: 69; rejected as overloaded: 443. Every accepted reply had a committed notebook revision. Eight subsequent recovery requests succeeded; zero leases remained. Accepted p95: 137 ms; overloaded p95: 123 ms. These figures measure this fixture on this workstation, not the real model, network or database. The final focused rerun during compilation accepted 71 and shed 441, with p95 700/685 ms respectively; it again recovered with zero leases. Workstation load affects these timings. The JSON report contains the latest run, not a benchmark of production throughput.

A first attempt to open 512 fresh TCP connections at once hit connection refusals on the desktop listener. The repeatable fixture uses 64 concurrent clients; it does not establish a production socket/backlog limit. The independent admission test also checks a 100,000-acquisition burst; that is a limiter test, not a user load test.

All 330 targeted server checks passed, including account routing, signed-in ownership checks, overload recovery, notebook races, story/action validation, save conflicts, streaming failures and asset-cache policy. Full build passed (5/5 tasks), full typecheck passed (8/8 tasks), and the four affected tests passed again after fixing test-only TypeScript types. Evidence: pvz-scale-contracts.log, pvz-scale-build.log, pvz-scale-typecheck.log, output/pvz-capacity/local-http.json.

Follow-up verification: all 336 targeted server checks passed after the shutdown fix, including six deployment-drain cases (natural, forced and an unresponsive provider, each over JSON and SSE). Full build/typecheck passed again. The uploader passed an actual loopback S3 fixture covering upload order, cache headers, repeat publication, a stored engine collision, corrupt local bytes and dry-run isolation. The real 0c135b36f2 launch package passed its dry run; no S3 writes were made. Evidence: pvz-shutdown-contracts.log, pvz-shutdown-build.log, pvz-shutdown-typecheck.log, pvz-launch-upload.log.

The open-source export also installed, typechecked, built and passed verification: 1,485 server tests passed, four skipped, zero failed. The hosted-only shutdown follow-up does not enter that export. Evidence: pvz-scale-oss.log.

Live infrastructure inspection — read-only

• First-load package: 79,861,079 bytes before transport compression and local browser caching. 100,000 fresh downloads are about 7.99 TB of payload. The art pack alone is 61,022,112 bytes; the engine WASM is 17,761,725 bytes.
• HEAD of the published art returned a five-minute edge TTL and EXPIRED at inspection. The candidate changes this narrowly for immutable packaged art.
• Cloudflare's existing /cdn/ rule permits origin-controlled caching. A game download rule already covers engine JS/WASM. HTML remains uncached; model controls are excluded from the game edge Worker. No rule changes were made.
• Railway exposes a testing environment with app and Redis services. The production app manifest specifies two US-west replicas. Regional game and ingress services exist in the US and Asia. These observations are configuration, not proof of CPU, connection, provider or relay capacity.
• No production variables, infrastructure settings or deployments were modified.

Staging acceptance before a broad launch

Use the prepared branch codex/pvz-release-20260920 and static package 0c135b36f2. Keep the existing live HTML and save slot available for rollback. Follow the public AI release order.

1. Establish an approved isolated staging target with Postgres and Redis, the same proxy/auth/streaming setup, and explicit bounded provider spending. Do not point a burst generator at the main production service.
2. Run real-account smoke checks for official and private models, key/profile changes, guest sign-in, interrupted replies, mobile return from settings and save reload. Confirm debit/usage records and account separation.
3. Calibrate a traffic model from players' actions, not total audience alone. Example for planning only: 10% of 10,000 players making one AI request per minute produces about 16.7 requests/second. At 15 seconds per request that is about 250 concurrent generations before headroom. Different model latency or AI participation changes this directly. Two default 64-slot replicas would not cover that example; use measurements to size separate AI capacity and provider quotas rather than merely raising the protective limit.
4. Ramp the authenticated mixed workload, including config/model reads, notebook visits, generation and saves. Measure p50/p95/p99, slot occupancy, overload rate, unexpected errors, memory, event-loop lag, DB pool wait and Redis latency. Proposed gate: zero duplicate charges or cross-account writes, no growing queue/memory trend, and less than 1% overload at the agreed sustained target. Provider reply latency must be reported separately from app latency.
5. Test Redis loss/recovery, provider 429/timeouts, abandoned SSE streams and a rolling deployment. Verify the real distributed limits and transaction behavior. Local in-memory fallbacks are not evidence of safe multi-replica capacity.
6. Exercise Versus connections, room creation, reconnect/resync and full matches through both regional relays at the agreed room count. Adventure HTTP results do not establish WebSocket-room capacity.
7. Check cold/warm game downloads from multiple regions and actual edge HITs for the immutable package. Confirm origin egress, download failure rates, mobile memory and first-load duration. Apply any remaining CDN changes only against a reviewed rule set, without caching account APIs or mutable HTML.
8. Promote gradually, with a canary cohort and explicit rollback thresholds; observe before increasing the audience. Keep the existing AI disable switch usable while ordinary gameplay remains available.

Hundreds of thousands of visitors is a plausible release target to engineer and measure toward. It is not signed off by the local tests above. Staging deployment, provider budget, realistic ramp/soak results and relay capacity remain open gates.