Relay code and network-path comparison, 2026-09-11

Status: network-path mitigation under controlled rollout. Broad campaign still requires real-player session and staged capacity validation; the exact failing provider network component is not established.

Question

Would returning to the single-server relay from before the September optimizations eliminate the current WebSocket disconnects?

Testing rt.yumina.io alone cannot answer this: that endpoint now runs newer code. This experiment restores the earlier implementation in an isolated service and compares it with the current implementation under identical conditions.

Controlled deployment

• Baseline: 421ff3d55700caec0a234172a46837d2574a6efd, before the September 6 relay hardening. Its relay implementation was last changed August 29.
• Current: e15336380.
• Third control: a small two-seat WebSocket forwarder with the same 10-second ping interval, without Yumina game logic.
• One Singapore Railway instance, Node 22.23.2, ws 8.21.3, same process, separate /old, /current, and /minimal endpoints.
• Deployment: 23e07d04-995b-49ca-aac4-80d15522c87f.
• Boot: 685c679a-d608-40fd-8f1e-6057f60d3182.
• Test clients send input rounds every 50 ms, plus a peer RTT exchange every 100 rounds. Each test stops on its first unexpected close or after 900 seconds.
• Authentication uses a newly generated test-only secret and isolated signed tickets. The service has no production account keys, database, Redis, AI, analytics sink, room authority or lifecycle reporter.
• No production room directory entries or player routes point at this service.
• A one-hour process deadline and NEVER restart policy bound abandoned tests.

The current-code variant uses legacy ticket admission. This isolates the shared relay changes; it is not a full test of regional admission or the C++ simulation. The baseline's original Node/dependency deployment is not restored: holding the runtime constant isolates the relay-code variable.

First observed failure

At 10:38:50.87–10:38:50.88 UTC, all six public sockets from the Singapore probe closed with code 1006 across the baseline, current and minimal variants. They had received game inputs about 50 ms earlier. The application process did not restart. Event-loop delay in the client process remained below 23 ms.

The desktop public probe also lost one current-code socket at 10:38:53.406 UTC; its last received input was 205 ms earlier. This isolated failure by itself cannot distinguish implementations; the simultaneous old/minimal failures are the useful control.

The server detected the already-broken sockets later, at about 10:39:03 UTC. heartbeat-timeout in these later logs is detection, not evidence of the cause of the earlier client-side close. All affected server buffers were empty.

Relevant Railway request IDs:

Variant	Request IDs
Old	k330kbR6RXmxoiKUDcO5xA, lWRfHs9AR2K2dQCLipRofQ
Current	3xK4OA0bTJmkyAk60ubPiw, QiiZTfHiR4yCgk7tVOLIQQ
Minimal	Girdk5F-TuCzHmP0mrpb1w, npAes7OuQAC0e9zXoB_USg
Desktop current	szKT47CYReKyMf_zV7rehQ

Railway network flow records include six ingress NO_SOCKET records to the Singapore probe's ephemeral client ports at the same time. These may reflect packets arriving after socket closure; they do not identify who initiated it.

Independent path control

The same service is also tested through:

1. Loopback, using its actual Railway PORT=8080.
2. A Cloudflare Tunnel directly to loopback, bypassing Railway's public HTTP proxy. Connector 2026.9.0, HTTP/2 transport, Singapore Cloudflare connection.
3. A repeated Railway public run, overlapping the alternate route.

An initial loopback attempt used port 8300 and failed before opening any socket; it is excluded. The corrected loopback run is the relevant control.

The Cloudflare route is an authenticated diagnostic Quick Tunnel, not a production endpoint. Cloudflare documents Quick Tunnels as testing-only. TLS verification remains enabled. The Node container's bundled trusted root certificates were provided to the connector because the slim image did not provide the OS certificate bundle.

Completed 15-minute controls

Source and path	Old relay	Current relay	Minimal relay
Singapore, Railway public, first run	1006 at ~9 s	1006 at ~9 s	1006 at ~9 s
Singapore, Railway public, repeat	Completed	Completed	1006 at ~620 s
Singapore, loopback	Completed	Completed	Completed
Singapore, Cloudflare HTTP/2 tunnel	Completed	Completed	Completed
Desktop, Railway public	Completed	1006 at ~8 s	Completed
Desktop, Cloudflare HTTP/2 tunnel	Completed	Completed	Completed

"Completed" means no unexpected socket close, not zero latency spikes. For example, the desktop Cloudflare minimal-control run had a 2.189-second delivery gap. The loopback controls completed about 17,961 input rounds with a maximum peer RTT of 2 ms. The initial simultaneous closes preceded server heartbeat detection and did not coincide with an application restart.

Separate ingress services were provisioned in this Railway project during the repeat run. The A/B service was not redeployed, but this is not a claim that the entire project configuration was unchanged. The network records cannot establish whether project-level network propagation contributed to any failure.

Real regional-host canaries

Two unadvertised named tunnels were provisioned, games-us.yumina.io and games-asia.yumina.io. Each has its own supervised connector in the matching region, reaching the existing game process over private networking. Neither changes the room owner, game build, account credentials, or DB schema.

Live browser tests on both routes passed real guest admission, separate seats, matching seeds, English desktop/Chinese phone drafting, deliberate reconnect, missing-pick replay, and duplicate suppression. Resume required no new directory or account call. Their test rooms were cancelled. These browser canaries rewrote the WebSocket URL only in their isolated contexts; public player routing was unchanged.

A concurrent 15-minute test against both actual regional relays began at 10:54 UTC from Singapore and desktop, through both original public addresses and named tunnels. At about 11:02 UTC the desktop US tunnel route experienced delivery gaps of 6.957 and 10.214 seconds without closing. The simultaneous desktop US public and Singapore US tunnel connections did not show that gap. Consequently the HTTP/2 US tunnel has not passed the player experience gate, despite no disconnect: keeping a WebSocket open is insufficient for lockstep. There were no connector error logs in that interval. This does not establish the exact path segment or protocol as the cause.

Interruption and Wi-Fi evidence

The desktop production comparison subsequently lost all eight sockets at 11:07:29 UTC. Windows WLAN-AutoConfig event 8003 records Wi-Fi disconnection at 11:07:11.844 UTC (04:07:11 local), matching their last-input times. Exclude these final desktop closes from claims about either hosting route. All four cleanup requests also failed because local networking was unavailable.

The actual C++ Asia browser canary had completed before this outage: 180 seconds after a deliberate reconnect, no errors, and 94 common matching state-hash checkpoints through round 3720. It exercised an automatically drafted, otherwise idle match, not a complete human-played battle or a capacity test.

Agent work stopped at the interruption. QUIC configuration was prepared locally but not deployed; the endpoint transition commit 4aeadbc62 was not pushed. At 21:36 UTC both real regional relays still had their original boot IDs, valid leases, and zero rooms; production configuration still advertised only Railway public addresses. There was no overnight player-route cutover.

The separate Singapore production probe's last retrieved report covered 12 minutes without socket loss; its final file was not recovered before the one-hour diagnostic service deadline. Do not claim a completed 15-minute result for that run. Earlier completed A/B results were downloaded and are retained. Subsequent remote probes must log directly to Railway deployment logs so an operator Wi-Fi outage cannot strand the final evidence in ephemeral storage.

Resumed comparison with independent clients

At 21:37 UTC the unadvertised named connectors were changed to QUIC, with four registered connections each and successful UDP prechecks. The underlying US and Asia game hosts were not restarted. This changes the connector-to-edge transport; browser WebSockets remain TCP.

Two bounded probe containers then compared both regional public addresses and both named tunnels concurrently, one client in Singapore and one in US West. Their stdout goes directly to retained Railway logs. Neither uses the desktop network. Deployments:

• Singapore: c93e2384-b3aa-4449-a3e7-5854dc35712b.
• US West: b833ebc4-a4ab-4340-9478-fef0dd1e5911.
• US QUIC connector: 766b8ff0-6c7a-41ec-b50e-ef7ce70a3a74.
• Asia QUIC connector: 56035f31-575b-4a35-a6e1-b923d85b6e16.

At 21:43:09.930 UTC, the US probe lost its Asia public connection with code 1006 after approximately 276 seconds, room RRVWM, side 1. Its last received input was only 216 ms earlier. The same client's simultaneous Asia QUIC stream remained connected. This is a fresh reproduction independent of local Wi-Fi.

The upstream Railway request ID was jbbCyzGcSduKktBR0ubPiw. The server later reported room-revoked at 21:43:22.146 UTC, after the probe had detected failure and cancelled its synthetic room. That later cleanup is not the initiating cause. The other seat closed normally during cleanup. The game host retained boot 55d1f606-151c-4e30-9aa8-efb1cfb99bb5 and had zero buffered bytes.

The US QUIC browser check also exercised actual plant placement and phone-touch zombie placement, then 180 seconds after deliberate reconnect, with no reported errors and matching simulation checkpoints. The Asia draft replay check passed again through QUIC. These do not replace physical-device WAN or load testing.

Completed resumed results

Both independent clients completed their bounded runs by 21:54 UTC. All four QUIC paths completed 15 minutes with no unexpected close and no input gap above one second. The longest gap was 564 ms on Singapore-to-US traffic; do not describe the result as zero jitter. The other three original public routes completed; US-to-Asia public failed as detailed above. Traffic was a few synthetic rooms, not a capacity benchmark.

Probe client -> relay over QUIC	Input rounds	Largest delivery gap	Median peer RTT
Singapore -> Singapore	17,948	263 ms	13–14 ms
Singapore -> US West	17,947	564 ms	344 ms
US West -> US West	17,960	86 ms	13 ms
US West -> Singapore	17,961	324 ms	367 ms

These are server-probe timings, not predicted household/mobile latency. The US C++ browser match had 98 common matching checkpoints after gameplay actions and deliberate reconnect, through round 3880. Intentional room cancellation at the end closes the test sockets; those cleanup events are excluded from failures.

Both temporary probe services were deleted after downloading final logs. The real regional relays and named tunnel connectors remain; no live game process was restarted during this resumed comparison.

Route rollout

Commit 56d8ba656 adds validated alternate routes for a single owner. Commit 71d85da03 records the connector and evidence. The compatibility deployment 8eb2093d-01b0-4bb2-8c06-d63e858823b7 advertised both addresses while keeping the original assignment endpoints. Its previous API deployment was gone by 21:51:23 UTC. The 60-second client cache window elapsed before the preferred endpoints were changed at 21:54 UTC.

New preferred addresses are wss://games-us.yumina.io/relay and wss://games-asia.yumina.io/relay. Each reaches the same private game process as its old public URL. Host IDs, boot IDs, room generations, tickets, and saved matches are not migrated. Old addresses stay available for existing connections and cached resume capabilities, and remain permitted for rollback.

This is a mitigation supported by a controlled comparison. It is not proof that all historic player incidents had this cause, or that QUIC alone explains every difference from the earlier HTTP/2 test.

The preference deployment f0bd422e-b595-41bc-b800-e8275839010f completed by 21:56 UTC. Final unmodified public-page routing checks passed in both regions: US room R4T73 and Asia room RDY5T. Live admission returned the named-tunnel URL to both participants, native drafting worked, and forced reconnect replayed a deliberately lost pick exactly once without obtaining a new room or seat. Both tests exited successfully and cancelled their rooms. Only synthetic test input loss and the selected test region were controlled; no WebSocket URL or configuration response was rewritten in these final checks.

The final build/typecheck and 12 host configuration/room directory tests passed. No PvZ assets or game binary were changed for this route mitigation. Temporary probe services and local credential files were removed; raw diagnostic logs remain in the ignored evidence directory. The original public relay addresses are intentionally still reachable for existing sessions and rollback.

Before a campaign, validate complete real-player matches on the intended network routes, including same-account devices and guest/account pairs, then increase measured concurrent rooms in stages. Fifteen-minute synthetic stability and the configured room cap do not establish launch capacity.

Interpretation and decision gates

The reproduced public-path failure does not require the regional directory, database, identity system or September relay-code changes. An untested full rollback would reintroduce fixed security/room bugs without a demonstrated transport benefit. This does not establish that every player complaint has the same cause, nor prove that the earlier full deployment was unaffected.

Keep one owner and one simulation history per room. If an alternate ingress passes the comparison, validate a production-grade named tunnel or a separately hosted relay as a canary before changing new-room routing. A replacement route must reach the same room owner, not another replica with empty memory.

For Cloudflare, use a named tunnel and an independently supervised connector in each game's region. Route only the necessary game and health paths to that region's private host address. Preserve signed admission, resume capabilities, capacity limits and existing room endpoints. Merely enabling Cloudflare's DNS proxy in front of Railway would still traverse Railway's public ingress.

Before campaign rollout: completed long WAN matches, deliberate reconnects with matching state hashes, mixed account/guest pairs, and staged load with measured admission, disconnect and match-completion rates. Local throughput tests and short HTTP health responses cannot satisfy the WAN stability gate.

Evidence location and references

The reproducible source snapshots, build/probe scripts and raw JSONL are in the ignored output/relay-ab-20260911/ directory of the recovery worktree. Test credentials must not be committed or included in provider reports.

• Railway public networking specifications explicitly exempt WebSockets from HTTP duration/inactivity limits. The observed cuts are not an expected documented five-minute WebSocket timeout.
• Cloudflare Tunnel setup distinguishes temporary Quick Tunnels from named production tunnels.
• Existing incident evidence and a provider-report draft are in the PvZ repo's docs/RAILWAY-RELAY-TRANSPORT-2026-09-11.md. The report has not been sent.