Complete hosted deployment validation

Scope and fixed inputs

This followup exercises the real production Dockerfile and hosted server entrypoint through Railway's public HTTPS proxy. Two separate owned app services share an owned PostgreSQL database, a new dedicated Redis service, and a fresh authentication secret. Responses identify the exact test instance, so cross-app coverage does not depend on random load balancing.

The first application revision is a70a3aa87e3ed8c355978708e8ced087f2a2a74e. Its complete CI passed separately; see candidate validation.

The application source and container command are unchanged. A test-only Node preload verifies the owned database/Redis identity, denies external HTTP and non-owned database/socket connections, limits HTTP traffic, identifies instances, and stops the process after ten minutes. Application modules and startup timers are not mocked. The controller also bounds deployment time and cleans up on failure. Tests use HTTP and module execution, not a browser.

The database contains 713 eligible synthetic English story families, one Spanish sibling, five ineligible controls, and three synthetic accounts. It contains no copied production users, conversations, or catalog. The database role owns only this database's application objects, has no cluster privileges, and permits the unchanged delayed schema initialization. Redis is standalone, password-protected, 128 MiB, and configured with noeviction. Both app pools are capped at eight; Discover session caps are 24 overall, eight per account and three per guest. Experimental ranking, onboarding and analytics readiness remain disabled.

First attempt: packaging failure

Run efb1b57898585c6e2ecbe55636a19b64 failed at container build, before serving traffic. The test packager classified the tracked application module packages/server/src/lib/env.ts as a secret configuration file and excluded it. This was a harness error, not an application compile failure.

The packager now explicitly admits that source module and verifies relative imports; 12 local builder regressions pass. Frozen context03 differs from context02 only by the added module. All three temporary services, the owned database and role from the first attempt were removed and independently checked absent, without forced database disconnections.

Second attempt: deployed behavior and clock failure

Run 9af662afa8037b291c2f4b3ef6fe6e50 used context03, whose manifest SHA256 is 54ab6a74ec68616c2a47ceabd68b164c81f7070b6d5d0be201f92d1f958e6b13. The context has 2,435 payload files totaling 70,687,639 bytes, plus its manifest.

Both images built and deployed successfully. The first four phases passed:

• Actual static HTML and JavaScript served through both public HTTPS endpoints.
• Both servers stayed healthy for at least 105 seconds; real Redis connections, tokenizer and recommendation warming ran. Delayed schema initialization completed, including the scheduled SQL functions.
• Stored Better Auth sessions worked on both instances; a nonexistent token was rejected. Account reads were private/non-cacheable, and foreign-origin CORS did not grant access.
• Simultaneous requests for the same cursor committed an identical page on both instances. Retries preserved the page/receipt, another guest's cursor was rejected, and alternating instances traversed all 713 families in 15 pages, with no duplicates or omissions.

The run failed after 51 HTTP responses while checking canonical event counts: the expected impression row was absent. The HTTP event endpoint returned 204, which alone does not prove that an observation was accepted. Later save-family traversal, account-isolation and service-removal phases did not run.

Investigation found the client uses its device wall clock for occurredAt, while the receiver rejects observations more than five seconds in the future or before the signed exposure. A read-only clock comparison measured this desktop roughly 20.9 seconds ahead of the server. The sent event therefore fails the receiver's future-time rule. This is also a real client-data risk for ordinary devices with clock skew; adjusting only the test timestamp would conceal it.

All owned resources from attempt two were removed and independently checked absent, with no forced disconnections. No production feature flag or application deployment changed. The client timing fix is now implemented; a fresh deployed validation is required before this report can claim a complete pass.

Clock correction

Accepted canonical feed responses register their fresh HTTP Date and receipt exposure time against a monotonic browser clock. Events then use elapsed time from that anchor, with exposure/dwell floors to handle HTTP second precision. An old cursor receipt does not reset the clock to its old exposure time. Queue age and retry backoff also use monotonic time, and already-queued event identity, timestamp and receipt remain byte-identical across retries. Credentialed CORS responses expose Date; server receipt validation and expiry are unchanged.

The actual receiver reproduced rejection with clocks 20,927 ms ahead and 20,000 ms behind. After correction, 54 client regressions and five server receipt tests passed, along with app/server typechecks and the complete five-task build. Independent review passed 27 focused tests and the second-resolution preview dwell edge case. Missing/invalid Date samples preserve a prior good anchor; a runtime without any good anchor retains the historical local-clock fallback.

Third attempt: complete pass

Run f1663059960218b89d45d1781b0fab73 tested committed revision e5322c710faf7bb1f52489fd0c7b91a1ba858cbe in fresh owned resources. Context04's manifest SHA256 is 0592d1ddf9636faa9855812792d2f68b187d9ad04a440baa8bb1520781f85121. The result SHA256 is 4b84ba167eebf9f52e44e0e3125142f98f5c228f3936d72e4c30fe6505e4ec56.

All seven phases and 73 HTTP checks passed, including the previously passing startup/auth/static/cursor checks and the remaining cases:

• The actual client beacon source was bundled from the same revision, without changing its event logic. It generated an impression and click under injected device clocks 20 seconds ahead and 20 seconds behind the sampled server time. Both persisted. Retrying the same impression against the other instance kept one canonical row. The client bundle SHA256 is 413d3045b8c242df303365fce4f2ab4f0ece39776354fbbdf0fd76314638ac4c.
• Duplicate attributed saves across the two instances produced one confirmed save and one verified guest-to-account link. Saving the Spanish sibling also excluded its English family. The account then traversed exactly 711 remaining families in 15 alternating-instance pages, without omissions or duplicates.
• Another account could not reuse the cursor.
• After deleting the original app service, its committed cursor replayed the same page on the surviving instance and continued to a new, nonduplicate page.

The controller removed all owned resources. A separate post-run audit confirmed all nine temporary services, three databases and three roles from the three attempts absent. No forced database disconnections were used. These are bounded functional results; they do not establish a production traffic capacity ceiling.

Complete CI also passed on that same code revision: hosted build/typecheck, 776 server tests with eight explicit service-dependent skips, 24 Redis capacity tests, 102 client tests, 185 Python dataset/evaluation tests, and nine separate live ClickHouse tests. Boundary checks, GitGuardian, Cloudflare Pages and the full OSS export workflow passed. The successful code revision is e5322c710; subsequent report-only changes do not alter application code or the tested containers.

Limits

These are correctness checks on synthetic testing PostgreSQL, not production Neon capacity measurements or evidence of ranking uplift. Explicitly forwarded test cookies exercise shared authentication across two owned domains; this does not test browser cookie-domain behavior. No visual browser checks occurred.

The fresh fixture initially lacked a scheduled recovery function; its first background invocation logged that absence, and the real delayed schema installer subsequently created it. Model-popularity HTTP refreshes were denied by the test egress guard, as intended. Thus this test does not validate external provider availability or production background-workload cost.

The real ClickPipe was rechecked Running with 17 mappings; it still lacks discovery_events and discovery_erased_actors. CI warehouse tests and source schema preparation do not replace a real replication/erasure receipt. Experimental representation quality also remains below its promotion gate.