Recommendation preparation and serving: scale verification

This release removes measured repeated work from the existing PostgreSQL, ClickHouse and Redis stack. It preserves the 50% personalization assignment, ranking weights, minimum training support, outcome maturity and promotion gates. Infrastructure benchmarks do not establish recommendation-quality uplift.

Changes

• ClickHouse computes eight compact training metadata fields when canonical events are inserted. Migration 010 materializes historical rows and updates the existing erasure-filtered view. ID-only replacement, raw payloads, source versions, retention and erasure predicates are unchanged.
• The trainer verifies source files into private temporary storage before streaming JSON. Shared files count once against dataset budgets and are reverified when reread. Value training retains its 29 validated model inputs; full feature and vintage validation still happens before projection.
• Canonical session ownership uses indexed compact evidence instead of repeated scans. Evidence available after an opportunity matures cannot retroactively change that opportunity's label; conflicts known by maturity still reject it. Availability respects both event time and receipt time, including clock skew.
• ALS computes exact neighbors in bounded blocks and spools them before an atomic, batched publication. User and world factors identify the same publication. Serving rejects mismatched generations. Account erasure is protected by publication-time checks and an explicitly installed cascade FK. The publisher locks both previous and upcoming account rows in account-deletion order before changing child records. A real two-connection PostgreSQL test reproduces the former deadlock and verifies both execution orders after the fix.
• Committed cursor retries avoid unnecessary ranking/history loads. Public scoring snapshots coalesce concurrent loads within a process, use bounded last-good fallback, and share immutable values. Eligibility stays live.
• Legacy feed cache hits revalidate all cards against primary-database access, publication and Library exclusions. Invalid pages are recomputed in full. Expiring generation keys prevent late cache writes from undoing invalidation. Recommended responses bypass browser/CDN caches to retain these checks.
• Play-depth counting stops at the existing 100-message saturation point, preserving every scoring tier while bounding matching rows read.

Measurements

All load-test records were synthetic and isolated from consumer measurement. Production comparisons were read-only and emitted aggregate counts/hashes.

Component	Before	After	Scope
Production training-scope query, 586,484 rows	2.244 GB read; 1.97 s	126.4 MB read; 0.41 s	Same selection and visible-event hashes
Synthetic scope, 600,000 wide events	2.250 GB read; 250 MiB peak	126.2 MB read; 150 MiB peak	Real FINAL views and erasure joins, 256 MiB fixture limit
Dataset reader, 10,000 shaped examples	248.9 MB traced allocation	76.9 MB traced allocation	Identical model-input/label SHA and eligibility; about 69% less
Canonical ownership, 4,000 intervals	2,762 ms; 112.72 M SQLite steps	117 ms; 0.764 M steps	Three-run medians; valid-input hashes unchanged
Exact neighbors, 16,000 worlds × 64 factors	Dense similarity matrix alone: 976.6 MiB	1.75 MiB maximum block scratch; 42.3 MiB process peak	About 11 s; exact top 30, deterministic ties

The real LightGBM path also completed with 60,000 training, 4,000 validation and 8,000 test examples in 11.7 seconds at approximately 620 MB process peak. This used synthetic repeated patterns (one fitted tree), fake read-only model access and publish=False; it is neither a real quality result nor a worst-case fit. It excludes canonical preparation and native ALS fitting.

The production exporter retains its 512 MiB query limit and one thread. The ClickHouse fixture's smaller blocks apply only to its narrower test role. Timings are observations, not cold-cache or service-wide throughput guarantees.

Rollout and recovery

Migration 010 was applied before typed readers. All temporary maintenance privileges were revoked, and the normal ClickHouse connector remained read-only. Old readers are compatible with the additive columns, so code rollback does not require dropping data. Run the dedicated factor-erasure installer with an explicit target connection; never use a broad schema push for this change.

Validation covers real ClickHouse migration/replay/erasure behavior, checksummed reader corruption and cleanup, temporal canonical labels, exact neighbor equivalence, atomic publication, feed continuation/retry, cache invalidation and live card eligibility. Actual production trainer execution is checked separately from Railway image-build success.

Local validation passed 3,846 server tests (15 explicit skips), all monorepo typechecks, 264 Python tests, 21 job/publication tests and the production build. The isolated ClickHouse fixture passed all six migration/scale tests. Real development PostgreSQL passed six concurrency tests, including the intentionally failing old lock-order control, and removed its exclusively owned test schema afterward.

The production cascade FK was installed and validated with all 26,667 existing user vectors preserved and zero orphan rows. Failed trainer exits retain independently sanitized ALS/value outcomes before temporary files are cleaned; a partial success cannot be reported as a successful whole job.

Production execution

Release ac85e21b067bda5ca7f5488f2e87ca15f5dfe12f deployed successfully to the app, analytics worker, representation worker and trainer. The trainer's actual invocation on deployment a8e4bd11-1c60-471d-bb8c-b8a2c99c0e57 completed in 20.3 seconds with exit 0, preparation complete and temporary-data cleanup complete. ALS fitted 100,588 interaction edges and published 26,689 user vectors, 1,289 world vectors and 38,670 neighbor pairs atomically. PostgreSQL confirmed a single matching generation on both sides and zero orphaned or erased account vectors afterward.

The value model reported insufficient_training_support with zero eligible training/validation rows in the matured training windows and did not publish. The existing ranker remained model 32. Personalization rollout remained 50%, cursor rollout 100%, and measurement enabled. A public HTTP smoke check returned three full pages with 30 distinct stories; retrying page two preserved its order. These few guest requests did not generate synthetic playtime or authenticated training outcomes.

Remaining bounds and quality evidence

Full projected training rows and the fitted interaction graph still occupy memory. Exact neighbor comparison still requires quadratic computation even though its scratch memory is bounded. Full catalog snapshots and context maps still grow with the catalog. Cache coalescing is process-local, not a cross-replica refresh lease. These measurements do not establish unlimited capacity or a 100× whole-service traffic claim.

The canonical value trainer remains gated until there are enough eligible, matured training and holdout examples. A successful warmup or ALS publication must not be described as a new value model winning the 50/50 experiment. Cold-start policy, session recall and additional model features require their own versioned evaluation rather than an incidental change in this release.

Reproducible component commands and operator prerequisites are in the trainer README.