Discovery rollout and model-promotion gates

Measurement PR #177 builds on foundation PR #172. It adds trustworthy observations and evaluation tools; it does not publish a new trained model or establish causal improvement. The initial implementation changed no production configuration. The subsequent production preparation report records the completed source schema, history backfill and warehouse installation; feed flags remain off. Verification evidence is maintained below and in the linked component reports.

Implementation verification

Hosted typecheck/build, the 355-test affected server selection, 55 affected client tests, 86 Python tests, edition checks and the generated OSS boundary passed. The generated OSS tree installed, typechecked and built; its final serial test run passed 3,078 tests, zero failures, two service-dependent skips (47 shared, 985 engine, 1,296 server, 750 app). Serial execution avoids an existing Windows concurrent-build file-lock race.

The broader hosted suites still report two server and eight app failures. Independent baseline probes establish that all ten predate the discovery foundation. They are documented without claiming those full suites are green. See verification and baseline evidence for commands, scope and the corrected nondeterministic test fixture.

What is ready to measure

Cursor pages carry signed, actor-bound receipts. Committed opportunities retain ranking and delivery inputs, original story/group identity and versions through retries. Visible impressions, clicks, dwell and intent are distinct from saves, new sessions, dismissals and undo confirmed in the actual mutation transaction. Empty pages have their own canonical observation. Reading intervals identify the actual owned session, database receipt time and consumer eligibility.

The durable first-known ledger records saves and consumer starts even when a Library entry or chat is later removed. A separate bounded backfill imports surviving evidence; it cannot recover already-deleted history. Accounts predating continuous capture therefore have unknown novelty when no positive evidence exists. Anonymous history is also unknown. Report these cohorts separately; do not convert their missing history into eligible new-discovery negatives or successes.

Deployment order

1. Integrate and verify the foundation and this branch, preserving the legacy feed as the rollback path. Begin with DISCOVERY_CURSOR_ENABLED=false and DISCOVERY_MEASUREMENT_ENABLED=false.
2. Apply packages/server/scripts/discovery-measurement.sql to the explicitly selected source database. It is additive and transactional, with lock and statement timeouts (250 ms locks, 2 s statements). This installs event/erasure tables, the first-known ledger and capture triggers, and verified interval columns without scanning historical Library/session rows. Run the explicit bounded backfill below and reconcile both sources before enabling measurement. Do not substitute a Drizzle schema push: triggers and the history coverage marker are required.
3. Apply scripts/discovery-world-history.sql; run its bounded baseline function and reconcile every current world. Capture begins at observation time. Never backdate a present translation relationship to world creation.
4. Extend the existing publication using packages/server/src/analytics/migrations/source-discovery.sql. Prepare the replication role's explicit column permissions and ClickPipes mappings for the new event/erasure tables and interval columns. Preserve other members.
5. Apply warehouse migration 008_discovery_events.sql using the explicit analytics migration command. Check real ClickHouse execution, replacement, erasure, retention and source-to-archive reconciliation in isolation before setting ANALYTICS_DISCOVERY_READY=true.
6. Route discovery to a dedicated Redis service via DISCOVERY_REDIS_URL when material traffic is enabled. Its absence uses the existing Redis. Configure and verify standalone Redis with noeviction; arbitrary eviction would break the admission indexes and cursor guarantees. Select caps from measured memory and concurrency, not the defaults alone.
7. On an isolated deployment, explicitly set both DISCOVERY_CURSOR_ENABLED=true and DISCOVERY_CURSOR_ROLLOUT_PERCENT=100; the rollout percentage defaults to zero. For a restricted production canary, use zero percent plus DISCOVERY_CURSOR_USER_ALLOWLIST, explicit stored-visit and per-process work limits, and fixed replica count as described in the traffic and controlled-rollout report. Enable measurement only after its independent readiness gates. Verify real guest and signed-in serving, save/play/hide/undo, translated choices, failed starts, refresh, expiry, account erasure, source capture and archive coverage. Then use a controlled production release with immediate flag rollback. Turning the master cursor flag off makes continuations request an explicit restart; fresh requests use the legacy feed. Reducing only the cohort percentage lets already-owned cursors drain until their existing absolute expiry.

These are execution prerequisites, not an assertion that the deployments or live warehouse checks have happened. Existing shared testing services may have other work in progress; use an isolated deployment for application load tests.

Bounded first-known backfill

From packages/server, explicitly name the environment variable holding the selected source connection; do not place a credential in a command argument:

pnpm exec tsx scripts/backfill-discovery-known-stories.ts --database-env DATABASE_URL --source user_library --after NULL --max-batches 1
pnpm exec tsx scripts/backfill-discovery-known-stories.ts --database-env DATABASE_URL --source user_library --after NULL --max-batches 500 --apply --emit-cursor
pnpm exec tsx scripts/backfill-discovery-known-stories.ts --database-env DATABASE_URL --source play_sessions --after NULL --max-batches 500 --apply --emit-cursor

Dry-run reads only and its cursor must never advance an apply run. Each apply batch scans at most 1,000 source IDs in a separate transaction. Persist only the last acknowledged apply cursor when resuming; overlaps are idempotent and preserve the earliest evidence. Source exhaustion is not global reconciliation. Compare the surviving non-ephemeral Library/session evidence with ledger keys and first timestamps, while excluding erased accounts. Previously deleted history remains unknown. The capture-start marker denotes continuous trigger coverage, not completion of this backfill. Keep measurement off until both sources reconcile.

Capacity and rollback

The testing Redis adapter check used a distinct network endpoint from production, two clients and bounded synthetic records. All eight final checks passed using the complete 136-ranking/19-delivery-feature snapshot, and the exact 41 temporary keys were removed. Maximum single payload was 1,015,140 bytes; the pre-cleanup memory sample was 1,415,373 bytes and is not a peak measurement. This verifies adapter behavior, including fresh-request recovery at capacity. It did not run application HTTP traffic, rank database queries or establish production capacity. See the store scale report.

Before choosing production limits, measure expected peak and overload traffic: p50/p95/p99 page latency, error/retry rate, admission rejection by scope, Redis memory and bandwidth, primary database time/locks, event bytes, archive backlog and recovery. Ensure ordinary refresh/navigation does not exhaust per-actor admission. Reconcile accepted observations and confirmed mutations by immutable ID; do not use a warehouse heartbeat as proof of complete observations.

For the initial offline dataset, use the builder's declared 24-hour maximum arrival allowance. This is a release requirement, not a measured result. The archive should target one-hour freshness; before activation, prove its complete sweep plus CDC, scheduling and receipt delays remains below 24 hours at expected retained raw volume and under recovery load. The deterministic importer test needs roughly 400 minutes per million raw IDs at current defaults. Therefore these defaults cannot be extrapolated to a large retained source: increase measured catch-up throughput or adopt incremental ingestion before that gate fails. Do not conceal late data by increasing label allowances after an experiment starts. Live session adaptation uses the serving path; this archive freshness target is for offline evaluation/training.

To stop collection, disable measurement. Keep erasure hooks, ledger triggers and archive erasure enforcement active for previously collected data. To restore the legacy feed, disable cursor serving. Do not drop data tables or clear shared Redis as a rollback. Source pruning is dry-run by default, checks exact archived IDs, preserves identity links and requires an explicit apply command.

Scientific promotion criteria

Use the new dataset tooling only with reconciled source coverage and declared feature schemas. Preserve exact input exports and manifests, subject to account erasure. Require the full outcome window plus lateness allowance before assigning negatives, and use chronological splits with gaps. Inspect unknown-history, missing-snapshot, translation and interval coverage by cohort.

The first experiments should separately test session adaptation, multiple taste interests, semantic negative preferences, retrieval-source allocation and variety. Keep model/policy versions fixed within each comparison. Predeclare an actor-level primary discovery metric, guardrails, duration, minimum detectable effect and a stopping rule from observed variance. Measure anonymous discovery and verified new-story discovery separately where knownness differs. Offline served-slate metrics are diagnostics; they cannot estimate unseen-candidate recall or causal policy lift. Do not automatically promote the legacy nightly trainer on the strength of these new diagnostic reports.

Add randomized exploration with recorded assignment probabilities before making counterfactual claims. Retain an incumbent and require online outcome evidence before model promotion. D7 evidence necessarily requires real follow-up time.

Spending decision

No new vendor account is required for this release. Prioritize isolated Redis, testing capacity and warehouse retention within existing accounts. Keep Neon pgvector until representative filtered-retrieval tests demonstrate a need for a separate vector service. Add a feature store, event bus or GPU training only when defined freshness, throughput, reuse or quality experiments justify the cost. See 2026-09-19-long-term-recommendation-stack.md for researched tradeoffs.