Discovery foundations: implementation and release evidence

This is the first implementation milestone of the approved recommendation program, implemented on codex/discovery-foundations and integrated with main at edcbd8f39 (initial base 04bc25007). It fixes continuity and content inputs before changing the learning objective. It is not evidence of improved user satisfaction, nor a replacement for the event-attribution and experiment work in the implementation plan.

Implemented behavior

• Clean Recommended requests can opt into cursor pagination. A ranked initial pool is replenished by a stable catalog ID keyset in batches of 200. Each request scans at most three new batches; an empty filtered batch carries a continuation instead of pretending the catalog is exhausted. The former 600-offset cap remains on legacy requests only.
• Server-side sessions remember delivered worlds and language groups, retain recent creator/topic history across pages, and keep the delivered prefix stable. Diversity is a soft cost so a narrow catalog remains reachable. This is an initial heuristic, not a learned diversity policy or a statistically selected set of weights.
• Redis compare-and-set commits one page for concurrent retries. Sessions are bound to viewer, effective language/content filters and page size. The last three pages can be retried with their original opportunity IDs/order; current exclusions can remove a card on replay. Expired/evicted/mismatched cursors produce an explicit restart response, never an invented empty catalog.
• A signed, HttpOnly first-party guest cookie binds anonymous serving sessions. It does not yet implement guest preference learning, analytics identity linking, or the future training-event contract.
• Publication, blocks, audience, language/content rules and global saved/dismissed group exclusions are rechecked against the primary DB for delivery. Taste can remain language-scoped; exclusion is global. Legacy profile cache keys were versioned to avoid using older incomplete exclusion sets. Search/tag browsing retains its existing explicit-lookup behavior.
• The client follows server continuation metadata, preserves original card positions, rejects stale responses after filter changes, blocks concurrent appends, deduplicates within/across pages, and rearms scrolling after empty pages. After three empty pages it pauses automatic scanning but keeps a manual continuation button.
• Long grids virtualize rows above 96 items using the existing React Virtual dependency. Impression IDs remain in a parent ledger across row remounts. Library state hides already fetched saved stories/groups from clean discovery.
• The embedding service and backfill share a bounded, versioned builder that uses enabled published greeting entries in engine order. Conditional writes prevent a slow embedding result from overwriting changed/unpublished content. The backfill defaults to read-only dry-run and requires an explicit input version for apply. See embedding evidence.

Verification

• The final combined recommendation, engagement, ranking, continuation, eligibility, session, embedding and related publishing/version suites: 175 passing tests, zero failures, two existing environment-dependent skips. This uses the isolated server launcher. Embedding/publishing tests include real local PostgreSQL/pgvector conditional writes with a fake provider.
• After review fixes, client pagination, async request gating, saved-group behavior, remount impression deduplication, virtual rendering and page-scroll suites: 14 passing tests. Desktop and document-scrolling mobile fixtures load 1,725 items, mount fewer than 100 cards, update after scrolling and handle scroll-owner transitions. Component tests prove that duplicate-only and locally hidden pages stop automatic fetching after three zero-addition pages while preserving manual continuation. This uses jsdom, not a real browser.
• A real Hono guest HTTP fixture traverses 713 locally seeded stories, beyond the old cap, checks response privacy, retries the same cursor, and rejects another guest's cursor. It uses the isolated local test launcher, never production credentials.
• Six route integration tests cover that traversal, discovery/auth/event rate-limit separation, actual cached profile reuse through dismiss and undo across languages, and local-edition Hub suppression. Twenty-one related continuation/eligibility/rate-limit tests also pass.
• Full generated open-source export verification passed: install, typecheck, build and all exported test suites; 3,065 tests passed, zero failed, two Redis-dependent tests skipped. The export boundary regression checks generated files, and local-edition Hub requests return 404 before hosted work runs.
• pnpm typecheck passed across all workspaces. pnpm build passed; Vite emitted existing-style bundle size and mixed static/dynamic import warnings, not compilation failures.
• pnpm --filter @yumina/server exec tsx scripts/benchmark-discovery.ts 10000: all 10,000 candidates delivered once across 417 pages; maximum two catalog scans in an observed page. After review, the fixture uses realistic UUID-length world IDs and one distinct group UUID per world: approximately 545 ms total CPU time, 2.316 ms p95 pagination-core time, 780,687 bytes final state. At 1,725 cards the corresponding state was 135,233 bytes across 72 pages. An earlier eight-character-ID/no-group fixture understated storage cost. These are synthetic CPU measurements excluding database, ranking, Redis, network and browser cost. They are not a production capacity claim.
• Production read replica, explicit BEGIN READ ONLY: dry-run validated 1,725 published embedding inputs (1,709 existing vectors, 16 missing), maximum 6,860 bytes, mean 2,657 bytes. Zero provider calls, zero stored vectors, zero input failures. Observed duration about 11.5 seconds. No production settings, data or deployments changed.

The first direct test invocation lacked a synthetic auth secret; the next invocation used bare PGlite imports without the repo's shutdown/bootstrap harness. Those environment failures were resolved using test-only configuration for pure tests and the repository's isolated launcher for integration tests. No production secret was copied into the worktree.

Independent review identified five integration issues, all corrected and independently re-reviewed: mobile scroll ownership, the OSS export boundary, auth/discovery IP rate-limit sharing, dismissal profile-cache invalidation, and automatic scanning after duplicate/hidden-only pages. See initial review, fix evidence and scoped re-review.

Upstream integration

The branch rebased cleanly onto edcbd8f39, preserving the two intervening model-picker and Krew authentication changes. Hosted pnpm typecheck (eight tasks), pnpm build (five tasks), all 14 focused app tests and all six discovery HTTP/profile/local-edition tests passed again on the combined tree. The generated-export boundary regression also passed. Full generated OSS verification then passed again (exit 0), including uncached typecheck, build and all exported tests; 3,065 tests passed (47 shared, 985 engine, 737 app and 1,296 server), zero failed and two Redis-dependent tests skipped. The initial review/fix reports retain their original pre-integration evidence; this section records the final combined-tree checks.

Rollout and rollback

The master switch DISCOVERY_CURSOR_ENABLED defaults off. The subsequent controlled-rollout change also requires an explicit cohort percentage (default zero) or authenticated operator allowlist for fresh visits. The new client requests cursors only on clean Recommended; it accepts legacy offset responses for fresh requests outside the cohort. A continuation failure returns a retryable 503; turning the master switch off returns an explicit restart instead of silently jumping into legacy offset ranking. Production requires shared Redis: an outage does not fork sessions into replica-local memory. Development uses a bounded 100-session memory store.

The feature switch rolls back the transport. Embedding extraction, global exclusions and frontend rendering are code changes; reverting the release rolls those back. Do not assume the switch reverts the entire patch.

Before enabling production traffic:

1. Deploy the reviewed branch with the cursor flag off. Confirm Redis connectivity/eviction policy and primary DB headroom. There are no schema migrations in this milestone.
2. Validate cursor requests in a staging environment for anonymous and signed-in sessions, content-mode changes, login transitions, saves, dismissals and unpublication. A login changes cursor ownership and requires a clean refresh.
3. Capture real HTTP latency, source-query latency, primary CPU, Redis memory/evictions, error rate and empty-continuation rate under representative concurrent load. The client test and synthetic benchmark do not replace this gate. The initial guest ranking currently reuses existing graph/stat/model caches but does not cache whole cursor responses; removal of shared-page CDN caching changes origin load.
4. Use a deliberately limited deployment/cohort for initial exposure; the current flag is a deployment switch, not the future randomized experiment assignment. Monitor hub_serve with policy_version=discovery-cursor-v1, duration_ms, catalog_scans and has_more. Deduplicate analytical opportunities by request/world/actor; a retried response is not a new opportunity.
5. Roll back with the cursor flag off if eligibility, continuation, error rate or latency regresses. Old cursors expire naturally after one hour. Sessions retain only three replayable pages; older retries explicitly restart.

Embedding migration is separate from the deploy. With the intended database selected explicitly, inspect:

pnpm --filter @yumina/server exec tsx scripts/embed-worlds.ts --dry-run --all --input-version published-greetings-v1

Then, for an authorized apply, use --apply in place of --dry-run. Existing vectors need --all; this release logs input hashes but does not persist a per-vector input version. Preserve the dry-run/apply JSONL for comparison. The service keeps existing vectors if refreshing fails.

Work still needed in the program

The current trainer, leaked historical outcome definitions and nightly publication gate are unchanged. Do not interpret the current model's offline NDCG as proof of discovery uplift. The next milestone needs event occurrence IDs/time, actor/opportunity identity across login, authoritative successful play/save outcomes, serving-time feature/policy/model snapshots, selected ClickHouse mirrors and deduplicated active-day labels. Only then train with matured post-exposure outcomes and an untouched chronological test set.

Multi-interest user representations, semantic negatives, learned user-story interactions, structured semantic enrichment, controlled newcomer exploration and user-randomized satisfaction/qualified-discovery experiments remain subsequent releases. D7 evaluation requires real follow-up time.

Within a session this milestone does not recycle delivered stories. True exhaustion remains finite. The approved cooldown resurfacing policy requires durable actor-level exposure history from the measurement milestone; it is not implemented by resetting a cursor or looping the same pool. New rows inserted before a session's keyset position are picked up on a fresh visit. Unserved queued ordering is stable; recent intent affects newly replenished ranked batches. A later change can rerank the undelivered queue once feature snapshots and replay evaluation are ready.

Redis state grows linearly with cards/groups delivered during the one-hour session. The realistic 10,000-card measurement makes that cost explicit; monitor real session count/depth before expanding. The Redis compare-and-set currently parses the prior JSON and transfers the full updated state. Repeated first-page requests create separate sessions; the development store's 100-session bound does not apply to shared Redis. The initial IP limit alone could permit substantial aggregate session memory, so repeated-start admission, UUID/group serialization and concurrent actor load must be part of the pre-enable gate. Consider a smaller atomic revision record plus separate exposure sets or actor-level session admission based on those measurements. There is currently no evidence justifying a new vector database, Kafka/Flink or a GPU fleet. The user has budget flexibility and no launch deadline; purchasing infrastructure remains tied to measured need.