Yumina recommendation audit — September 18, 2026
Investigation and discussion proposal. No application, model, configuration, or production-data changes were made. Measurements were taken late September 18 Pacific / September 19 UTC. Rolling windows have slightly different endpoints because queries ran sequentially.
Product requirement confirmed with Jefray: Discover finds new stories for anonymous visitors and signed-in users. Library handles returning to saved stories. The reported problems are premature scroll exhaustion, repetitive ordering, and insufficient variety/personal relevance. Preserve the single discovery grid; a shelf redesign is not needed to solve these problems.
Assessment: The system has a substantial working foundation. Its main weaknesses are incomplete candidate coverage, guest identity, incorrectly attributed training outcomes, limited learned personalization, and weak exploration. A new vector database would not fix these. Improve the existing pipeline, then compare more advanced models against a trustworthy baseline.
Evidence and access. GitHub authentication, production Railway service/configuration access, the production Neon read replica, ClickHouse, and PostHog all worked. PostgreSQL queries ran inside read-only transactions with timeouts. Only aggregate behavioral data, schemas, model metadata, and code were inspected; private chat text was unnecessary. The inspected production deployment was 04bc2500797ac6603ca12335b3a253497764be2e. The local checkout had unrelated existing modifications and was behind production; the core recommendation, engagement, embedding, and trainer files were compared with production and matched. Relevant differences in surrounding files were inspected. No browser was opened.
1. What is actually running
The August investigation is historical context, not an accurate inventory of today's missing features. LightGBM, collaborative filtering, first-party feed logs, experiments, dismissal, and automatic scroll loading have since been implemented.
| Component | Current implementation / measured state |
|---|---|
| World semantics | OpenAI text-embedding-3-small, 1,536 dimensions, PostgreSQL pgvector/HNSW. Refresh hooks on approval and certain publication/update paths. |
| Collaborative representations | Nightly 64-dimensional ALS user/world factors plus item-item neighbors. |
| Learned ranking | Nightly LightGBM regression over 29 features; predicts log-transformed playtime plus a favorite bonus. Pure TypeScript inference. |
| Other ranking | Hand-set tag, creator, lineage, content-vector, craft, popularity, freshness, session, and collaborative terms. |
| Experiment allocation | With a model: signed-in users assigned 40% heuristic engagement, 40% learned engagement, 20% control. Guests use heuristic engagement. |
| Published catalog | 1,725 world rows across languages; 1,709 have content embeddings. Language variants are included in these counts. |
| Latest daily job | 2,936,137 training/evaluation impressions over 21 days; 94,398 user-world edges; 35,550 similarity pairs; 1,185 world and 25,440 user latent vectors. |
| Latest published model | Model 30, 300 trees. Reported NDCG@24: 0.64552 vs served-order 0.56308. These are not reliable evidence of online uplift for reasons below. |
| Existing warehouse | ClickHouse has fresh gameplay/activity/interval data, but the recommender trainer does not read it. |
The daily job trains behavioral models. It does not regenerate an LLM understanding of every user's tastes each night. Content embeddings and ALS embeddings are different representations with different update paths.
Source: latest trainer run, trainer, embedding service, engagement features.
2. Confirmed cause of premature exhaustion
Direct unauthenticated API checks, with matching English language and default safety scope:
| Request | Result |
|---|---|
| Default catalog, newest, limit 1 | 536 eligible worlds reported |
| Recommended, offset 0, limit 24 | 247 candidates reported; 24 returned |
| Recommended, offset 264, limit 24 | Zero returned, total still 247 |
| Recommended, offset 600 | Hard stop: zero returned, request ID absent |
The recommendation generator combines fixed-size retrieval routes and a popularity-sorted backfill of only 200. The total returned to the client is the size of that selected pool after filtering, not the complete eligible catalog. Changing the offset does not extend retrieval. Consequently the UI correctly stops loading a prematurely exhausted pool.
There is also an independent 600-offset server cap, originally introduced to bound repeated expensive ranking. Removing that cap alone will not repair the 247-item limit and would restore the workload problem that prompted it.
Seven-day PostHog serve telemetry supports the API result: typical cold pools are about 247 items; mature pools about 277–280. Median maximum recommended position per person-day is 11, p90 77; 219 of 16,100 person-days reached position 240 or beyond. This affects real deep browsing even though most visits are shallow.
Offset pagination adds another problem: clicks, newly saved worlds, stat refreshes, and 45-minute rotation boundaries change the order between requests. The frontend removes duplicates, but skipped candidates are not recovered. If an append adds only duplicates, the observer's dependency on rendered list length also deserves a regression check.
Sources: candidate backfill, server cap, frontend pagination.
3. Guests cannot build an individual taste profile
Guest feed pages are shared through CDN/Redis caching. The first-party beacon stores a nullable authenticated user ID, but no anonymous actor ID or browsing-session ID. Guest clicks therefore cannot update a guest-specific interest vector, seen-history, fatigue state, or experiment assignment. The session-signal loader only runs for signed-in users.
Cold treatment ranking uses aggregate world playtime/returner scores plus daily jitter, with genre/creator constraints on the first 24 items. That retention ordering replaces the personalized base score as the main sort key. Even a signed-in cold user can click several cards and remain cold: clicks are not among the signals that promote the profile tier. Their session boosts therefore have little opportunity to affect this ordering.
Seven-day PostHog event-time identity counts show approximately 194,461 anonymous recommendation impressions and 554,370 identified impressions. Anonymous events matter substantially; analytics identity alone does not connect them to the serving system.
The shared response ID also creates ambiguity in training/rollups. The engagement rollup deduplicates by (feed_request_id, world_id, event_type), although multiple guests can receive the same cached response. In one day, 32,936 guest impression events collapsed to 30,473 distinct shared-slate/world keys. This is not a unique-person loss estimate: the schema cannot distinguish different guests from repeated beacons.
A separate check found 5,198 events attributed to authenticated users referencing guest serves. Login transitions are one plausible cause; the exact causes were not traced. The trainer requires an authenticated event but does not require the serve's actor to match, so its comment promising individually attributable authenticated impressions is not fully enforced.
Sources: beacon fields, context loader, cold composer, profile tier.
4. A confirmed semantic-input bug weakens story understanding
Publication hooks and the embedding backfill read schema.firstMessage. Production aggregate inspection found that field in 0 of 1,725 published worlds. The actual engine reads greeting entries from schema.entries; 1,708 worlds have an enabled greeting entry under the inspected aggregate predicate.
Thus the current embedding paths omit the intended opening text for current world schemas. They still embed titles, descriptions, tags, and announcements, so semantic retrieval exists, but its input is less informative than the code comments imply. Full character descriptions, narrative dynamics, gameplay mechanics, and lore are not otherwise represented by this embedding builder.
Repair the extraction from the actual published schema, use the engine's greeting semantics, and introduce an embedding-input version/content hash before a deliberate backfill. An updated_at later than the embedding timestamp is only a freshness flag: counters and unrelated edits may also update a world. It does not by itself prove a stale semantic vector.
Sources: approval extraction, backfill extraction, prompt builder.
5. Training outcomes do not mean “this recommendation caused a good new discovery”
The trainer sums capped playtime for each user-world pair across sessions created in a rolling 21-day window. It attaches the same value to that pair's impressions throughout the window and adds a bonus if a favorite currently exists. It does not require the playtime/favorite to happen after the impression, identify which exposure led to the start, or verify that this was a new discovery.
A bounded check of the latest day's recommended user-world pairs found 997 with positive playtime in that 21-day label window; 339 already had a session started before their first observed impression that day. This does not prove all their playtime preceded that impression, or that they had never seen an earlier recommendation. It confirms that the current label cannot attribute incremental new discovery from that exposure.
Further problems:
- Training uses current engagement statistics and current craft preferences for past impressions. Historical stat snapshots exist from August 19 onward but are only written, never used by feature extraction.
- An 85/15 chronological row split does not prevent user-world pairs and their shared outcomes from crossing the boundary. Recent examples have less time to accumulate outcomes.
- The same evaluation slice is used for early stopping and publication. There is no separate untouched test window.
- Offline scoring compares the model alone against served order, while production adds the model contribution to many fixed terms and reranks the result. The gate does not evaluate the policy actually served.
- NDCG ignores all-zero slates. That metric cannot tell us whether the system reduced visits where nothing useful was discovered.
- Setting position to zero at evaluation/serving is not a full solution to position and selection bias.
- Exponentiating a prediction of mean log-time does not automatically produce calibrated expected minutes.
Fix attribution and time correctness before increasing model complexity or treating offline uplift as proof. Log features as they existed at serving, use explicit future outcome windows, leave time for labels to mature, and evaluate the complete serving pipeline against the incumbent.
Sources: label extraction, craft extraction, evaluation and publication. Google's guidance explicitly recommends logging serving-time features to avoid this kind of training/serving mismatch: Rules of ML.
6. The learned model does not yet learn the strongest user-story connections
Its feature contract consists mostly of world popularity, exposure, quality, age, presentation, surface, profile tier, and the user's preference for UI/audio/token volume. Missing from the learned ranker are semantic user-world similarity, ALS affinity, theme/relationship/character preferences, recent behavior sequence, negative-interest similarity, and retrieval-source scores.
Some of these signals are present elsewhere, but added with fixed weights. The model cannot learn when a semantic match should outweigh popularity or when recent intent should outweigh a longstanding interest. A single average content vector also blends distinct interests together: liking both horror and gentle romance should not require every candidate to resemble their midpoint.
Other limitations amplify repetition:
- Long-term tag weights do not actually apply the recency decay calculated nearby; creator/content-vector weights do.
- Recent session interests use an unweighted union/centroid of up to eight clicked/played worlds from 45 minutes. Preview duration and depth are ignored.
- Any click/play in the 14-day fatigue window exempts that world from the no-click fatigue penalty, even if it was subsequently ignored many times.
- “Not interested” hides the exact world; it does not teach the model to show fewer semantically similar stories, despite the confirmation text promising that.
- Collaborative training uses all historical sessions, gives zero-playtime sessions positive weight, and does not exclude ephemeral sessions or creator self-play. These should be separated from consumer preference evidence.
- Craft features differ between training and serving: training averages qualifying sessions; serving averages distinct library worlds with message-depth weights. More UI code is also given a fixed presentation bonus; code length is not evidence of enjoyment.
Sources: feature contract, score composition, fatigue/session signals, ALS extraction.
7. Data exists, but not all of it belongs in the model or reaches it
| Signal | Available now | Recommendation use today | Proposed use |
|---|---|---|---|
| Confirmed viewport impression, click, play intent | First-party feed log and PostHog | Yes, with identity/attribution limitations | Opportunity IDs, deduplication, contextual negatives, attribution |
| Preview dwell | PostHog; 63,506 closes in sampled 7 days, median 5.29s, p90 24.02s | No direct ranker use; no surface/request ID in close-event contract | Foreground dwell plus preview context, normalized for length/device |
| Library adds/favorites | DB and some PostHog events | Exclusion, heuristic taste, favorite bonus | Explicit positive labels; distinct save-to-play funnel |
| Active play intervals | ClickHouse source view; about 4.01M intervals, 5,468 users in available portion of sampled week | Trainer uses cumulative session counters instead | Actual post-discovery engaged time, distinct active days, returns |
| Qualified activity | ClickHouse; about 726,582 foreground-engaged records, 4,443 users in available window | No trainer read path | Quality labels with event-time semantics |
| Dismissals | Postgres | Exact-world exclusion | Exact/group exclusions plus a separately tested soft negative preference |
| Search/tag exploration | Some context and events exist | Not a durable learned intent sequence | Intent features where explicitly and reliably logged |
| Community/creator activity, check-ins, technical errors | Analytics | Mostly absent | Context/guardrails only where predictive; do not count every action as liking a story |
ClickHouse began collecting these newer interval/activity sources around September 13; “seven-day” aggregates are partial coverage, not a full historical week. Raw interval sums may overlap and must be deduplicated before becoming outcome labels.
The PostHog-to-ClickHouse importer currently accepts only signup_completed; it does not import the recommendation funnel. The CDC inventory has gameplay and activity tables but no feed_events/feed_serves mirror. Unify relevant events by stable identifiers and event time, with explicit source coverage. Mirrored copies of an event are not extra evidence.
Existing rollups also define a returner as someone with at least two saved play-session rows. A user returning on three days to the same ongoing chat is missed. Use active-day/interval evidence instead. Exclude creator testing, ephemeral play, bots and technical failures from consumer-quality labels as appropriate.
Source: PostHog importer, rollup definitions.
8. What current outcome measurements do and do not establish
Seven-day PostHog raw event ratios: Recommended CTR 4.02% (748,894 impressions), Newest 5.88%, Popular 5.85%, Search 7.15%. These surfaces attract different intentions and users. This is descriptive evidence, not proof that switching everyone to Newest would improve recommendations.
A separate PostgreSQL comparison joined recommendation events to matching authenticated serves, validated world membership, and deduplicated opportunities by request/world/user. Among observations labelled mature:
| Arm | Users represented | Viewed opportunities | Clicked opportunities | CTR | Play-intent rate |
|---|---|---|---|---|---|
| Control | 356 | 54,019 | 2,173 | 4.02% | 0.79% |
| Heuristic engagement v1 | 724 | 118,063 | 5,849 | 4.95% | 1.00% |
| Learned engagement v2 | 689 | 121,929 | 4,440 | 3.64% | 0.84% |
The learned arm is not a demonstrated winner from these numbers. But it optimizes a different objective, and this is not a completed causal readout: repeated observations are clustered by user, tier can change after assignment, models change nightly, and play intent is not successful qualified play. Warm users show a different intent-rate pattern. Do not disable a variant solely from this table. Measure the intended discovery outcome with stable assignment, model-version logging, user-level uncertainty, and sufficient follow-up.
9. What to borrow from Meta and TikTok
Public sources expose designs and principles, not their complete current proprietary algorithms. The following is a grounded adaptation to Yumina, not a claim to reproduce those companies' systems.
| Published approach | Relevant lesson for Yumina |
|---|---|
| Instagram Explore describes multiple retrieval sources, recent and long-term signals, learned ranking, and a final diversity pass. Its two-tower model is trained on user-item engagement. | Preserve multiple ways to find a story; learn affinities from outcomes, then apply diversity across the actual browsing sequence. A vector index stores/searches representations; it does not learn taste by itself. Meta engineering, 2023 |
| Facebook Reels added an interest-satisfaction model using sampled user surveys alongside engagement predictions. | Enjoyment and perceived relevance are not equivalent to time spent. A small, optional “was this a good discovery?” signal can validate inferred tastes and niche recommendations. Meta engineering, January 2026 |
| Meta's sequence-modeling publication combines cached long histories with fresh online context and reports benefits from diverse action types and semantic features. This publication concerns ads. | Apply the separation of durable taste and current intent; do not treat ad-specific architectures or reported gains as evidence for our organic story feed. Meta engineering, August 2026 |
| TikTok describes learning from interactions, completion/skip signals, deliberate diversity, and avoiding repeated creators/content. | Relevant variety should be a deliberate policy. Do not mistake a fleeting card impression for watching or rejecting an entire story. TikTok's published explanation, 2020 |
| TikTok added topic-frequency controls and semantic keyword filters. | Let people correct inferred interests, including “less of this theme,” without making onboarding mandatory. TikTok, June 2025 |
| ByteDance's Monolith publication studies online learning; it explicitly identifies BytePlus Recommend as a deployment. | Fresh feedback matters. This does not establish that installing Monolith reproduces today's TikTok feed. Research paper |
10. Recommended target design
flowchart LR
E[Browsing and gameplay events] --> I[Stable identity and event validation]
I --> W[ClickHouse event history and outcome labels]
I --> S[Redis current interests and seen history]
C[Published world content] --> V[Semantic descriptors and pgvector]
W --> T[Training and evaluation]
T --> B[Behavioral factors and ranking models]
V --> R[Multiple candidate sources]
B --> R
S --> R
R --> K[Predict new discovery value]
K --> D[Diversity and exploration]
D --> F[Cursor feed with continuing retrieval]
F --> EMeasure successful new discovery. Primary product measures should include the share of Discover visits leading to at least one newly qualified story and distinct successful new discoveries per visit/user. A story is new relative to that person's library/history at exposure. Saves, engaged first play, satisfaction, and subsequent returns to that newly discovered story provide complementary labels. Old Library engagement must not be credited to today's unrelated impression. CTR is a diagnostic; excessive scrolling is not success by itself.
Represent several interests. Maintain a small set of weighted long-term interest vectors, a rapidly updated session representation, and collaborative factors. A candidate can match any credible interest without forcing all interests into one centroid. Keep uncertainty/confidence, recency and negative feedback. Cross-language behavior can inform taste while language eligibility remains explicit.
Understand worlds more deeply. Extract versioned descriptors from published creator-authored content: genre, tone, relationship dynamics, character archetypes, setting, player role, narrative style, gameplay mechanics and interaction format. Use the actual greeting entries. Store provenance/confidence. Combine descriptor embeddings with behavioral representations; let evaluation decide whether a larger/multilingual embedding model helps. More dimensions alone are not a plan.
Learn user-story interactions. First extend LightGBM with semantic similarity to each interest, ALS affinity, recent-intent match, exposure fatigue, negative similarities, and source scores. Train separate outcomes or a calibrated composite for save, qualified new play, return to the newly discovered story, and rejection. Product-owned weights remain visible. Later compare a two-tower retriever or compact sequence model once the corrected event data and baseline support a useful experiment.
Give guests a real session. Use a pseudonymous first-party anonymous identifier and server-recognized browsing session, carry it through events and pagination, and link it deliberately at authentication. Keep common candidate pools/cacheable metadata, but personalize final ordering and exposure IDs. Current public shared-page caching cannot remain the identity boundary. PostHog's identity mechanism can complement this, but the serving system needs its own coherent identity contract. PostHog identification.
Make variety persistent. Apply creator, theme and semantic-near-duplicate controls over recently served items across pages, rather than only the first 24/100. Reserve a small measured share for relevant adjacent interests and underexposed worlds. A possible experiment starts around 10% underexposed candidates after a few strong initial anchors, plus adjacent-interest candidates; these are proposed test settings, not industry constants. The current policy forbids cold-user exploration and starts warm/mature exploration only at positions 16/10. Changing that policy is a product decision, not just a coefficient tweak. New-world exploration needs a candidate source and exposure accounting, not only reranking the same popular pool.
Continue retrieval as scrolling continues. Replace moving offsets with a feed-session ID and opaque cursor. Keep the served prefix stable, remember seen world/group IDs, fetch additional unseen candidates as queues drain, and adapt only the unserved tail. Recheck eligibility at delivery, make cursor retries idempotent, bound work per page, prefetch, and virtualize the grid for long browsing. At today's catalog size, scoring all eligible lightweight candidates once per session is also a credible benchmark against complicated retrieval; preserve later refresh/replenishment.
Hard exclusions stay hard: saved stories/groups, dismissals, visibility/blocks, language choices, and content eligibility. A finite catalog cannot provide infinitely many unique unseen eligible stories. The first requirement is to exhaust the real eligible pool, not an arbitrary 247-item subset. After actual exhaustion, a proposed policy is to revisit merely overlooked, unsaved stories after a cooldown, with fresh diversification; never quietly recycle saved or explicitly rejected stories. That tradeoff needs product agreement.
11. Stack and implementation order for discussion
Keep Neon + pgvector for authoritative content/eligibility and semantic retrieval, ClickHouse for event history and training datasets, Redis for session features/seen state/feed cursors, Python + LightGBM/implicit for initial models, and PostHog for product/experiment analysis. Add versioned model/feature artifacts and monitoring. There is no measured need yet for a separate graph database, dedicated vector service, GPU fleet, or Kafka/Flink deployment. Evaluate PyTorch sequence/two-tower models later on corrected data; introduce infrastructure when serving/training measurements justify it.
Suggested sequence, with each stage independently reviewable:
- Fix candidate completeness/cursor pagination and the greeting extraction bug. Preserve all exclusion contracts. Add synthetic exhaustion/deduplication checks and audit the actual world-input payload without logging private content.
- Repair anonymous identity, impression attribution, event-time labels, feature snapshots, model-version logging, and distinct-day engagement. Connect selected warehouse signals. Ensure logging failures and feature/model staleness are observable.
- Improve guest adaptation, multi-interest features, learned user-story matching, semantic negatives, and controlled exploration. Keep the current policy as a measured baseline.
- Run stable randomized experiments against new-discovery outcomes, with an untouched chronological test window, user-level uncertainty, relevance/variety/creator coverage, latency/error rates, and an immediate rollback path. A/B gains must justify promoting advanced models.
Additional contract risk to verify during implementation: the profile loader language-filters library/favorite rows before building group exclusions. Saving a foreign-language variant may therefore fail to exclude its group in another language hub, despite the pure filtering tests intending whole-group exclusion. Load global exclusion state independently from language-scoped taste evidence.
Decisions left for discussion: the resurfacing policy after true eligible-catalog exhaustion; how much measured exploration to allow near the first screen; and whether future personalization should ever use private conversation content. The initial design can deliver substantial improvements from creator-authored world semantics and behavioral metadata without inspecting private conversations. No implementation or deployment has been started.