Yumina Docs

English

Español
日本語
中文

English

Español
日本語
中文

Theme

Recommendation System Investigation — Data, Current State, Industry, Proposal ​

Date: 2026-08-18 · Status: investigation complete, awaiting discussion — NO code changes made. Sources: PostHog (30d behavioral data), public hub API catalog sample, full code map of packages/server/src/lib/recommendations.ts + hub UI, five deep research reports (TikTok/ByteDance, Xiaohongshu, YouTube/Google lineage, X/the-algorithm, Roblox+Steam, OSS stacks).

Part 1 — What our data actually says (30 days, PostHog, to 2026-08-18) ​

Scale & funnel ​

MetricValue
DAU (server-active)~1,400–2,250 and climbing (was ~794 mid-July)
Signups / 30d5,253
Hub browsers / 30d14,549 persons (8,059 guests, 6,647 identified)
Feed impressions / 30d1,378,862 (position + surface + feed_request_id on every one)
Card clicks / 30d72,648
Preview opens / 30d90,346 (median dwell 5.5s, p90 20.9s)
Play starts from hub / 30d35,002
Library adds / 30d2,655
Published catalog~612 worlds total (~304 in the zh recommended pool); ~900 world ids incl. variants got ≥1 impression

The damning findings (= the opportunity) ​

  1. The personalized feed underperforms dumb feeds. CTR by surface: recommended 4.09% < popular 6.62% < newest 7.96% < search 8.5%. The one surface we spend all engineering on is the worst-converting.
  2. Personalization tiers don't pay off. CTR by profile tier (7d, via feed_request_id join): mature (most personalized) 4.04%, cold 3.54%, warm 5.41%. Users with 4+ signals get an essentially cold-quality feed.
  3. Position-CTR curve is flat. pos 0-3: 6.62% → pos 4-63: ~3.5-3.9% → 64+: 2.76%. A well-ordered feed decays steeply from the top; ours barely does → the ranking order carries little information.
  4. Impressions are sprayed; demand is concentrated. Top-20 worlds get 18.3% of impressions but 44.4% of plays; top-100: 47.5% of impressions, 72.7% of plays. CTR across worlds with ~10-17k impressions ranges 0.84% → 32.7% — the feed keeps paying impressions to proven losers and starving proven winners.
  5. Median scroll depth = position 11 (p90 = 71). Half of users never see beyond the first dozen cards. The "Load more" button caps inventory consumption. The first screen decides nearly everything.
  6. 40% of browsers (5,775/14,549) click NOTHING in 30 days. Another 8.6% click but never play. Only 52% ever start a world from the hub.
  7. Activation is bimodal. 36.8% of signups play within 24h — median time-to-first-play is 2 minutes. The other 63% mostly never activate. (Paid Reddit cohort: 0% D1 return.) D1 retention 12.8%, D7 7.7% overall.
  8. Players are loyal, not variety-seeking. Median worlds-played per player in 30d = 1 (avg 2.3, p90 = 5). 33.5% of authed hub plays are returns to library worlds. We are a Roblox-shaped loyalty product, not a TikTok-shaped variety product. Retention lever #1 is reconnecting a user with their one world, then carefully expanding to world #2.
  9. Guests are a big blocked audience: 8,059 guest browsers (more persons than identified!), 121k impressions, 3.76% CTR, 5,020 play attempts hitting the login wall.

Catalog reality (public API sample, 200 newest + tags endpoint) ​

  • Tag vocabulary is noisy and multilingual: 316 distinct tags on 200 worlds; duplicates like 世界卡/world card, Alpha/Alfa, anime/动漫; structural tags (角色卡/世界卡) mixed with genre (恋爱) and audience (男性向). Top global tags: 角色卡 228, 男性向 212, 世界卡 196, 模拟器 123, 恋爱 108, NSFW 80. Median 3 tags/world.
  • Newest-200 language mix: en 100, zh 78, es 21 (Spanish omegaverse cluster is real), ja 1.
  • Median newest world: 150 messages, 1 favorite, 9 downloads, 0 reviews (ratings essentially unused as a signal). p90: 8,063 messages. Power-law engagement.
  • 71 creators in newest 200; one creator published 33 (flood risk → diversity rules matter).
  • Audience clusters (from 2026-07 growth audit): K-pop 团体模拟器 fangirls = 64% of MAU (TW/HK, mobile 85%), zh male NSFW = 23% of MAU but 100% of top subscription worlds, third cluster horror/battle-royale engine games.

Part 2 — What we already have (better than expected) ​

packages/server/src/lib/recommendations.ts (2,936 lines) is already a real 3-stage pipeline:

  • Recall: 8 parallel candidate routes with per-tier budgets — popular_recent, exact_tag, related_tag, creator_memory, creator_affinity, followed_recent, similar_played (pgvector kNN on worlds.embedding vector(1536), HNSW, user centroid = engagement-weighted average), adult_popular. Cold tier fans out per-genre-cluster popular routes.
  • Scoring: hand-tuned pointwise formula = tag-IDF mindshare match (L1-normalized per-user weights from favorites/ratings/library/follows, with play-depth tiers <5/<20/<100/100+ messages and recency decay) + creator weights + lineage + related-tag graph + quality counters (log downloads/favorites/reviews/rating/messages) + freshness half-life 14d + editorial boosts (DB table) + vector cosine² (weight 60) + presentation bonuses (customUiLoc, hasAudio).
  • Re-rank: greedy listwise diversity over top 100 (same-char 20n, creator 12·ln n, tag-cluster 7·ln n penalties).
  • Cold-start slate composer: constraint satisfaction for the first 24 cards (cluster cap 3, creator cap 2, MMR cosine 0.82, adult quota, deterministic daily jitter).
  • Infra: 3-layer caching (graph 5min / profile 300s / feed page 120s Redis + CF edge for guests); full funnel instrumentation in PostHog with feed_request_id joining serve→impression→click.

What's missing (the actual gaps, ranked):

  1. Nothing learns. All ~30 constants hand-tuned; no training loop; no offline eval (NDCG/replay); no feedback from the 1.4M impressions/month back into ranking.
  2. No training table. Impressions/clicks/slates exist ONLY in PostHog — never in Postgres, not joinable. world_click_history lacks position/surface/request-id and is user-deletable. Score breakdowns are computed then discarded.
  3. No A/B system. Zero experiment assignment for the feed (only support_prompt_events.holdout exists platform-wide).
  4. No behavioral quality signals in ranking. play_sessions.playtimeSeconds, message depth per (user,world), swipe/regen counts, D1-return-to-world — all in the DB, none feed the score. Quality = lifetime cumulative counters (no per-user normalization, no velocity/trending window — "hot right now" is inexpressible).
  5. No co-play collaborative filtering. "Users who played X also played Y" never computed despite the data sitting in user_library/play_sessions.
  6. No explicit negatives. No "not interested"/hide on worlds; preview_close dwell (the best implicit negative) is PostHog-only; the close reason enum is broken (100% "other").
  7. Dark surfaces: featured & category cards emit no impressions/clicks (HubCard rendered without surface prop). Scroll-depth/viewport-reach untracked → "never seen" vs "seen and rejected" is confounded.
  8. No user cold-start inputs. Onboarding collects only language + audiencePreference + birthYear. No interest/fandom picker; no gender.
  9. UI limits signal + consumption: single flat grid, manual Load-more (median depth 11), no Continue-Playing shelf on top despite 33% of plays being returns, preview modal is an extra funnel step (but also a dwell-label generator — keep it).

Part 3 — Industry synthesis (5 research reports, condensed) ​

Universal architecture (TikTok, XHS, YouTube, X, Roblox all converge): multi-source recall → (light rank) → heavy rank = multi-task engagement probabilities combined as score = Σ wᵢ·P(actionᵢ) with product-owned weights → re-rank rules (diversity, fatigue, exploration quota) → serve; impression logs with position feed the trainer; negative feedback weighted enormously (X: like 0.5, reply 13.5, reply-engaged-by-author 75, "show less" −74, report −369).

Labels by product shape:

  • TikTok: dwell/completion dominates (WSJ: full personalization from dwell alone in ~40min). Feed = autoplay → every impression is a labeled dwell.
  • Xiaohongshu (our closest UI analog): dual-column masonry → card-tap CTR is the first-order label, then CES = like 1 / save 1 / comment 4 / share 4 / follow 8 post-click.
  • Roblox/Steam (our closest product analog): the label is qualified, retained play — never the click. Roblox ranks Home on: qualified play-through rate (engaging plays ÷ impressions), 7d playtime per user capped at 60min/day (anti-whale), 7d play-days, co-play, spend-days — all per-user-averaged so small-but-loved games win. Steam's Interactive Recommender trains purely on playtime sequences, ignores tags/reviews entirely; Discovery Queue = forced-verdict deck that manufactures explicit labels (incl. "Not Interested").
  • YouTube: watch-time-weighted logistic regression (2016) → MMoE multi-task + shallow tower for position bias (2019) → semantic IDs / Gemini-based LRM (2024-25). Log position at training, fix it at serving.

Cold start:

  • New items: tiered traffic pools (Douyin/XHS: ~200-500 guaranteed impressions to an interest-matched test audience; promotion gated on engagement thresholds; "200-view jail" if failed). Roblox: explore-and-expand cohorts. Steam: least-views-first new-release queue + event-triggered visibility rounds. All three make the ladder legible to creators — it doubles as creator retention.
  • New users: interest picker at signup (XHS collects gender/age/interests) + popularity prior + bandit-fast convergence; TikTok proves session-level adaptation is enough.

Wide & Deep answer: it's 2016 Google Play tech; its DNA (parallel memorization+generalization, shallow tower) was absorbed into DCN-v2 → MaskNet (X's Heavy Ranker) → sequence transformers (Pinterest TransAct, Meta HSTU +12.4%, Netflix/YouTube foundation models). Nobody picks literal Wide&Deep in 2026. For a 1-2 engineer team the industry-standard lean stack is: LLM text embeddings + pgvector retrieval + GBDT (LightGBM LambdaMART) ranker on logged impressions, two-tower later, never HSTU-class at our scale.

OSS verdicts: Gorse = real but bus-factor-1, black-box, wastes our LLM-native advantage — skip. NVIDIA Merlin = maintenance mode, avoid. LightFM dead. Adopt libraries, not systems: implicit (ALS co-play, healthy), LightGBM → ONNX → onnxruntime-node (sub-ms in-process scoring in Hono), pgvector (already ours, Neon-native), hand-rolled Thompson sampling (~30 lines), PostHog batch export or direct server-side logging. No Kafka/Flink/Feast at <1M users — Postgres + cron rollups.

X's most copyable ideas: the explicit weight table as a product dial; author-diversity decay score × (0.75 × 0.5^position + 0.25); 14-day feedback-fatigue window; SimClusters-lite = factorize our user×world matrix into a few hundred taste communities (matches our known K-pop/NSFW/horror clusters); "simple + fresh data beats fancy + stale."

UI patterns that manufacture signal: Roblox shelves (per-shelf generators + personalized shelf ORDER — their cheapest 2025 win); XHS masonry (we're already 2-col mobile); Steam Discovery Queue (daily 8-12 forced-verdict deck with a small reward = explicit labels at low volume); Continue-Playing as the retention anchor; Roblox "Moments" (short clips as a join funnel into long sessions).

Part 4 — Proposed direction (for discussion) ​

North star: qualified-play-through rate per impression (qPTR: sessions reaching ≥N user messages ÷ impressions) and D7 return — with feed CTR as the fast intermediate metric. Guardrails: new-world survival rate (creator supply), catalog coverage, diversity.

Why not raw CTR alone: with a 33% preview→play drop and clickbait covers as the obvious failure mode, every analog platform (Roblox/Steam/YouTube) converged on qualified-engagement labels. We should chase clicks at stage 1 and qualified play at stage 2.

Phase 0 — Instrument + heuristic quick wins (1-2 weeks, no ML) ​

  1. feed_impressions training log in Postgres (append-only, partitioned/prunable): the /hub handler already holds feed_request_id + ordered world_ids + tier + user; add client beacon confirmation (viewed positions) + click/play join. Persist the existing ScoreBreakdown per serve. This single table converts 1.4M impressions/month into joinable training data.
  2. Feed experiment infra: deterministic user-hash bucketing, variant logged on hub_serve. Without this nothing can be proven.
  3. Fix dark telemetry: featured/category surface props; hub_play_start missing surface/frid; preview_close reason bug; max-scroll-position event.
  4. Heuristic rank patch (while data accrues): add to the existing formula (a) empirical CTR/qPTR priors per world (Bayesian-smoothed, backfilled from PostHog), (b) velocity term (Δ engagement 7d vs prior — "trending" is currently inexpressible), (c) demote high-impression/low-CTR worlds. Given finding #4 (0.84%–32.7% CTR at equal impressions), reallocating impressions toward demonstrated CTR should lift feed CTR substantially on its own.
  5. "Not interested" on cards (world + optionally tag) with a heavy negative weight + 14d fatigue window.
  6. New-world visibility rounds: ring-fenced ~10% exploration slice; each new world gets ~300-500 impressions over 48-72h to taste-matched users (Thompson sampling per world); promotion thresholds on CTR + qualified-session rate; surfaced in the creator dashboard (Roblox-style legibility).

Phase 1 — Learned ranker v1 (month 2) ​

  1. Nightly LightGBM on the impression log: features = current score components + CTR/velocity priors + user-tag affinities + co-play similarity + position (fixed at serve); labels = click → upgraded to qualified-play-weighted. Export ONNX, score in-process (onnxruntime-node) over the ~200-600 candidates. At ~600 worlds we can score the whole catalog per user — no retrieval bottleneck yet.
  2. Co-play ALS (implicit) nightly on user×world engagement → new recall route + item-item "because you played X" + taste clusters (SimClusters-lite) for cold-start pool matching.
  3. Onboarding interest picker (fandoms/genres/tones — K-pop, 乙女, omegaverse, horror, 模拟器…) seeding tag weights day-0. Directly attacks: 63% of signups never play, 0% paid-cohort D1.
  4. Multi-task weight table (X-style, explicit and product-owned): score = w·P(click) + w·P(qualified session) + w·P(D1 return) + w·P(favorite) − big·P(not interested). Start with observed rates as P's; upgrade to per-head models as data grows.

Phase 2 — Product surface (months 2-4) ​

  1. Shelf-based home: Continue Playing (top — 33% of plays are returns; median player has ONE world), For You, New & Rising (the ring-fenced pool), Trending Now, taste-cluster shelves, Because-you-played-X; editorial Today's Picks stays. Personalized shelf order later.
  2. Infinite scroll + prefetch (replace Load-more; median depth 11 is partly UI-inflicted), masonry-style card emphasis A/B, cover-image A/B tooling for creators (the card is the ranked asset — XHS lesson).
  3. Optional: Discovery-queue deck ("today's 8 picks", swipe verdicts, small mushie reward) = explicit labels + a daily habit loop that fits the check-in system we already have.
  4. Session-level adaptation: within-session tap history reweights the next page (cheap bandit, no infra).

Phase 3 — Scale (10k+ worlds, 100k+ DAU) ​

  1. Two-tower trained on own interactions replacing the centroid; ANN stays pgvector until proven insufficient; per-user embedding table; near-real-time feature freshness (TikTok's real lesson = short feedback loops, not exotic models); revisit dedicated vector DB / feature store only on measured need.

New stack required: one Python nightly training job (Railway cron) + onnxruntime-node + 2-3 new Postgres tables. No Kafka, no new datastore, no GPU, no Gorse.

Open questions for discussion ​

  1. North-star tradeoff: optimize clicks (fast, risk: clickbait) vs qualified play (slow, robust)? Proposal: CTR for stage-1 candidate pruning, qualified-play for final ranking + creator incentives.
  2. Guest strategy: 8k guest browsers/month blocked at login. Allow N free guest messages (top funnel fix flagged in the growth audit) → also feeds the recommender day-0 signals?
  3. UI ambition: patch the current grid (Phase 0-1 only) vs commit to the shelf home (Phase 2)? Shelves change the product's face — worth a design pass.
  4. Tag taxonomy: invest in canonical genre/fandom taxonomy + auto-tagging via LLM (NoteLLM pattern — cheap for us) now, or defer? Current tags are noisy but IDF partially compensates; taxonomy pays off most for onboarding picker + shelves + cold-start pools.
  5. NSFW segregation in ranking: the two economies (K-pop fangirls vs NSFW subs) barely overlap — separate ranking models/pools per content mode, or one model with contentLevel features?
  6. Prod DB access for deeper audit: Railway CLI session expired; railway login would unlock playtime/message-depth distributions and per-cluster economics to sharpen the label design.

Appendix — Key raw numbers (PostHog queries, 30d to 2026-08-18) ​

  • Event volumes: hub_impression 1,378,862 / hub_serve 335,166 / hub_preview_open 90,346 / hub_click 72,648 / hub_play_start 35,002 / signup_completed 5,390 / hub_library_add 2,655.
  • CTR by surface: recommended 4.09% (938,705 imp) / popular 6.62% / newest 7.96% / search 8.5% / following 5.36% (2,145 imp — social graph unused).
  • CTR by position (recommended): 0-3: 6.62% / 4-7: 3.62% / 8-15: 3.49% / 16-31: 3.87% / 32-63: 3.82% / 64+: 2.76%.
  • CTR by tier (7d): mature 4.04% / warm 5.41% / cold 3.54% / other-surface 9.41%.
  • Concentration: top-20 worlds = 18.3% imp / 26.3% clicks / 44.4% plays; top-100 = 47.5% / 58.3% / 72.7%. 905 worlds ≥1 impression.
  • Users: 14,549 browsers = 5,775 pure scrollers (39.7%) + 1,246 click-no-play (8.6%) + 7,528 players (51.7%); avg 95 impressions/user, median 14.
  • Active days (server-active persons): 1 day 67.2% / 2-3 12.8% / 4-7 6.7% / 8-14 5.4% / 15-29 7.2% / 30 0.6%.
  • Retention (signups 8-37d old, n=3,741): D1 12.8%, D7 7.7%. Activation: 36.8% of signups play <24h, median 2 min.
  • Plays split: discovery-authed 18,245 (52%) / library-authed 11,736 (33.5%) / guest 5,020 (14.3%, login-walled).
  • Worlds per player: median 1, avg 2.3, p90 5, max 110.
  • Scroll depth (recommended): median max-position 11, p90 71, p99 239.
  • Preview dwell: median 5.5s, p90 20.9s (n=87,280; reason enum 100% "other" — broken).
  • Catalog: total_count 612 (default) / 304 (recommended zh); cache hit 17.9% recommended, 0% default.