Discover ranking for time spent — data, design, and build plan

September 24, 2026. Owner direction: the one goal is time spent. Users should stay longer and find stories they will spend time on. The model, not hand rules, decides what "good" means; it learns that from our data by grouping users and stories. Discover never shows played or saved stories. Within one scroll a card shows once; a refresh or a new session ranks afresh and a passed card may return.

This document records what the data says about how our users and worlds behave, what the strongest consumer recommenders do that fits us, and the concrete changes to the training pipeline. The September 18 audit (2026-09-18-recommendation-system-audit.md) remains the reference for the pipeline inventory; this doc builds on it rather than repeating it.

1. How our users and worlds behave (measured September 2026)

Sources: ClickHouse analytics_play_intervals and discovery_events, production Postgres read-only, PostHog. Windows are stated per number.

Users are new and loyal.

Measure	Value
Signed-in players, last 14 days	8,901
Of those, accounts created in the last 14 days	57%
Accounts older than 90 days	11%
Paid	5%
Median minutes per player, 28 days	35 (mean 233, p90 614)
Median worlds per player, 28 days	1 (mean 3.2, p90 7)
Players who spend 80%+ of their time in one world	70%
Players back on 2+ days / 7+ days, 28 days	51% / 15%

Most of our active base is a few days old and looking for its first world. Once found, one world absorbs most of a player's time. A recommendation is therefore a bet on "does this become their home world", not on a click.

Returning is the time lever. Per user-world pair (September 13 to 17, the window with interval capture):

Measure	Value
Pairs under 1 minute total	26%
Pairs with 5+ minutes / 30+ minutes	53% / 29%
Pairs played again on another day	24%
Share of all minutes from pairs that returned	78%
First day's share of a pair's minutes	52%
P(return) given 5+ minutes on day one	38%
P(return) given under 2 minutes on day one	9%
P(return) given 30+ minutes on day one	54%

A label that stops at 24 hours sees half the value. Return days must be inside the target.

The catalog is head-heavy and mostly English, but Chinese players play twice as much.

Measure	Value
Published worlds	1,859 (en 1,024, zh 377, es 226, ja 225)
Creators / worlds from creators with 10+ worlds	329 / 61%
New worlds, last 30 days	501
Worlds with content embeddings and LLM descriptors	1,750
Worlds with behavioral (ALS) vectors	1,313
Worlds played in 28 days (incl. translations)	3,942; median 1 player each; 273 with 20+ players
Share of minutes: top 10 / top 50 / top 200 worlds	34% / 56% / 76%
Players and hours, 28 days: zh vs en	2,445 players, 14,784 h vs 5,057 players, 13,957 h

Popularity is a strong prior. Personalisation has to beat "popular with diversity", and cold-start content understanding matters because half the catalog has too few plays for behavioral vectors.

Discover drives new plays. Of 23,767 first plays of a world in 14 days, 72.5% had a Discover impression of that world in the prior 24 hours and 67.8% a Discover click in the prior hour.

Served is not seen. Of served cards only 24% ever enter the viewport (position 0-5: 24%; 12-23: 22%; 96+: 62%, survivor bias). Training on served cards makes 76% of the negatives noise.

Passed cards keep most of their value. Click rate by how many times the same user has seen the same card (14 days, viewport impressions):

Impression number	Impressions	Click rate
1st	130,366	4.93%
2nd	43,148	4.17%
3rd	19,790	3.92%
4th to 7th	21,728	3.65%
8th to 15th	4,339	2.93%
16th+	488	2.25%

By gap since the previous showing: same visit 3.92%, later the same day 4.07%, 1 to 3 days later 3.17%. A second showing keeps 85% of the first's click rate. Hiding a passed card for the rest of the visit throws that away; a cliff at eight views is arbitrary. The decay should be learned per user and card, which is what YouTube and LinkedIn do (impression-count and recency features, learned discounts).

Current experiment. Last 14 days, signed-in, recommended surface:

Arm	Plays per 100 impressions	Minutes per user	Users back 2+ days
control (pre-engagement scoring)	0.85	324	61.0%
engage_v1 (hand-tuned engagement terms)	1.12	374	62.8%
engage_v2 (learned model)	0.97	336	59.7%

The learned arm has never beaten the hand-tuned arm. Its 29 features describe the card and the user's coarse tier; none describe the user's history with the card, their taste vector, or the semantic match. Those live outside the model as fixed additive terms, so it cannot learn when they matter.

2. What the strongest recommenders do that fits us

Details and sources are in the research notes of September 24 (summarised here).

• YouTube (2016, 2019). Rank by expected watch time, not clicks, with unclicked impressions as unit-weight negatives. Feed the model the count and recency of past impressions of the same video so it learns to demote what a user keeps passing. Feed candidate-generation scores to the ranker. Handle position with a separate shallow tower and position dropout.
• Roblox. Labels are play-through rate, first-play bounce, play days per user (D1, D2-7, D8-28) and playtime capped at 60 minutes per user per game per day, all averaged per user. New experiences get a test cohort and expand if it engages ("explore and expand").
• TikTok / Kuaishou. Objective is time and returns; fresh feedback beats model complexity; debias duration so long items don't win by default.
• X. Multitask predictions combined with a visible weight table; communities learned from behavior, not tags; negative feedback decays over 14 days.
• Steam. Learns from playtime history only; hides owned games; throttles games shown but ignored.
• Re-exposure literature (LinkedIn 2014, impression-aware survey 2023). Half of impressions are repeats; learn a discount over count and recency instead of hard caps.
• Scale. At about 40k impressions a day, well-tuned matrix factorisation plus gradient-boosted trees matches neural two-tower and sequence models; those need roughly 10x our data.

3. Design

One goal, two stages, one learned model, four fixed rules.

Retrieve a few hundred candidates from the existing sources (taste-vector neighbours, co-play, content similarity, popular, new, followed creators). Each source's flag is a ranker feature.

Rank with one LightGBM model that predicts attributed active minutes on the story over a horizon after the impression, capped per day, which folds return days into the target.

Fixed rules: hide played and saved stories; honour "not interested"; no repeat within one scroll; eligibility (language, age rating, publication). Everything else, including fatigue and how much to prefer unseen cards, is a learned weight.

3.1 Label

• Unit: a seen card (viewport impression, or a click without a recorded impression), not a served card. Unseen served cards carry no information about the user's decision.
• Target: minutes on the story from attributed active intervals within H days of the impression, summed per UTC day with each day capped at 60 minutes. H is 7. Unclicked impressions get 0.
• Attribution stays as built: a server-confirmed session_started anchored to the opportunity within 24 hours, then intervals on that session or the same story.
• Objective: Tweedie regression on raw capped minutes (variance power 1.5), which models zero-inflated positive targets directly and predicts expected minutes. A log1p-L2 candidate is trained alongside and the validation slate NDCG@24 picks between them.
• Horizon coverage, measured on September 13 to 17 pairs with the 60-minute day cap: the first day holds 39% of a pair's first-week minutes, three days 66%, seven days 82%. A 24-hour label sees less than half of the value it should credit.
• Ramp: the measurement log began on September 21, and every split gap in the rolling plan equals the label maturity (horizon plus one day of receipt grace), so a longer horizon trains on older data. Plan windows: 1 day → train on days 9 to 21 ago; 3 days → 15 to 27; 7 days → 27 to 39. The nightly job reads TRAINER_LABEL_HORIZON_DAYS (default 1) and records the horizon in the model's metrics. Raise it to 3 once about four weeks of log exist, then to 7.

3.2 Training cohort

• All arms (control, engage_v1, engage_v2) and all signed-in users. Training only on the incumbent's own slates biases the data; other arms give counterfactual exposure diversity.
• Older accounts whose "known at serve" state is null are included as new-story exposures. Served cards are already excluded when the story is in the user's library, favourites or reviews, so a served card is new to that account. The server now records outcomes for them (isNewStory = not known, instead of known-false only).
• The ClickHouse column training_known_state (materialised from the payload) makes this cheap; the exporter falls back to the old training_known_new=1 predicate until the column exists.

3.3 Features (snapshot version discovery-features-v3)

Everything below already exists in our tables. The ranker gains 24 inputs; the full as-served snapshot gains the same plus always-present starter and personalisation keys, so one schema covers every feed.

Group	Features
Story facts (existing)	popularity, quality, velocity, age, freshness, audio, UI tier, tokens, rating
History with this card (new)	seen before; impressions 7d and 30d; hours since last impression; clicks 30d; hours since last click; seen in the last hour; unclicked impressions 14d
Similarity (new)	ALS cosine and known flag; long-term content-taste cosine and known flag; session-centroid cosine; session tag overlap
Retrieval source (new)	co-played, similar-played, creator affinity, followed creator, popular
User and context (new)	log signal count; hour of day (sin, cos); weekday
User craft taste (existing)	UI, audio, token preferences

Kept out: raw user and story IDs (the model would memorise them at our volume) and spend (never a ranking input). Position stays fixed at 0 in the model input, as today.

3.4 Serving

The published model's raw output becomes expected minutes (Tweedie: exp(raw); legacy regression: expm1(raw)), then the existing lift term versus the model's validation baseline. Models are matched by feature name, so a v2 model keeps serving while v3 rows accumulate, and a v3 model serves once it passes the gate.

3.5 Exploration (next step, not in this change)

Reserve about 10% of slots for new or under-shown stories chosen by sampling from each story's estimated minutes per impression; expand winners to similar audiences. This also gives the model varied repeat-show data.

3.6 Evaluation

• Offline gate unchanged in form: equal-actor slate NDCG@24 on an untouched daily holdout, paired bootstrap against both the incumbent and the served order; now on seen-card rows with the new label.
• Online: two arms, at least two weeks, per-user minutes per week and return days as the primary metrics; bounce under a minute and dismiss rate as guardrails.

3.7 Operating the change

• Apply the ClickHouse column once (the exporter falls back to the explicit-false cohort until it exists): from packages/server, railway run --service yumina-analytics --environment production -- npx tsx src/analytics/worker.ts --apply-discovery-known-state.
• Set TRAINER_LABEL_HORIZON_DAYS on the trainer service: 1 now, 3 in mid-October, 7 in November.
• Each nightly report records the selected candidate objective, both candidates' validation NDCG, the label horizon, the cohort predicate and the seen-row counts.

4. Build plan

1. Server (PR #312, merged September 24): exposure loader from feed_events; v3 ranker features; always-present starter and personalisation keys; snapshot version v3; Tweedie model kind; outcomes recorded for null-known accounts.
2. Trainer (also PR #312; the image-build memory fix is PR #313): seen-card unit; horizon-parameterised day-capped label; cohorts admit signed-in unknown history with the explicit-false fallback; Tweedie and log1p candidates with validation selection; ClickHouse migration for training_known_state. Pooling the two engagement arms into one cohort is the next data multiplier and is not in this change.
3. Exploration slots.
4. Two-arm experiment on minutes per user.

5. Open decisions for the owner

• Retire engage_v2 until a v3 model passes the gate, moving its 40% of users to engage_v1.
• Whether creators should see when their story is in a test round.