Story understanding and reader interests: rollout

This change builds on PR #172 (continuation and published greetings) and PR #177 (canonical outcomes, bounded sessions and temporal datasets). It does not enable production flags, install production schema, backfill profiles or replace the published ranker automatically. User authorization covers implementation; the gates below are engineering verification requirements, not a request for more credentials or another design approval.

Existing stack and responsibilities

The quality/performance follow-up adds an experimental, independently reviewed evidence path that is not connected to serving. Do not treat its proposals as production descriptors. Likewise keep DISCOVERY_SESSION_ENCODING=raw; enabling optional compact-v1 first requires every reader upgraded, and old-binary rollback requires raw writers plus a full one-hour TTL drain. Current paired measurements establish byte/memory savings, not a consistent latency benefit or production capacity.

Component	What this implementation uses it for
Railway app	Eligibility, candidate recall, ranking and optional interest API/UI.
Neon/pgvector	Published catalog, explicit preferences, versioned story descriptors, separate content and experience vectors.
Railway Redis	Existing bounded cursor sessions. Hybrid recall keeps at most 247 initial queued snapshots; later catalog scans preserve reachability.
Background worker on Railway or an operator-selected environment	Bounded, restartable story extraction/embedding outside request latency. Same existing OpenRouter/OpenAI accounts.
ClickHouse and existing export jobs	Canonical events, historical features and matured outcome datasets. PostHog remains product analytics; mirrored clicks do not become extra training evidence.
Python offline evaluation	Reproducible baseline/content/multiple-interest ablations on identical eligible cohorts. Results are offline artifacts, never automatically published models.

No Kafka, graph database, GPU cluster, dedicated vector vendor or new subscription is required for this stage. Existing Redis is real Redis; another Railway Redis instance would be workload isolation, not a different technology. Resize or split services only after measuring headroom and latency with this final feature shape.

The local benchmark-discovery-store.ts 10000 --personalization JSON-codec probe delivered all 10,000 synthetic families in 417 pages with no duplicate delivery. It retained at most 247 queued and 72 committed-page snapshots, each with the 147-field ranking contract, queued recall provenance and a preference revision. Maximum serialized session size was 1,948,268 bytes (about 1.86 MiB); 1,000 sessions at that shape alone project to 1.95 GB before Redis overhead, replication and headroom. This is a payload-sizing observation, not a production concurrency or latency benchmark. The existing shared Redis must not be assumed to have that capacity. Measure the actual traffic/session distribution and size or isolate Redis before raising admission limits.

Flags and compatibility

Flag	Default	Effect
DISCOVERY_PERSONALIZATION_ENABLED	off	Loads new pair features for cursor Discover. Does not by itself turn on new ranking.
DISCOVERY_PERSONALIZATION_ROLLOUT_PERCENT	0	Stable actor assignment to the experimental policy, valid range 0–100. Invalid values select no treatment.
DISCOVERY_INTEREST_ONBOARDING_ENABLED	off	Enables the optional chooser API/UI when personalization is also enabled.

The existing cursor/measurement flags and their schema/warehouse prerequisites still apply. Feature version discovery-features-v1 retains its 136 numeric ranking inputs. discovery-features-v2 adds eight pair inputs plus availability, policy-applied and actual-score diagnostics (147 total). Ranking snapshots keep the exact feature version; actor-level preference events have their own contract and never become positive reading labels.

Content is extracted from the current eligible published row. The worker records input hash/version, descriptor version/model, embedding model and source update timestamp. Content and experience embeddings remain separate even with the same dimension count. Runtime loads reject incompatible versions/models and stale source timestamps. A bounded cache holds public story representations, with authoritative freshness/publication checks on every read; it holds no reader profiles. No private conversation text is used by this feature.

Schema and bounded extraction

Install the existing measurement schema first, then run the additive packages/server/scripts/discovery-personalization.sql against the explicitly selected environment. The installer creates two tables and their indexes; it does not create descriptors or call providers. Its transactional HNSW index creation is intended for initially empty tables. Do not use it as a live, concurrent index rebuild procedure for a large existing catalog.

From packages/server, with database and provider variables supplied by the existing environment (never pasted into chat or committed):

# Default dry-run: read-only DB session, no provider calls.
pnpm exec tsx scripts/personalization-worker.ts --descriptor-model MODEL --limit 10

# Explicit bounded apply; run in testing before production.
pnpm exec tsx scripts/personalization-worker.ts --apply --descriptor-model MODEL --limit 10 --max-requests 20 --max-units 30 --journal-path /var/lib/yumina/operator-journals/catalog-run-001.jsonl

MODEL must be a verified exact OpenRouter model identifier supporting structured outputs. Requests require supporting provider parameters and validate the returned model identity. There is no automatic model alias/fallback promotion. Embeddings use verified text-embedding-3-small responses. The worker accepts OPENROUTER_API_KEY or the existing YUMINA_OPENROUTER_KEY, and OPENAI_API_KEY.

Both dry-run and apply require an explicitly supplied direct/session-stableDATABASE_URL. Use Neon's direct endpoint, not its -pooler hostname: dry-run's read-only session setting and apply's advisory lock must stay on the same backend connection throughout the job. The CLI rejects known pooler/PgBouncer hosts and explicit transaction-pooling options before connecting; custom proxies must also preserve sessions. Lock contention stops before provider work, and connection loss aborts the active job.

Provision the journal parent directory on durable operator storage outside Git. Every run needs a new absolute .jsonl path; on Windows, for example, C:/Users/OPERATOR/operator-journals/catalog-run-001.jsonl. Dry-run creates no journal. Apply fsyncs request begin before spending, observed usage before semantic validation, and a separate terminal outcome. Missing usage/cost remains unknown; journal failure stops further spend. These are platform-funded shared catalog costs, never user wallet charges. See operator cost policy.

One successful story needs one extraction request and one batch embedding request containing one or two inputs. Failed attempts and retries consume the declared request/unit budget. Units bound workload, not dollars; obtain current provider prices and observe actual usage before a catalog-wide backfill. Canonical inputs, completion length, response size, retry count and request duration are bounded. Database operations have their own timeout and can finish after the worker's between-step run deadline. nextAfter restarts the keyset pass; edited rows behind that cursor require a later pass from the beginning. Oversized/invalid rows stop with a sanitized failure and need investigation, not an unbounded automatic loop.

Provider output requires literal source evidence and a valid supported taxonomy. That proves provenance, not semantic truth. Review examples in each catalog language and category, including fandom/character relationships, idols, simulation, cultivation, and survival. Audit failed extraction as well as accepted facets; strong validation can otherwise hide a coverage problem.

Descriptor v3 first rejects malformed envelopes, wrong hashes/versions, duplicate proposals and exceeded budgets. It then keeps only individually supported quotes and drops facets with no surviving evidence, without rewriting labels or quotes. The resulting descriptor passes the same strict validator used for storage. The CLI reports proposed/retained facet and quote counts plus empty descriptors; an empty descriptor can still receive a content embedding, but no experience embedding. These counters describe accepted provider responses, not committed rows or accuracy. A failed or superseded write can follow an accepted response.

Scientific and operational gates

1. Complete the foundation runbook's warehouse source coverage, completeness, erasure, TTL/capacity and isolated HTTP/SQL load checks. Recheck current main conflicts before landing the stacked changes.
2. On testing, install schema, dry-run and apply a small explicit extraction budget. Exercise edited/unpublished worlds, provider failure, overlapping jobs, missing profiles, guest handoff, account deletion and optional-API failure.
3. Review descriptor acceptance, language/category coverage and source evidence on real catalog samples. Do not infer extraction quality from syntactic validity or synthetic tests alone. Record exact model/prompt/input versions.
4. Enable personalization with rollout 0 first. Confirm shadow mode does not change ranking/recall and measure feature availability, vector-cache hit rate, profile/recall latency, timeouts, database work and Redis bytes for v2 snapshots.
5. Run a small stable ranking treatment after coverage and performance are acceptable. Monitor qualified new-story reading, new saves, family/creator diversity, reachability, hides, latency and errors. Record policy failures and missing features. Roll back treatment percentage before changing infrastructure.
6. Test the optional chooser separately. Analyze every assigned eligible actor, including skips; picker completion is not success. Existing explicit choices remain editable/resettable and unselected choices remain unknown.
7. After complete temporal windows mature, run the offline comparison, freeze validation decisions and inspect test only for a final readout. The transparent logistic reference is an ablation tool, not a replacement for the current LightGBM deployment. Compare a trained extension of that baseline on the same corrected outcomes before publishing a new serving model. No copied/post-exposure vectors may stand in for missing historical features.

The initial affinity weights and clustering thresholds are experimental. The system deliberately discounts one-family evidence and keeps enduring interests, current-visit intent and explicit choices distinct. Story content describes available experiences, not proof that a reader encountered a route or possesses a general personality trait. Arbitrary game-state variable names are not yet cross-world preference labels; meaningful action instrumentation needs explicit semantics and server-confirmed opportunity/outcome contracts.

Published source: OpenRouter structured output support and provider routing.

Verification evidence

• The affected hosted server suite passed 672 tests with eight environment-gated integration skips. Those skips include live ClickHouse/Redis checks; the local suite is not a production load test. A final focused run of the changed story, worker, preferences, admission and session paths passed 298 tests without skips.
• The affected client suite passed 107 tests. Temporal dataset/evaluation tests passed 134 tests. Hosted typecheck and build passed; the client build retains its existing large-chunk warnings.
• Full generated OSS verification passed, including its build/typecheck and tests; the exported server suite passed 1,296 tests with two skips. The explicit hosted-discovery boundary check also passed. No browser verification was run.
• Independent reviews and scoped follow-ups covered source extraction, reader interests, worker concurrency/provenance, preferences/identity/erasure, serving continuity, UI orchestration and temporal evaluation. Review regressions were fixed with targeted tests.
• A two-model extraction pilot used eight real public worlds and exposed remaining semantic coverage gaps. Both final runs returned valid descriptors, but neither model was promoted.

The subsequent readiness execution applied the schema in isolated testing storage and exercised real workers, pgvector, HTTP and Railway Redis. Its frozen 24-world benchmark exposed weak facet coverage; it did not promote a model. The managed ClickHouse migration suite subsequently passed nine checks with isolated cleanup and a restricted migration identity. Production capture/CDC, semantic quality, full origin capacity and controlled reader outcomes remain gates. No production schema, flags or ranking model were changed.