Offline personalization evaluation

Task 5 adds trainer/personalization_evaluation.py and its unittest suite. It consumes a completed, immutable discovery dataset, fits small experimental models using training rows only, and compares them on validation. It does not change train.py, connect to a service, reconstruct historical features, or publish a production model. Python 3.10+ and the standard library are sufficient.

Run locally

Run from the repository root with an existing output directory. The paths below are examples; supply a completed local dataset whose source watermarks and provenance have been reviewed.

python -m unittest trainer.test_discovery_data trainer.test_personalization_evaluation -v

python -m trainer.personalization_evaluation fit .local-artifacts/discovery-dataset-2026-09-19 --label qualifiedNewPlay --k 24 --output .local-artifacts/personalization-2026-09-19.personalization.json

The default label is qualifiedNewPlay. The only other labels are newSave and d7Continuation, under the existing new-discovery-story-utc-d7-v3 contract. The artifact records the label, cutoff, complete source configuration, verified file checksums, implementation hash, Python version, fit settings, exact feature order, model coefficients, cohort support and validation diagnostics. Identical bytes, code, Python runtime and arguments produce identical JSON; neither wall-clock timestamps nor temporary paths enter the result. The dataset's explicit UTC timestamps identify the experiment period.

After choosing and freezing the development experiment, consuming test is a separate explicit final action:

python -m trainer.personalization_evaluation final .local-artifacts/discovery-dataset-2026-09-19 --frozen .local-artifacts/personalization-2026-09-19.personalization.json --allow-test --output .local-artifacts/personalization-final-2026-09-19.personalization.json

Final evaluation has no label, feature-selection, cutoff or fitting overrides. It validates the frozen experiment fingerprint, exact model whitelist, implementation hash and source manifest hash, then scores test with the saved coefficients. It never refits or reads train.jsonl/validation.jsonl. The final artifact references the frozen file's SHA-256 and marks test consumed. A local tool cannot enforce one-time use against copied files or a determined operator: repeated test inspection contaminates the scientific holdout. Do not use final results to tune and then describe the same test as untouched.

Temporal and integrity contract

The extension accepts discovery-dataset-v3, validates BuildConfig, and rechecks chronological half-open train/validation/test intervals. Both gaps cover the full eight-day follow-up plus declared lateness. Each consumed candidate must belong to the declared exposure split, have rankedAt <= exposedAt, carry the exact label-window/maturation timestamps, and mature by the source as_of watermark. Each consumed visit must fit wholly within its split and mature through its final timestamp. Duplicate opportunities, inconsistent visit counts, visits/feeds crossing splits, invalid actor consistency and malformed labels/eligibility fail closed.

For supported snapshots, ranked_at_ms must agree with rankedAt; candidate update time and known aggregate-stat update time cannot lie in the future relative to ranking. These checks help detect future joins but cannot authenticate exported JSON or independently prove that every supplied feature and source watermark is truthful. The dataset builder's episode isolation, historical family attribution and verified outcome semantics remain upstream requirements. Original actor identities are preserved; guests are never retroactively rewritten here.

Development verifies complete train.jsonl, validation.jsonl and visits.jsonl hashes/byte sizes/row counts before fitting, then parses the exact verified bytes without reopening them. It does not open test.jsonl. visits.jsonl is an existing shared file: its bytes, including held-out summaries, are verified and decoded, but test summaries are excluded before validation support, fitting or diagnostics. Final evaluation verifies test.jsonl and this shared visit file. Checksums of files not opened are declarations pinned by the manifest, not claims of independently verified bytes. There are no current-database joins, remote reads, timestamp randomization or test-driven fitting.

Snapshot schemas and predictor whitelist

The module contains a frozen full snapshot schema, separate from its much smaller predictor whitelist:

Snapshot	Validation and availability
discovery-features-v1	Exact original 136 finite numeric fields; usable for the broad baseline. Pair features are absent.
discovery-features-v2	Exact 147 finite numeric fields: previous 136, eight reader pair features and three availability/policy diagnostics.
Other declared versions	Validate against their declared schema, count and exclude the whole affected eligible visit. No inferred compatibility.
Missing/undeclared version, wrong known schema, extra/missing field, bool/string/null/nonfinite feature	Reject the artifact. No silent coercion or zero filling.

The fixed baseline whitelist is:

log1p_imp7d ctr_lift_ln log1p_plays7d qualified_rate_sm velocity_ln
log1p_imp_total log1p_favorites log1p_messages log1p_downloads
age_days_capped freshness has_audio custom_ui_tier rating_conf
log1p_total_tokens user_ui_pref user_audio_pref user_log_token_pref
user_craft_known candidate_review_count candidate_average_rating

The fixed reader pair block is:

personalization_content_match
personalization_session_match
personalization_experience_match
personalization_explicit_match
personalization_experience_affinity
personalization_support_families
personalization_cluster_count
personalization_hidden

The new raw candidateSource="personalized" (or a source-list entry) needs no numeric source_personalized field. It remains provenance only, is already accepted by the existing builder, and never changes the 136/147-field contracts.

Historical statistics and craft preferences above are used only as captured at ranking. Outcomes, observations, post-exposure dismissals, identifiers, timestamps, positions, all rank_*/score_* components, rank_final_score, policy/treatment/source flags and delivery fields are never predictors. personalization_available gates availability; personalization_applied and personalization_score are diagnostics only. The current server may record an enabled policy (applied=1) for an unavailable candidate (available=0); that is valid, but never qualifies the pair features for learning. Changing prohibited fields alone cannot affect learned coefficients or predictions, as tested.

Ablations and fitting

Model	Predictors	Train/evaluation cohort
baseline_all	Baseline whitelist	All eligible visits with supported complete snapshots, including v1. Context only.
baseline	Baseline whitelist	Matched pair-available visits.
content	Baseline plus personalization_content_match	Same rows as baseline.
multiinterest	Baseline plus all eight reader pair fields	Same rows as baseline.

Only exact newDiscoveryEligible=true candidates from eligibleForEvaluation=true visits can train or enter ranking metrics. Known/unknown prior-story state is never fabricated. An unavailable pair or unsupported version drops the whole affected eligible visit from the corresponding comparison so no model receives a shorter/easier slate. Ineligible candidates do not cause availability exclusions. Coverage reports versions, missing-pair rows, unavailable-pair rows, dropped visits/rows, and exact training support; a SHA-256 of ordered training opportunity IDs makes matched membership auditable. There is no imputation of historical reader profiles, clusters, vectors or pair features.

V1-only datasets remain executable for baseline_all; the matched models report unavailable. A training cohort with no rows or only one class yields no fitted model and an explicit reason. There is no fabricated constant-model win or substitution from validation/test. baseline_all may have different data and must not be presented as a matched comparison against content or multiinterest.

The model is deterministic binary logistic regression with full-batch gradient descent, L2 regularization, no randomness and no early stopping. Each predictor is centered by its training mean and divided by max(1, maximum absolute training value). Training prevalence initializes the intercept. Defaults: 100 iterations, initial learning rate 0.2, L2 0.01. --iterations permits 1–500; the callable settings accept initial learning rate and L2 in [0.01,1]. A training-only curvature bound caps every step; deterministic doubling approaches that cap. Loss must not increase beyond numerical tolerance, and the final training gradient must meet the convergence tolerance. Overflow, unstable steps or insufficient iterations reject the experiment instead of freezing a bad model. All fitted transforms and parameters use training only. Models are equally weighted per opportunity; no propensity weighting, positive-only sampling or validation-based recalibration occurs. The callable API accepts a fitter for lightweight instrumentation, validates its coefficient format/whitelist and optimizer evidence, and passes only the selected training matrix and labels.

This is a reproducible, inexpensive scientific baseline, not an empirically selected model family. Content-match may itself summarize multiple interests; this feature ablation does not prove the value of single versus multiple reconstructed reader profiles. A richer learner or a different feature whitelist would require a separately reviewed experiment/version.

Metrics and interpretation

Every evaluated model reports candidate positives and counts, slate/visit/actor support, genuinely empty visits, availability exclusions, and mixed-identity visits omitted from actor diagnostics. Rankings are computed only within recorded (visitId, feedRequestId) slates, with probability descending and opportunity ID ascending for ties. Delivered rank and labels never break ties.

• Primary NDCG includes eligible zero-positive slates as zero. Positive-only NDCG is a separately named diagnostic; MRR@k and recall@k also retain zero-positive slates.
• Per-visit records report slate-mean NDCG, observed positives, a hit among the per-page top-k, observed success anywhere, Brier score and distinct successful story families. Genuine complete empty visits contribute zero to visit hit/success denominators without creating candidate rows or NDCG slates. Missing-feature or incomplete visits never become negative visits.
• Per-actor records average visits with equal weight, and top-level actor metrics give each original actor equal weight. Mixed actor identities are excluded only from actor aggregation. Distinct story counts deduplicate recorded language groups, falling back to world IDs. This does not make repeated observations independent.
• Calibration diagnostics include Brier score, clipped log loss, ten fixed probability bins with counts/positive support, observed rate, predicted mean, and descriptive expected calibration error. ROC AUC needs both classes and otherwise has a null value and reason. Empty cohorts produce null metrics. These are uncalibrated model probabilities; a small bin or one-class cohort is not evidence of reliable calibration.

All outputs explicitly set causalPolicyUplift=null. Shown-slate selection, exposure/position effects, correlated views, repeated readers/families and omitted unseen candidates limit inference. Per-actor aggregation is descriptive, not a randomized experiment, propensity correction, clustered uncertainty estimate or proof of online discovery improvement. No IID confidence intervals or causal claims are emitted. Obtain adequately supported mature datasets and controlled online outcomes before rollout decisions.

Actor preference events and separate onboarding ITT

The canonical exporter in scripts/export-discovery-data.ts preserves all event types. Its warehouse archive/import path also preserves them; no exporter or warehouse filter change is needed. The narrow compatibility extension in trainer/discovery_data.py recognizes exactly preference_assigned, preference_offered, preference_selected, preference_skipped, preference_reset and preference_handoff. After the existing actor/payload/time checks, it validates featureVersion="preferences-v1", nonempty policy version, empty visit/feed/opportunity/world IDs, null group/model and position -1. It increments coverage.ignored_actor_preference_events and the corresponding ignored_preference_* counter, then skips the event before inserting an opportunity/visit/event used by recommendation training. Existing event-ID deduplication and conflicting-ID quarantine still run first. These are coarse counts of valid unique exported events, not user-conversion denominators; malformed actor-only records remain rejected, and unrecognized preference_* names do not acquire an exemption.

A separate onboarding intent-to-treat (ITT) report is required. Build its cohort from the original eligible actor assignments and preserve offered/control assignment regardless of selecting, skipping, resetting, later usage or availability of pair features. Deduplicate assignment/revision events; define the guest/account handoff unit and prevent double enrollment without rewriting recommendation identities. Declare assignment/exposure timing, follow-up windows, maturation/completeness rules, sample support and checks of assignment balance. Selected-versus-unselected comparisons are self-selected and cannot replace ITT. Selection, offer, skip, reset and handoff are onboarding process diagnostics, never click, newSave, qualifiedNewPlay or d7Continuation labels, candidate impressions, or automatic positive taste evidence. Reader outcomes for an ITT report need their own preregistered attribution and mature observation denominators.

This evaluator does not implement or claim that onboarding report. It deliberately discards the actor-only event payloads after counting; retain the canonical source export for that separate analysis. Parent integration requires only the small builder branch described above and its three compatibility regressions; no changes to the exporter, production trainer, server/UI, or numeric feature schema are needed from this task.

Bounds and artifact safety

Defaults cap the aggregate loaded JSONL rows at 20,000, total read bytes at 128 MiB (including the manifest), and a line at 1 MiB. CLI bounds are explicit and hard-capped at 100,000 rows, 512 MiB and 2 MiB per line; the source's smaller line limit also applies. The manifest is capped at 4 MiB. Exceeding a limit aborts instead of truncating or sampling; memory and full-batch fitting cost grow with these bounds. Artifacts are capped at 64 MiB. The evaluator has no external dependencies or package installation step.

CLI output must be outside the immutable dataset. Artifacts require the .personalization.json suffix. A completed, fsynced same-directory temporary file is atomically linked into a previously absent destination; existing files are never replaced. If hard links are unsupported, publication fails without leaving partial output; the tool does not fall back to a racy overwrite. Temporary files are cleaned on ordinary failure. This provides atomic visibility, not a backup or universal power-loss guarantee.

The JSON has a distinct offline-personalization-evaluation-v1 artifact version, productionCompatible=false, an offline coefficient model kind and exact whitelisted feature order. It contains neither the live LightGBM tree_info/feature_names schema nor a database/model-publication hook. Even a coincidentally equal feature count is not compatibility with the existing 29-feature serving model. No serialized executable model/pickle is loaded or written. Artifacts contain original actor and visit identifiers, so keep them in the same controlled local storage as the dataset.

Verification

trainer/test_personalization_evaluation.py creates realistic synthetic exports through the existing immutable builder. Test-first cycles established missing-module failures, then caught and fixed feature-vintage validation and frozen-experiment integrity gaps. Tests cover actual learning and CLI execution, injected inspection of train-only matrices, validation-label invariance, prohibited-feature invariance, test-file access guards, final no-refit behavior, schema/checksum/temporal adversarial cases, legacy and unavailable pairs, matched cohorts, incomplete/empty/mixed-identity visits, ties/calibration, bounds, atomic failure and incompatible artifacts.

The execution evidence is recorded in .local-artifacts/personalization-evaluation-report.md. All reported model metrics from these tests are fixture diagnostics, not measured product improvement. No production queries, network calls, browser, agents, commits or production-trainer modifications are part of this extension. The existing dataset builder and its tests received only the explicitly authorized actor-preference compatibility change.