Discovery dataset and offline evaluation — 2026-09-19

This slice builds reproducible datasets from canonical discovery event exports and verified play intervals. It does not read a database, load credentials, call a network service, train a model, publish a model, or modify trainer/train.py. Python 3.10+ and its standard library are sufficient; verification used Python 3.11.15.

Current versions are discovery-dataset-v3 / new-discovery-story-utc-d7-v3. V3 adds verified consumer story-return attribution (including new Library chats), frozen delivery snapshots, server hide/undo state, and original-actor diagnostics. Page coverage and mandatory session/receipt provenance remain. Rebuild artifacts for this contract; the evaluator rejects older manifests. Outcome windows remain unchanged.

Files and interfaces

• trainer/discovery_data.py: bounded JSONL ingestion, disk-backed joins/deduplication, attribution, labels, chronological isolation and artifact manifest.
• trainer/discovery_evaluation.py: checksum-verified descriptive evaluation of delivered order. Defaults to validation; test requires an explicit final-readout flag.
• trainer/test_discovery_data.py: dataset and evaluator regression tests, including both CLIs. Fixtures use temporary local files only.

from trainer.discovery_data import BuildConfig, build_dataset
from trainer.discovery_evaluation import evaluate_dataset

manifest = build_dataset(
    events_path, intervals_path, new_output_directory, BuildConfig(**configuration),
    dimensions_path=optional_world_group_history_path,
)
report = evaluate_dataset(new_output_directory, split="validation",
                          label="qualifiedNewPlay", k=24)

From the repository root:

python -B trainer/discovery_data.py --events events.jsonl --intervals intervals.jsonl --config discovery-config.json --output dataset-v3
python -B trainer/discovery_data.py --events events.jsonl --intervals intervals.jsonl --dimensions world-groups.jsonl --config discovery-config.json --output dataset-v3-with-groups
python -B trainer/discovery_evaluation.py dataset-v3
python -B trainer/discovery_evaluation.py dataset-v3 --label newSave --k 24
python -B trainer/discovery_evaluation.py dataset-v3 --split test --allow-test
python -B -m unittest trainer.test_discovery_data -v

The two builder examples require different fresh directories. Outputs are never overwritten. Evaluation prints JSON and does not mutate the dataset. Treat --allow-test as the one final readout after choices are frozen, not as an automated selection gate; the CLI cannot prevent someone from repeatedly requesting it.

Configuration and feature contract

The config JSON maps directly to BuildConfig. This is a synthetic two-feature example, not the production feature schema:

{
  "as_of": "2026-03-01T00:00:00Z",
  "train_start": "2026-01-01T00:00:00Z",
  "train_end": "2026-01-10T00:00:00Z",
  "validation_start": "2026-01-19T00:00:00Z",
  "validation_end": "2026-01-20T00:00:00Z",
  "test_start": "2026-01-29T00:00:00Z",
  "test_end": "2026-02-01T00:00:00Z",
  "feature_schemas": {
    "synthetic-features-v1": ["affinity", "score_semantic"]
  },
  "delivery_feature_schemas": {
    "synthetic-features-v1": ["cost", "creator_repeats"]
  },
  "lateness_seconds": 86400,
  "qualified_play_seconds": 300,
  "interval_product": "main-app",
  "allow_legacy_intervals": false
}

Supply the declared feature-name list for each real featureVersion, currently discovery-features-v1. There is no requirement for exactly 29 names: declare all model inputs plus the score_* scorer inputs in that snapshot version. Every snapshot must have exactly its version's declared names, with finite numeric values; booleans, strings, vectors, nested values, missing names, undeclared extra names and unknown versions are rejected. A consistent superset is accepted by declaring that full superset. Schemas are stored in the manifest; no features are invented, zero-filled, selected from test data or joined from current user/world statistics.

delivery_feature_schemas defaults to {} and declares a separate, optional numeric snapshot per existing feature version. A served payload may omit deliveryFeatures; when present it must be an object with exactly the declared delivery names and finite numeric values. Explicit null, undeclared names, booleans and nested values are rejected. This preserves delivery-time diversity inputs without mixing them into earlier ranking features. The current server fixture declares 136 ranking inputs and 19 delivery inputs; neither count is hardcoded. Its delivery names are queue_position,pre_diversity_position,post_diversity_position,page_position,creator_repeats,topic_repeats,consecutive_creator,creator_weight,topic_weight,consecutive_weight,cost,queued_count,eligible_count,ineligible_count,excluded_seen_id_count,excluded_seen_group_count,history_size,seen_id_count,seen_group_count. Declare the full list for discovery-features-v1 when consuming that snapshot.

Optional deliveryMetadata is the frozen object {creatorId,topic,languageGroupId} with exactly those keys. creatorId is a nonempty bounded string; topic is a string of at most 512 characters and may be empty; nullable languageGroupId must equal the serving event's group. No current metadata is loaded or joined. Missing optional delivery objects export as null; malformed present objects reject the snapshot. Metadata from outcomes cannot overwrite the serving snapshot. Ranking time remains rankedAt; these separate delivery fields describe the actual serving decision at occurredAt.

Other config defaults: success_window_seconds=86400, d7_start_seconds=604800, followup_seconds=691200, max_interval_seconds=90, max_receipt_clock_skew_seconds=5, max_line_bytes=1048576, max_input_rows=10000000 per input, sqlite_cache_mib=32, max_sqlite_mib=4096. The first three windows are fixed by label version v3; changing them requires a new label definition, not silently relabeling a 24h or D7 field. qualified_play_seconds is configurable for predeclared sensitivity runs, each in a fresh directory with a different manifest. Five minutes is a working engagement threshold, not a validated satisfaction threshold. allow_legacy_intervals must be a JSON boolean; a string such as "true" is rejected.

Canonical input

Events are one JSON object per line, in any order, using the requested camelCase fields:

id,eventType,occurredAt,receivedAt,actorId,userId,visitId,feedRequestId,opportunityId,worldId,languageGroupId,position,policyVersion,featureVersion,modelId,payload.

userId, languageGroupId and modelId may be null; canonical candidate modelId is an integer. Candidate position is the delivered position, independent of the queued snapshot's rankPosition; page events use the sentinel fields below. IDs/version strings are bounded to 512 characters. All timestamps need explicit timezones and are normalized to UTC integer microseconds. Equivalent Z, offset and fractional-second encodings deduplicate. JSON duplicate keys, non-finite numbers and oversized lines are rejected.

Use the database's canonical receipt timestamp for receivedAt. If a mirror supplies sourceReceivedAt, that field takes precedence and is normalized to receivedAt before deduplication; warehouse ingestion time must not become a different event. Input hashes still cover the exact original bytes. A maximum five-second occurrence/receipt inversion is accepted and counted, matching bounded client clock skew and accommodating database transaction-start receipt timestamps. Both timestamps must be at or before as_of. Set the tolerance to zero for exports that guarantee strict receipt ordering.

Type	Interpretation
page	Canonical page coverage; worldId="", languageGroupId=null, position=-1, modelId=null. Payload is cardCount and measurementStatus; see the exact page contract below. Extends visit exposure timing but never becomes a candidate.
served	Immutable payload.rankedAt, features, candidateSource (canonical nonempty string array; a single string is also accepted), rankPosition, variant, and wasKnownAtServe (true, false, or null). rankedAt <= occurredAt; older queued snapshots are valid. Optional featuresAsOf must be at or before rankedAt.
impression, click, play_intent, save_intent, preview_dwell	Observations only. Stored as per-opportunity presence flags, never as successful discovery or active-play time. Client play must already be normalized by the canonical server exporter to play_intent; legacy raw play is rejected.
save_confirmed, session_started	Must match the original opportunity's visit, request, world/group, position and versions. Require signed-in actorId=user:ID, matching userId, payload.authoritative=true, originActorId, actualWorldId, boolean isNewStory, and sessionId for starts.
dismiss_confirmed, dismiss_undone	Verified negatives, requiring the same original context, signed-in identity, authoritative=true, originActorId, verified actualWorldId, and nonempty dismissalId. An undo must match its original hide mutation, opportunity, user and actual world, and occur at or after the hide. Never creates a positive.
identity_link	Requires guestActorId, userActorId, signed-in event identity, and basis="same-browser-confirmed-outcome". A broad login link does not authorize attribution. Conflicting guest-to-user mappings fail closed.

Authoritative isNewStory is checked against the source snapshot and retained Library/session state as of serving, before the mutation. A save followed by a start remains the same new episode and both confirmations can be true. The builder never changes a false confirmation to true by inspecting another event. Primary eligibility also requires served.payload.wasKnownAtServe is false. Null knownness remains null and ineligible for both guests and signed-in users, even when a later server confirmation says new. Their zero-valued labels are placeholders, not eligible negative examples. The server only claims complete history for accounts created after durable ledger capture began; older accounts stay unknown rather than becoming false-new. The builder neither reconstructs that history nor treats an absent record as proof of newness. A null/nonboolean isNewStory confirmation is rejected; it cannot establish a positive.

Guest bridging requires the explicit link's occurrence time to be no later than the outcome, an exact origin match and a receipt no later than label maturation. A link emitted at the same instant as the first bridged outcome is valid. Original event/sample actor and nullable user IDs are preserved. Identity links are used separately for conservative episode isolation; they never rewrite past features or guest knownness.

The builder trusts canonical source provenance and the server's authoritativeness/newness assertions. JSON cannot independently prove a server transaction occurred. Do not feed client beacons or client-asserted outcomes into this export.

Exact page contract and visit interpretation

Use eventType="page" with all 16 canonical fields. id is the canonical event ID. opportunityId must equal discoveryEventId("page", feedRequestId): lowercase SHA-256 hex of UTF-8 compact JSON ["page",feedRequestId], matching the server hash helper. It is verified, not interpreted as a candidate opportunity. Required sentinels are worldId="", languageGroupId=null, integer position=-1, and modelId=null. Keep the serving actor/user, visit and request IDs, page occurrence/receipt timestamps, and page policy/feature versions. Page feature versions need not appear in the candidate feature_schemas declaration because a page contains no model features.

Payload decisions:

measurementStatus	cardCount	Meaning
empty	Exactly integer 0	A known empty delivered page.
recorded	Positive integer	Every delivered card is expected to have a matching valid serving snapshot.
unavailable	Nonnegative integer, including 0	Measurement could not supply the necessary snapshots; never certify this as an empty/negative visit.

cardCount must be a safe integer in 0..2^53-1; booleans are invalid. A page is immutable per feedRequestId: identical retries with the same or different event IDs deduplicate; conflicting or invalid sibling records prevent that visit from being certified complete. Page-only visits participate in chronological boundary checks and full label maturation exactly as candidate visits do. A page received after its own maturity deadline is rejected. No candidate features, candidate rows or success labels are fabricated from a page.

visits.jsonl adds pages, emptyPages, recordedPages, unavailablePages, reportedCards, missingSnapshotCards, measurementComplete, and genuinelyEmpty. For a page-backed visit, completeness requires matching page/candidate visit, request and actor/user context, exact declared snapshot counts, coverage of every candidate page, and no orphan observations. Page versions are intentionally independent of candidate versions. Invalid/conflicting pages, unavailable pages, count/context mismatches, or unaccounted candidate pages/observations make measurementComplete=false. missingSnapshotCards sums deficits against context-matching valid snapshots; an overcount also makes the visit incomplete.

An all-empty, complete visit has genuinelyEmpty=true and eligibleForEvaluation=true, with zero opportunities and zero discoveries. It contributes a zero to the visit success-rate denominator, not a synthetic candidate or NDCG slate. This is a directly observed lack of delivered candidates and does not assert anything about the actor's prior story history. Unavailable/incomplete visits remain in output and coverage but have eligibleForEvaluation=false, even if some candidates were observed. Complete nonempty visits still require at least one candidate with known-false serving-time knownness to enter the eligible visit denominator; unknown stories are never promoted by page coverage.

Older exports without any page event retain the existing opportunity-based visit eligibility, but measurementComplete, reportedCards, and missingSnapshotCards are null. They cannot certify full page coverage or a genuinely empty visit. New canonical exports should emit one page event for every delivered page, including empty and unavailable cases.

Intervals and translations

Interval JSONL fields are id,userId,worldId,startedAt,endedAt,product,sessionId,receivedAt,consumerEligible, from verified analytics_play_intervals. The last three fields are optional only under explicit legacy opt-in. The canonical source permits null historical values; the strict exporter excludes those rows. New sessionId values must be verified real sessions, and receivedAt must be the original database insertion time from the clock_timestamp() insert trigger. consumerEligible=true is captured inside the owned-session/heartbeat transaction after checking the locked session is user-owned and the world is published, noncreator and non-ephemeral. Do not backfill historical receipt times or derive a session from an opaque interval ID. The builder never does either. A mirror may supply the same canonical time as sourceReceivedAt; that alias takes precedence and normalizes to receivedAt, just as for events.

Strict mode is the default (allow_legacy_intervals=false):

• All three provenance fields are required: sessionId (nonempty string), receivedAt (explicit-timezone timestamp), and exact JSON boolean consumerEligible=true. Missing/null fields produce interval_provenance_missing; false eligibility and malformed provided values are rejected in every mode.
• Every episode needs a first attributable eligible authoritative new session_started in the original exposure's first 24h. A matching confirmed session binds user and exact actual world and clips time at that session's own start; it may disambiguate overlapping opportunities. Session identity contradictions cannot fall back to a story join.
• A different verified session, including a new chat started from Library on D7 with no discovery session_started event of its own, may contribute after that anchor when exactly one user/story opportunity matches. This is based on verified interval ownership/consumer eligibility plus known story identity, not an arbitrary join or all-time session count. Two possible opportunities reject the interval as ambiguous_interval_attribution.
• Only the configured consumer product (main-app by default) is accepted. startedAt < endedAt, duration is at most 90 seconds by default, and endedAt and receivedAt must be at or before as_of. Receipt may precede the interval end only within the declared clock-skew tolerance. It must also satisfy receivedAt <= original exposure + 8d + lateness_seconds; equality is allowed, and no tolerance extends this maturity deadline.
• Each interval is clipped at its matched session's confirmed start (session attribution) or the original first new start (story attribution), and at exposure+8d. Qualification uses only the portion before exposure+24h. Multiple confirmed or Library-created sessions can contribute to one opportunity; all their time is unioned. A session ID claimed by different users/worlds in confirmed starts or validated interval rows fails closed. Interval identity-conflict evidence received after a label's maturity cannot revise that label. Conflicting canonical session confirmations are quarantined structurally.

allow_legacy_intervals=true explicitly permits missing/null provenance fields. Any provided field remains binding: malformed/empty values, false consumer eligibility, conflicting session identity and late receipts are rejected. Failed strict matching never downgrades to legacy. A supplied session in a weaker row must match a confirmed start exactly; only fully strict rows may attribute a different Library-created session by story. Without a session field, weaker attribution requires a unique anchored user/story/opportunity. Without a receipt field, historical arrival completeness remains an upstream assumption; without the consumer flag, eligibility remains an upstream assumption. This weaker mode is recorded in config and semantics.intervalMode="legacy-opt-in"; the default is "strict".

A save alone never authorizes play time. Exact interval IDs deduplicate; conflicting IDs are quarantined, retaining validated identity claims so dropping one conflicted ID cannot conceal a session ownership contradiction. All overlapping/touching intervals for an opportunity are unioned before seconds or dates are counted, including overlapping verified sessions and duplicates with different IDs. Per-candidate intervalAttribution={session,story,legacy} contains accepted interval counts before union. Manifest attributed_strict_intervals, attributed_session_intervals, attributed_story_intervals and attributed_legacy_intervals report provenance; missing-field counters report coverage. These counts are not seconds or distinct days. There is no parsing of interval IDs, world IDs or session IDs.

The original story identity is languageGroupId when present, otherwise worldId. Exact-world outcomes and intervals need no dimension export. A translated confirmed actualWorldId, or story-attributed intervals in another translated world, require explicit historical membership:

{"worldId":"translation-id","languageGroupId":"group-id","validFrom":"2026-01-01T00:00:00Z","validTo":null}

Use --dimensions for these JSONL rows. languageGroupId is required but nullable: null means observed ungrouped and never synthesizes translation membership. Validity timestamps preserve microseconds. Membership must cover exposure time, not the potentially earlier queued ranking time, and match the served group's metadata. validTo is exclusive. Overlapping/conflicting dimension histories abort the build. Future-effective membership cannot join a past exposure. This input only resolves identity for labels; it never modifies serving features. Missing or unverified translation membership rejects attribution and is counted. Historical baseline capture cannot establish fabricated prebaseline membership.

Provenance limit: the strict path enforces all three verified fields and their attribution constraints but cannot independently authenticate a JSON export or prove source completeness. Product alone does not prove consumer eligibility. Legacy opt-in explicitly gives up session precision, enforceable arrival timing and/or verified consumer eligibility where fields are missing. Strict and legacy artifacts must not be presented as equivalent evidence.

Verified negative state

Hide/undo events must occur in [exposure,exposure+8d) and arrive by label maturity. The output negatives object has everDismissed, finalHidden, dismissalCount, undoCount, and stateKnown. Counts refer to distinct verified original dismissal mutations, not event retries. No events yields false/false/0/0/true. Undoing a matched hide clears that mutation; another still-active mutation keeps finalHidden=true. Unmatched, wrong-owner/world/opportunity, pre-hide, pre-exposure, late and nonauthoritative undos cannot clear a valid hide. Conflicting negative event IDs or reused mutation IDs mark the affected state unknown (stateKnown=false, finalHidden=null). Unknown state must not be interpreted as no dismissal.

These are negative diagnostics separate from positive labels; hide and undo never create saves, qualified play or return days. An independently verified positive can coexist with a dismissal. isNewStory in a negative payload is not used to change eligibility or positive labels. State is final only within the fixed observation window, not an all-time current hide lookup.

Labels and maturity

Let t = served.occurredAt (delivery exposure, not proof of viewport visibility). All ranges are half-open.

Output label	Definition
newSave	At least one qualifying authoritative new save in [t,t+24h).
qualifiedNewPlay	Eligible authoritative new start in [t,t+24h) plus at least the configured unioned active seconds in [start,t+24h).
activeSeconds24h	Unioned, attributed play seconds from the first qualifying start through t+24h.
activeSecondsFollowup	Unioned attributed seconds through t+8d.
returnUtcDays	Number of distinct UTC dates with positive attributed duration after the first start's UTC date, through t+8d, conditional on qualified new play. Session counts and repeated intervals on one date do not increase this value. Midnight endpoints with zero duration on the next day do not create another day.
d7Continuation	Qualified new play plus positive attributed duration in [t+7d,t+8d). This is an elapsed D7 window, distinct from the UTC-date convention for return-day counts.

Even a row with an immediate positive outcome waits until its entire D7 followup has matured. labelWindowEnd=t+8d; labelMaturesAt=t+8d+lateness_seconds. as_of must be a caller-attested complete-through watermark for both input sources, not merely the time a command was run. Events, observations, links and strict intervals received after an opportunity's maturity deadline cannot supply its labels/diagnostics. Late corrections are counted, not backfilled into earlier splits. Only explicitly permitted legacy intervals missing receipt time rely on an upstream historical-arrival assumption.

Temporal isolation

Train, validation and test are explicit, chronological half-open exposure intervals. Both gaps must be at least 8d + lateness_seconds. There is no random split. A visit whose page/serving/observation timestamps cross a split/gap boundary is excluded in full. A visit is also excluded until its latest such timestamp has full followup; this is deliberately conservative. Outcome timestamps do not extend a visit's exposure range.

Within a resolved actor/story, connected eight-day attribution windows form an episode. Opportunities in the gaps participate in connectivity, so a chain cannot bridge train to validation or validation to test unnoticed. Any episode touching multiple splits excludes all of its visits. Confirmed guest identity links can only make this exclusion stricter, including links learned after the original guest exposure. All-zero opportunities participate too. This is episode isolation, not an assertion that test actors are unseen during training.

Artifacts, audit and bounds

A completed directory contains:

• train.jsonl, validation.jsonl, test.jsonl: one valid mature served opportunity per row, ordered by exposure time then opportunity ID. Includes original identity/context, frozen features, optional deliveryFeatures/deliveryMetadata, ranking metadata, eligibility, observations, negatives, intervalAttribution, label-window timestamps and labels. Ineligible known/unknown stories remain visible with newDiscoveryEligible=false.
• visits.jsonl: every in-split mature visit observable from accepted pages/serves/observations, including genuinely empty, missing-snapshot, all-zero and orphan impression-only visits. Base fields: visitId,actorId,userId,actorIdentityConsistent,split,startedAt,endedAt,opportunities,eligibleOpportunities,qualifiedNewStories,savedNewStories,eligibleForEvaluation; page fields are listed above. Original actor/user comes from the earliest exposure event (timestamp then ID), including pages. A visit with multiple actor/user pairs has actorIdentityConsistent=false and is excluded from actor diagnostics, without rewriting identities or hiding the visit. Story counts deduplicate language groups. A visit without valid candidate snapshots is ineligible unless its pages certify it is genuinely empty.
• manifest.json: completion marker, dataset/label versions, builder source SHA-256, exact config/schema/windows/provenance mode, input byte hashes/sizes/line counts, output byte hashes/sizes/row counts, candidate versions, separate pageVersions, coverage, rejection counters and interpretation rules. Input paths, wall-clock generation time and random temporary paths are omitted, making identical inputs/config/code reproducible.

Coverage includes duplicate records/pages, accepted event types, attributed outcomes/strict/legacy intervals, missing interval provenance, exported/eligible opportunities, valid/guest/unknown/known snapshots, linked guest confirmations, orphan observations, zero-label slates and visits without valid snapshots. empty_pages, recorded_pages and unavailable_pages describe valid unique source pages before temporal filtering. page_backed_visits, genuinely_empty_visits and measurement_incomplete_visits describe final exported visits. Guest/snapshot counters also describe valid source opportunities before temporal filtering. Manifest zero_label_slates counts exported candidate slates with neither a new save nor qualified new play, including ineligible slates; genuinely empty pages create no candidate slate. Evaluation reports eligibility exclusions separately. Rejection counters describe stages and can overlap; their sum is not a unique rejected-event denominator. Zero-valued or absent reason counters both mean no occurrences. Conflicting rows are excluded without retaining sensitive raw payloads in a quarantine output.

Deduplication uses a disk-backed SQLite primary key for every event and interval ID. Conflicting event IDs, conflicting snapshots for one opportunity and invalid sibling snapshots fail closed. Ordinary mirrored retries, equivalent timestamp encodings and duplicate served snapshots do not multiply rows. Observation flags and boolean outcomes cannot be inflated by repeated events with different IDs. Interval union handles overlapping IDs independently of event deduplication.

Input reading is line-bounded and drained incrementally even for an oversized line. SQLite indexes and temporary sorts provide out-of-core joins and ordering; Python does not accumulate the full dataset, all actor histories, all intervals, or whole slates. The return-day set is bounded by eight days. Row, line, SQLite page/cache, feature/schema and metadata-cardinality limits stop pathological builds. The database page limit is not a filesystem quota for its journal/temp-sort files or final outputs; provision local scratch/output space. Large exports may take time despite bounded memory. Temporary SQLite files are removed on exit. Failed builds have no manifest and cannot be reused as successful artifacts; choose a fresh output directory for a retry.

Hashes detect later file changes; the artifact directory is not a tamper-proof store. Archive completed directories with their manifest and make them read-only in the downstream storage policy. No staging, committing, production deployment or external storage write is performed here.

Evaluation and remaining limitations

The evaluator verifies the selected split and visits file against the manifest. It sorts recorded slates by delivered position, using opportunity ID to break ties; queued rankPosition is retained as metadata but is not the delivered-order baseline. Ineligible candidates are removed from the metric ranking; excluded counts and visits remain visible. This measures the recorded eligible-candidate subset, not a replay of unseen retrieval candidates or a counterfactual policy.

Supported binary labels: qualifiedNewPlay (default), newSave, d7Continuation. The report includes observed/eligible opportunities, slates and visits, selected-label successful-visit rate, zero-label slates, pageBackedVisits, genuinelyEmptyVisits, measurementIncompleteVisits, ineligibleVisits, no-eligible slates/visits, source manifest hash and two clearly named NDCG@k means. visitsWithoutEligibleCandidates counts zero-eligible-candidate visits independently of denominator eligibility: a certified empty visit belongs to that count and to eligibleVisits. Missing-snapshot visits do not enter the primary visit denominator. NDCG remains a diagnostic over observed eligible candidates and does not claim full-page coverage for incomplete visits.

• ndcgIncludingZeroSlates: eligible zero-positive slates contribute zero by explicit convention, even though their ideal DCG is zero.
• ndcgPositiveSlatesOnly: separate diagnostic, never the primary denominator.

Actor diagnostics use the selected label and eligible, identity-consistent visits, including certified empty visits as zeros. eligibleActors counts distinct original actors; actorMeanVisitSuccessRate gives each actor equal weight after calculating that actor's successful-visit rate. actorMeanDistinctSuccessfulStories averages each actor's distinct successful language groups (world fallback) across eligible visits. actorExcludedVisits counts eligible visits with mixed identity excluded from these diagnostics. For ten successful visits by actor A and one empty visit by actor B, the visit rate is 10/11 and the actor mean is 0.5. Explicit guest bridges do not merge metric identities retroactively; these are original-actor descriptive metrics, not unique-person or causal estimates.

Empty denominators produce JSON null, not NaN or a fabricated success. causalPolicyUplift is always null. There is no AUC, propensity estimate, confidence interval, model fitting, policy selection or publication gate. Frozen creator/topic delivery metadata is retained for later diagnostics; this evaluator does not calculate creator/topic coverage. No logged randomized assignment probabilities, full retrieval candidate sets or satisfaction measures are present. Retrieval recall and causal incremental lift require additional inputs and an appropriate experiment. Test is exported separately and not evaluated by the builder.

Integration requirements/limits for the controller:

1. Export canonical IDs, database receipt times, authoritative outcome flags, exact original context and the complete declared per-version numeric feature-name superset. Current server integer model IDs and source arrays are supported. Do not merge legacy feed events into this stream.
2. Supply complete-through watermarks plus retained source coverage across exposure, followup, embargo and linked episodes. Omitted gap/history rows can conceal connected episodes; a JSONL reader cannot prove a complete upstream export.
3. Supply exposure-valid world/group history for translated attribution; otherwise those outcomes fail closed. This needs no current-stat/feature join.
4. Export verified real sessionId, original database receivedAt, and exact consumerEligible=true for strict intervals; exclude null historical provenance and never backfill receipt times. Preserve transactional consumer/session verification. Story-D7 includes new Library chats after the unique original new-start anchor. Any deliberate weak historical run requires allow_legacy_intervals=true and retains weaker-provenance counts.
5. Emit canonical page events for recorded, genuinely empty and unavailable pages with the exact sentinels/hash/payload above. Completely unlogged visits remain unobservable. Page events preserve missing-snapshot visits without manufacturing candidates or negatives.
6. Keep original wasKnownAtServe=null for accounts whose pre-ledger history cannot establish newness. Do not replace unknown history with false. An authoritative new confirmation must never override unknown serving-time knownness.
7. Export frozen optional delivery objects and declare their delivery feature schema separately. Export canonical server negatives with the original dismissalId on undo; never substitute a new undo-request ID for the original mutation ID. Builder output/label versions are now v3; the unshipped server feature version can remain discovery-features-v1 with its complete declared schemas.

Verification evidence

python -B -m unittest trainer.test_discovery_data -q — 86 passed, no failures, on Python 3.11.15 after the v3 integration. This includes red-to-green regressions for the new contracts and corrupted negative/interval ID provenance. Local AST/whitespace checks passed and git diff --exit-code HEAD -- trainer/train.py confirmed the legacy trainer was unchanged.

A separate local streaming smoke check generated 5,500 event lines (5,000 opportunities plus 500 mirrors). The builder exported exactly 5,000 opportunities and 5,000 visits; evaluation retained all 5,000 zero-label slates and reported zero NDCG and zero successful-visit rate. The run used temporary files and no services. This is a correctness/streaming smoke check, not a production capacity benchmark.

The v2 streaming smoke check then processed 5,300 canonical events and 5,000 strict intervals, including 300 mirrored page events. It exported 1,000 candidate opportunities and 3,000 visits: 1,000 successful recorded visits, 1,000 certified empty visits and 1,000 unavailable visits. The evaluator correctly used 2,000 eligible visits, reported a 0.5 visit success rate, and kept the unavailable visits visible outside that denominator. Every attributed interval had verified session/receipt provenance. This also used temporary files and no services.

The v3 integration smoke check consumed the controller's actual snapshot fixture (136 ranking names, 19 delivery names) with 3,000 canonical events and 3,000 strict consumer intervals. It exported 500 candidates and 1,500 visits, attributed 2,500 intervals by confirmed session and 500 Library-chat intervals by story, and measured 500 D7 successes. Including 500 certified empty visits and excluding 500 unavailable visits gave 1,000 eligible visits, a 0.5 visit success rate, a 0.5 equal-actor rate, and one distinct successful story per eligible actor. Artifacts stayed in temporary local storage.

The regression suite covers mirrored events and timestamps; ID/snapshot/page conflicts; page-only empties, unavailable and partially missing snapshots; page sentinels/hash/status validation and temporal isolation; strict/missing/null/partial interval provenance, source receipt mirrors and opaque IDs; exact session/world matching and per-session clipping; multiple sessions unioned within one episode; legacy opt-in and rejection without downgrade; pre-exposure, non-authoritative, late and wrong-actor outcomes; intents/dwell not being success; frozen/schema-valid feature supersets; numeric models and queued snapshots; union/clipping/product constraints; full D7 censoring; distinct UTC return dates; midnight boundaries; authoritative save-to-play newness; unknown guest and signed-in account history, same-time outcome bridges and rejected login/late links; nullable microsecond translation history; visit/episode leakage through gaps; untouched-test evaluation gating; zero-label/empty-visit denominators; checksums; reproducibility; malformed/bounded input; and both CLIs. V3 also covers strict true/false/null/malformed consumer eligibility, new Library-chat D7, story ambiguity, contradictory session identities (including quarantined IDs), post-maturity conflict evidence, frozen delivery features/metadata, hide/undo/rehide mutation binding and corrupt negative state, original actors on empty pages, mixed-identity exclusions, and equal-actor metrics.

Tests are local and deterministic. They are not proof of production export completeness, real event coverage, serving quality, satisfaction, or causal uplift. The legacy trainer and its publication workflow remain outside this slice.