Discovery warehouse/archive contract — 2026-09-19

Implemented in the codex/discovery-measurement worktree. No production migration, CDC change, import, export or prune has been run by this implementation task. Collection and warehouse readiness stay opt-in. This adds no service.

Files and interfaces

File	Interface / responsibility
packages/server/src/analytics/discovery-import.ts	importDiscoveryEvents(warehouse, now, signal, { batchSize?, maxBatches? }) returns DiscoveryImportObservation; injected Warehouse.query/insert only, no Postgres connection.
packages/server/src/analytics/migrations/008_discovery_events.sql	Immutable discovery_events archive, monotonic discovery_erased_actors, erasure-filtered source_discovery_events view.
packages/server/src/analytics/migrations/source-discovery.sql	Transactional, repeatable publication extension for both discovery source tables plus session_id,received_at,consumer_eligible on the existing interval member. Other members remain unchanged; no publication recreation, operational table creation or privilege grants.
packages/server/src/analytics/config.ts	ANALYTICS_DISCOVERY_READY=false by default; enabling it requires ANALYTICS_CDC_READY=true. analyticsMigrationFile(args) selects exactly one explicit migration.
packages/server/src/analytics/worker.ts	--apply-discovery-schema applies migration 008 only. Normal import runs under the existing worker lease, at most once per minute including failures.
packages/server/tsup.config.ts	Includes migration 008 in the deployed dist/migrations copy list; tested against every selectable warehouse migration.
scripts/export-discovery-data.ts	Read-only exportDiscoveryData(warehouse, options, signal?), parseExportArgs(args); explicit files, no .env, no app startup.
scripts/prune-discovery-events.ts	pruneDiscoveryEvents(source, warehouse, options, signal?), parsePruneArgs(args); dry-run default, exact-ID reconciliation before optional deletion.
scripts/discovery-world-history.sql	Explicit source history table/trigger, observed mapping intervals view, bounded operator baseline function; no automatic scan or CDC enrollment.
scripts/export-discovery-dimensions.ts	exportDiscoveryDimensions(dedicatedSourceConnection, options, signal?), parseDimensionArgs(args); bounded read-only repeatable-read source history JSONL.
packages/server/src/analytics/{config,discovery-import,discovery-export,discovery-prune,discovery-world-history,discovery-dimensions-export}.test.ts	Configuration, archive retry/late-row/erasure/coverage, migration, export, source retention and world-history tests.

The controller owns packages/server/src/db/schema.ts, the operational packages/server/scripts/discovery-measurement.sql, ingestion and confirmed outcomes. Other workers own client/store/offline dataset paths.

Retention and erasure agreement with the controller / Helmholtz

• Warehouse events: 180 days from received_at. The archive uses ReplacingMergeTree(version) ORDER BY id, with no timestamp in replacement identity and no month partition that could split an ID's copies. Read with FINAL through source_discovery_events. Source CDC versions remain UInt64 strings in JavaScript; event timestamps retain six fractional digits.
• Source retention: eligible after 30 days, only when reconciled per ID. No automatic job is scheduled. The explicit pruning CLI is the implementation, not a claim that production has 30-day retention enabled. Rows absent from the archive remain in Postgres. Rows older than 179 days are outside this pruning tool's safe window and need separate review; a one-day warehouse TTL margin prevents deletion at archive expiry. Unreconciled rows must never be forced out merely because storage or time thresholds were reached.
• Do not prune identity_link source events. The account-erasure worker derives linked guest actors from them. Deleting those links at 30 days would break erasure of guest events retained for 180 days. This CLI explicitly excludes identity links from selection and deletion. They remain until explicit account erasure or a separately implemented durable-link mechanism replaces their role. This is intentional low-volume metadata retention, not unbounded retention of rank snapshots.
• Erasure tombstones have no automatic expiry in source or archive. Keep at least through all associated archive/data-artifact retention and replay horizons; this implementation chooses indefinite retention of actor + cutoff metadata. The pruner never touches discovery_erased_actors. Source and archive cutoffs must only increase. Archive replacement version is derived from the cutoff itself, so replaying an older marker cannot undo erasure.
• Retention deletion and account erasure are different operations. Ordinary source CDC _peerdb_is_deleted rows are not copied as archive deletes. Explicit discovery_erased_actors(actor_id, erased_through) markers are imported first. If the marker batch budget is exhausted, events wait until a later cycle.
• User erasure is permanent; guest erasure checks both clocks. Any matching user: marker excludes a row regardless of timestamps. For guest markers, occurred_at <= erased_through OR received_at <= erased_through excludes the row, including a previously received event with a future-skewed client clock. Both rules apply to the actor, account user_id, payload.originActorId, and an explicit identity link's payload.userActorId and payload.guestActorId. They mirror the erasure worker's delete predicates. Later anonymous events survive only when both clocks are after cutoff. Erased served rows cannot enter new training exports. All intervals for a tombstoned user are excluded permanently too, irrespective of their interval timestamps. Reapplying migration 008 replaces the read view so an earlier definition cannot silently retain weaker erasure predicates.
• Ingest rejection is controller-owned. User markers permanently reject associated events; guest markers apply the occurrence/receipt cutoff rule. Archive exclusion is defense in depth, not a substitute for that check.

Erasure here is durable logical exclusion of archived copies. The physical event table may retain bytes until its TTL; this task does not claim immediate physical purging. Training/report credentials must be restricted to the filtered view and permitted interval inputs. Old JSONL exports, datasets and trained artifacts are separate copies: later erasure requires invalidation/removal under the controller's artifact lifecycle. Do not reuse an old export merely because it was valid at creation.

Coverage and limits

Every successful import writes source_cursors.source='discovery_events', cursor='archive-observation-v1', complete=0, and null coverage_from and coverage_through. The JSON note and return value include:

type DiscoveryImportObservation = {
  available: boolean;
  rowsImported: number;
  rowsScanned: number;
  batches: number;
  erasuresImported: number;
  erasuresExhausted: boolean;
  rawExhausted: boolean;
  heartbeatAt: string | null;
  sweepAfterId: string | null;
  sweepStartedAt: string | null;
  sweepCompletedAt: string | null;
  sweepAgeMs: number | null;
  completedSweepDurationMs: number | null;
  sweepCompletionAgeMs: number | null;
  observedReceiptLagMs: number | null;
  scanBefore: string;
  retainedFrom: string;
  sourceComplete: false;
  observationComplete: false;
};

rawExhausted means only that the last ID page found fewer than a page of currently visible raw IDs after the cursor, so the next cycle will start over. It says nothing about unseen WAL, partial initial snapshots, late source transactions, missing client beacons or uninstrumented traffic. A heartbeat measures replicated liveness only. Invalid/absent/future heartbeats are reported as null; known rows can still be archived. A failed import does not publish a new observation. An older observation is not a current success signal; inspect its version/age.

There is no permanent timestamp high-water exclusion or global archive membership set. Each event page reads at most 250 raw FINAL rows using WHERE id > afterId ORDER BY id LIMIT batchSize. The importer checks archive membership with WHERE id IN only those eligible page IDs, then inserts missing canonical rows. Receipt bounds and source deletion flags are applied after reading the page. Expired, recently received, deleted and already archived raw rows all advance the cursor, preventing a long filtered tail from repeatedly consuming the same scan budget. Source IDs must be nonempty immutable strings; both raw and archive tables must have ID-leading keys (verify their actual plans).

After archive acknowledgement, each page writes a separate source_cursors.source='discovery_events_sweep' checkpoint. Its JSON cursor is {afterId,startedAt,completedAt,completedDurationMs}; replacement versions increase monotonically. Losing an archive acknowledgement retries the page and rechecks its IDs; losing a checkpoint acknowledgement either resumes or repeats that acknowledged page. Both paths are idempotent. The existing worker lease serializes writers. At EOF the checkpoint resets afterId='' and stops that run. The next cycle revisits IDs inserted behind the old cursor, including rows with old occurrence/receipt timestamps. Only receipt age determines the 180-day archive eligibility boundary; expired rows cannot resurrect after TTL.

rowsScanned counts raw rows visited this run, including those already archived or excluded. sweepAgeMs measures the current cycle's age at the run clock; completedSweepDurationMs retains the previous cycle's full duration after reset; sweepCompletionAgeMs measures time since its EOF. Missing/unstarted sweeps report null. observedReceiptLagMs measures the newest eligible receipt whose archive presence was confirmed during this run, or null for none. That is an observation of these pages, not the newest source row or a lateness guarantee. Metrics use the supplied run clock; budget up to the 30-second run duration too. All sweep checkpoints also keep complete=0 and null coverage bounds.

Operation	Bound
Import	250 rows/page, at most 10 erasure batches then 10 raw event pages; 2,500 raw rows examined/run, not 2,500 guaranteed new rows; one bounded archive lookup and post-ack checkpoint/page; 30-second overall deadline. Smaller test/operator parameters allowed; larger ones rejected.
Import recency	Receipt time at least 30 seconds behind worker clock, within last 180 days. Heartbeat does not define completeness.
Archive queries	10-second server execution limit, 256 MiB memory, 10 million read rows, 64 MiB anti-join set; overflow throws. Existing Warehouse HTTP response cap is 16 MiB.
Export	Default 250 rows/page (maximum 250), default 100,000 rows/input (maximum 1,000,000), at most 180-day requested range; 1 MiB/JSONL line, 512 MiB/input file, five-minute deadline.
Export erasure audit	Exact ordered SHA-256 over at most 1,000,000 actor/cutoff rows, before/after export; pending raw erasures or changed generation abort.
Source dimension export	Default/maximum 1,000 rows/page; default 100,000/maximum 1,000,000 rows; 1 MiB/line, 128 MiB/file; 180-day window, five-minute abort budget. Source statement 10 seconds, query 15 seconds, lock 500 ms and idle transaction 30 seconds.
Prune	Exactly 1,000 candidates/batch maximum; default/maximum 10 batches, five-minute abort budget (an in-flight source statement may finish); source statement 12 seconds, connection/query limits 5/15 seconds and lock timeout 500 ms.

These are failure limits, not a scale certification. Archive lookup membership is bounded by a page; physical reads still depend on sorting keys, FINAL parts, and the deployed ClickHouse query plan. Validate read-in-order/range pruning on raw IDs and selective archive ID lookups in a disposable representative instance before readiness. The pending-erasure join still compares the retained actor tables and has a measured cardinality ceiling; its caps fail closed. Track row bytes, full-cycle latency, erasure backlog, parts and query costs before enabling full traffic. The controller's sampling/capture controls should bound snapshot volume. At a measured average of B bytes and E events/day, raw logical event storage is approximately B * E * 180 before compression, indexes, CDC copies and replication. No traffic/compression/cost estimate is fabricated here. Reaching a query cap fails closed and needs measured redesign/tuning.

Full-sweep latency is an enable blocker. At the current maximum 2,500 raw rows/minute, one million raw IDs require 400 minutes and 401 worker invocations including the confirming EOF page; already archived IDs consume this budget too. The targeted virtual-volume test exercises those million rows through the real importer and preserves that duration across reset. It is not a ClickHouse performance benchmark. Source CDC tombstones/expired raw history also lengthen the cycle until physically removed. New rows behind the cursor can wait a whole cycle; continuous growth, retries, erasure backlog or slower scheduling can make it longer. A 30-second recency filter is not a freshness guarantee.

Before enabling readiness, the controller must declare a maximum acceptable archive lateness, then demonstrate at expected retained volume that the maximum of current sweep age, previous completed duration and time since completion, plus scheduling/run time, the 30-second eligibility delay and separately measured CDC lag, fits that budget and the retention horizon. No numeric SLO has been assumed here. Missing metrics, breached budget or unmeasured query plans block training/rollout readiness; sampling/retention/capacity must be adjusted and remeasured. Never stop the catch-up importer merely because its sweep is stale. The current CLI remains a partial-observation export and does not itself certify this operational gate.

Explicit preparation and enablement

1. Apply the controller's operational migration for both tables with collection still disabled. Verify account deletion and stale-event rejection together.

2. Review/apply source-discovery.sql to the existing allowlisted publication. An existing broad/different column list or row filter fails closed for manual review. Configure replication-role column SELECT and ctid access separately. Include both tables in the existing ClickPipe; wait for their initial snapshot. The controller's migration must first add nullable session_id, received_at and boolean consumer_eligible to analytics_play_intervals, with no legacy backfill. PostgreSQL does not provide a per-member publication ADD COLUMN: inside the same transaction, this migration removes/re-adds only the existing interval member with the exact old six-column list extended by those three columns (or upgrades the known interim eight-column list). Other members remain untouched. A different/broad existing interval list fails for manual review. Repeated execution recognizes the exact nine-column list and does nothing. This is not SET TABLE and never recreates the publication. Manual ClickPipes mapping gate: explicitly refresh/extend that existing source table's column mapping, confirm the destination has nullable session_id, nullable timestamp received_at and nullable Bool consumer_eligible; confirm new rows contain the real session ID, DB-stamped receipt and source eligibility, and verify old rows remain NULL. The controller sets eligibility true only for a locked, owned, non-ephemeral session on a published world whose creator differs from the user. Export never reconstructs that eligibility from product, session existence or events. Publication DDL alone does not prove mapping/snapshot readiness. Do not enable export readiness until these checks pass. Missing destination columns cause the strict exporter to fail; it never falls back to legacy interval rows.

3. Apply warehouse migration 008 using a separately scoped migration credential:

   pnpm --filter @yumina/server exec tsx src/analytics/worker.ts --apply-discovery-schema

   Supply CLICKHOUSE_MIGRATION_USERNAME and CLICKHOUSE_MIGRATION_PASSWORD through the secret store. Explicit migrations require both and never fall back to the runtime account. Configuration still requires CLICKHOUSE_URL, the target database and REDIS_URL, but this command returns before connecting Redis. There is no startup DDL or automatic publication change.

   On affected ClickHouse versions, CREATE OR REPLACE VIEW can report an access error while checking an internal replacement object, despite correctly scoped view grants (upstream issue #90919). An error does not prove that the view stayed unchanged. On production 26.2.1.641, the September 20 column repair returned code 497 but the replacement was present and passed independent reads. Before any retry, reconcile the actual definition, UUID, all consumer column names/types, archive identities, erasure predicates and a real consumer query. Preserve failed command evidence separately from reconciliation. Do not automatically fall back to DROP/CREATE or widen database grants. A maintenance recreation is a separate non-atomic operation requiring an idle consumer window and explicit recovery handling.

4. packages/server/tsup.config.ts now copies migration 008 to dist/migrations/008_discovery_events.sql; the selector/copy-list regression test covers all worker migrations. Verify the final deployment artifact during the controller's build. Source PostgreSQL world-history SQL remains a separate explicit migration, not a ClickHouse worker migration.

5. In a disposable ClickHouse environment, validate the migration, raw ClickPipe types, ID-leading keys/selective query plans, full-sweep lateness budget, FINAL deduplication, TTL and erasure-filtered reads. This task tested SQL contracts and the read predicate locally; the subsequent managed-service readiness run passed all nine live checks including owned-database cleanup. That run used synthetic raw CDC fixtures and does not certify the production ClickPipe. Configure consumer permissions to disallow bypassing the filtered view.

6. Reconcile source IDs/content against raw snapshot and durable archive within explicit windows. Compare exact erasure actor/cutoff pairs too. Do not use counts or heartbeat alone. Record source/observation gaps; empty sources do not prove successful snapshot completion. Only then set the default-off ANALYTICS_CDC_READY=true, ANALYTICS_DISCOVERY_READY=true and existing ANALYTICS_ENABLED=true for the worker. Discovery collection has its own controller-owned flag; readiness does not enable collection.

7. Exercise read-only export and pruning dry-run first. Review skipped candidates, latency and volume before scheduling any explicitly authorized prune applies.

Disabling ANALYTICS_DISCOVERY_READY stops future archive refresh attempts and CLI readiness checks; it does not delete archived facts. If erasure import is paused or lagging, suspend training/export consumption too, since existing views cannot suppress tombstones they have not received.

Read-only JSONL export

The event export and pruning CLIs require explicit CLICKHOUSE_URL, CLICKHOUSE_DATABASE, CLICKHOUSE_USERNAME, CLICKHOUSE_PASSWORD, ANALYTICS_CDC_READY=true and ANALYTICS_DISCOVERY_READY=true in the environment. Use separately scoped credentials. They never load .env, connect through the app, modify ClickHouse, or log credentials/raw rows. No credentials belong in command-line arguments.

Example invocation (illustrative paths, not executed):

pnpm --filter @yumina/server exec tsx ../../scripts/export-discovery-data.ts --from 2026-09-01T00:00:00Z --through 2026-09-19T00:00:00Z --events-out /private/discovery/events.jsonl --intervals-out /private/discovery/intervals.jsonl --manifest-out /private/discovery/export.json --batch-size 250 --max-rows 100000

Parent directories must exist. All three paths must be distinct and new; existing files are never overwritten. Data is written a page at a time, with SHA-256/byte counts. A valid manifest is written last and is the completion marker; failures remove only files created by this attempt. A process crash may leave partial files without a valid manifest; choose fresh output paths for retry.

Events come exclusively from source_discovery_events, paginated by unique ID, with occurrence in [from, through) and receipt <= through. JSONL has exactly the canonical camelCase fields: id, eventType, occurredAt, receivedAt, actorId, nullable userId, visitId, feedRequestId, opportunityId, worldId, nullable languageGroupId, position, policyVersion, featureVersion, nullable numeric modelId, and parsed object payload. No feature reconstruction or PostHog union occurs. ISO UTC timestamps preserve microseconds from the warehouse. eventType='page' is retained verbatim, including worldId='', position=-1, the page opportunity ID, and payload.cardCount/measurementStatus (recorded, empty or unavailable). It is not reinterpreted as a card exposure or discarded.

Intervals come only from yumina_raw.analytics_play_intervals FINAL with live CDC state, product='main-app', valid duration (0,90s], end after from and at or before through, and no account erasure marker. Strict rows additionally require non-NULL/nonempty session_id, non-NULL received_at, receipt at or before through, and consumer_eligible=true. Output keys are id, userId, worldId, startedAt, endedAt, product, sessionId, receivedAt, consumerEligible (JSON boolean true). NULL/false eligibility is excluded, and malformed transport flags fail the export instead of becoming truthy strings. No synthetic session ID/receipt/eligibility, historical playtime counter, game interval or network duration is substituted. This source is not extended to 180-day durable interval storage by this slice; retained interval coverage can be shorter than archived event coverage.

The manifest's intervalCoverage reports legacyExcluded (missing any of the three strict fields), missingSession (NULL or empty), missingReceipt (NULL), missingEligibility (NULL), consumerIneligible (explicit false), and receivedAfterCutoff. These counters overlap, so do not sum them. They are bounded-query observations of retained, live, nonerased, valid-duration main-app intervals ending in the requested window, not estimates of missing historical rows or a transactionally frozen denominator. They do not promote source or observation coverage to complete. Legacy NULLs stay excluded, even if the source row is still present after mapping changes.

The exporter verifies no visible pending erasure and compares the entire bounded durable erasure generation before/after writing. It fails if that generation changes. It cannot certify a transactionally frozen event/interval snapshot or unseen CDC erasures: the manifest explicitly sets snapshotConsistent=false, sourceComplete=false, observationComplete=false, even when scanCompleted=true. Reruns can include late arrivals. Keep the original JSONL and hashes for exact reproducibility, subject to later erasure invalidation.

trainer/discovery_data.py consumes the two JSONL schemas. Its as_of is a caller-attested complete-through watermark for BOTH sources, not the export clock, through, heartbeat, or scanCompleted. Reconciliation and observation coverage are hard blockers before treating matured missing outcomes as negatives or promoting any model. An export succeeding does not satisfy them.

Concrete operational pruning

Explicitly set a source connection in a dedicated environment variable (no .env lookup), then use its name, never its value, in the command:

pnpm --filter @yumina/server exec tsx ../../scripts/prune-discovery-events.ts --source-url-env DISCOVERY_PRUNE_DATABASE_URL --cutoff 2026-08-19T00:00:00Z --max-batches 10

This is dry-run. Only adding --apply permits deletion. The cutoff must be at least 30 days before execution time. Each source SELECT takes up to 1,000 immutable candidate IDs in ID order, excludes identity_link, and limits receipts to [now-179d, cutoff). A read-only warehouse query checks those exact IDs against physical discovery_events FINAL within the same TTL-safe range. Only the candidate/archive intersection is passed to a parameterized source DELETE, which repeats the receipt/type guards. Neither a count nor a heartbeat authorizes deletion. Source/warehouse failures stop the command; already completed batches remain safe to retry. No distributed transaction or all-batches rollback is claimed. The operational immutable-ID contract is required.

The JSON result reports dryRun, scanned, reconciled, unreconciled, deleted, exhausted, afterId. Use --after-id with the last returned ID to continue a bounded scan past unreconciled candidates. A full-size last batch sets exhausted=false even if it happens to be the last page. After finishing a sweep, start a new sweep without --after-id to revisit rows that were unarchived on the earlier pass. Receipts older than 179 days and identity-link/erasure metadata remain untouched; they are not included in these per-sweep counters.

Source world/group history capture

scripts/discovery-world-history.sql is a separate, explicit PostgreSQL migration. It installs public.discovery_world_history(id,world_id,language_group_id, observed_at,is_deleted,capture_kind) and the discovery_world_history_capture trigger on world INSERT, actual language_group_id changes and DELETE. IDs are immutable; timestamps advance by at least one microsecond per world, even on clock regression or repeated changes inside a transaction. History writes roll back with the world mutation. Null groups remain null; they are not replaced with a guessed historical family. The table has no world FK cascade, so deletion preserves the earlier observations.

Installation does not scan existing worlds or use their creation timestamps to backdate a mapping. An authorized migration/maintenance owner can explicitly run SELECT public.backfill_discovery_world_history(1000) in bounded transactions. Allowed batch sizes are 1–1,000. It locks source rows with SKIP LOCKED, rechecks under the lock and inserts a current baseline only for worlds without history. Execute privilege is revoked from PUBLIC. A zero result can reflect skipped locks, so it is not a completeness certificate; repeat/reconcile after writers drain. Trigger/migration roles need the corresponding table/sequence privileges.

public.discovery_world_history_intervals derives world_id, nullable language_group_id, valid_from, nullable valid_to, capture_kind with LEAD over all observations before excluding deletion rows. Deletion therefore closes the previous mapping; it never leaves a falsely open historical family. Before a world's first captured observation the mapping is unknown. Capture timestamps are database observation times inside the source transaction, not a reconstructed WAL commit clock; future strict attribution should reject conflicting boundary evidence rather than invent mappings.

This table/view is not enrolled in CDC. The separate source dimension export below supplies its public mapping history directly. No source pruning is enabled for history until a durable successor/replay policy exists. This migration never touches the controller's discovery_known_stories or discovery_history_coverage; those internal first-known/coverage tables also remain outside CDC.

Read-only source dimension export

After explicitly preparing the history table/view and any bounded baseline, set a source connection in a dedicated environment variable. No .env is loaded, no ClickHouse connection is needed, and the CLI never defaults to DATABASE_URL. Use an account with SELECT on public.discovery_world_history_intervals only:

pnpm --filter @yumina/server exec tsx ../../scripts/export-discovery-dimensions.ts --source-url-env DISCOVERY_DIMENSION_DATABASE_URL --from 2026-09-01T00:00:00Z --through 2026-09-19T00:00:00Z --out /private/discovery/dimensions.jsonl --batch-size 1000 --max-rows 100000

The single dedicated source connection uses REPEATABLE READ READ ONLY. The database transaction timestamp must be at/after through, so a snapshot taken before the full requested window fails. Each page selects observed intervals overlapping [from, through), with a (world_id COLLATE "C", valid_from) keyset. It does not clip or backdate source boundaries. The exclusive new JSONL file has exactly worldId, nullable languageGroupId, validFrom, nullable validTo, preserving UTC microseconds. A NULL group means observed ungrouped history; it must not match another world's non-NULL family. Before a baseline there is no membership row. Deletion closes the preceding interval.

Pass this file to the trainer's --dimensions input with the matching event window. CLI success prints rows, bytes, SHA-256, window and snapshotAt; preserve that receipt with the dataset. Errors roll back and remove only this attempt's file, while existing outputs survive. A process crash can leave a partial file; without a successful receipt/hash it is not a completed input.

snapshotConsistent=true refers only to the source dimension transaction. historyComplete=false remains explicit: installation/baseline gaps and changes whose transaction commits after the snapshot cannot be inferred. Trigger times are observations inside transactions, not commit clocks. Reconcile ambiguous boundaries before using them as strict family evidence; do not treat dimension export success as a complete-through watermark for events or play intervals.

Verification and source semantics

Focused tests use injected warehouse transport and local in-memory PGlite. They cover lost insert/checkpoint responses, retries, late old IDs behind a reset, one-million-row virtual sweep capacity, tied timestamps, bounded pagination/resume, source retention survival, TTL non-resurrection, pending/updated erasures, permanent account exclusion, both guest clocks and future guest events, partial/empty coverage, repeatable publication extension and allowlist rejection, canonical JSONL, partial-file cleanup, changed erasure generation, CLI isolation and exact-ID pruning (dry-run/apply/failure/unreconciled cases).

node packages/server/scripts/test-local.mjs src/analytics/config.test.ts src/analytics/discovery-import.test.ts src/analytics/discovery-export.test.ts src/analytics/discovery-prune.test.ts src/analytics/discovery-world-history.test.ts src/analytics/discovery-dimensions-export.test.ts src/analytics/game-import.test.ts

The source-publication migration is executed twice against in-memory PostgreSQL, and the actual view predicate runs there with small ClickHouse syntax shims. ReplacingMergeTree/TTL syntax is contract-tested, not validated against a running ClickHouse instance. Controller global builds, OSS export and production enablement remain separate.

References: ClickHouse replacement semantics define identity by ORDER BY and require FINAL before background deduplication; ClickHouse TTL deletes physical data during background merges, so the read view also enforces the retention window. PostgreSQL ALTER PUBLICATION provides additive table membership; SET TABLE would replace existing members and is deliberately absent.