Production discovery ClickPipe validation — 2026-09-20
The missing-table blocker is resolved. The existing yumina-admin-analytics ClickPipe is Running with 19 mappings. Its original 17 mappings and unrelated configuration were preserved. A bounded synthetic check through the production Postgres → ClickPipe → raw ClickHouse → archive path passed, including explicit erasure. Recommendation, measurement and analytics-readiness flags remain off.
Configuration change
| Source | Destination | Sorting key | Columns |
|---|---|---|---|
public.discovery_events | yumina_raw.discovery_events | id | All 17 |
public.discovery_erased_actors | yumina_raw.discovery_erased_actors | actor_id | Both |
Both raw tables use SharedReplacingMergeTree, have no partition expression, and include _peerdb_version and _peerdb_is_deleted. Read-only preflight confirmed publication membership, all column grants and snapshot ctid access; the existing source publication and grants required no changes.
The separately supplied Cloud API credential could manage this exact pipe, although the installed Cloud MCP connector only exposes reads. The credential was passed through hidden process input, kept in memory and cleared afterward. Connector credentials and permission settings were not changed. All temporary credential processes were terminated. No subscription or additional service was needed.
The controller journaled each mutation before sending it and checked hashes of the original mappings and unrelated configuration. The additive PATCH contained only source.postgres.tableMappingsToAdd. The pipe reached Paused before the update, and the API returned it to Running after adding the tables; no additional restart was needed. An independent MCP list confirmed Running / 19 mappings.
One preliminary pause request used action from ClickHouse's blog example and received HTTP 400 without changing the pipe. The current official API model uses command. The corrected command: stop request and the additive table update both returned HTTP 200. No uncertain mutation was blindly repeated.
Live pipeline check
Run 4d79ce098b5d66aff083c007 passed all ten checkpoints. It used one explicitly synthetic pipeline_canary event, a synthetic actor and world, and the unchanged candidate importer. It did not create a real user, story, recommendation or model call. The source and discovery archive were empty at preflight.
| Assertion | Observed result |
|---|---|
| Source insert reaches raw ClickHouse | One matching live event |
| Candidate importer archives the event | One archive row and one consumer-visible row |
| Repeated import | Zero new events; still one archive row |
| Source retention-style deletion reaches raw | Matching CDC tombstone |
| Import after source deletion | Archive and consumer view retain the event |
| Explicit erasure reaches raw | Matching live actor marker |
| Import after explicit erasure | Marker archived; consumer-visible event count becomes zero |
| Final read-only audit | Source event absent, raw tombstone present, archive row retained, erasure marker present, consumer view excludes event |
The test event has been removed from Postgres. Its synthetic raw tombstone, archived event and permanent synthetic erasure marker intentionally remain as the bounded audit fixture; recommendation consumers cannot see the event through source_discovery_events. Nothing belonging to a real user was modified.
The final audit independently confirmed all four application flags off, ANALYTICS_DISCOVERY_READY off on the worker, and the dedicated discovery Redis still unwired. Every importer observation retained sourceComplete=false and observationComplete=false; a drained queue does not prove complete collection.
Evidence and limits
Application source remains the previously tested candidate. Importer SHA-256: c177ca016f46cc953866af38f7751440d38589b5af35d31d909892c4c931fa19. The existing application CI and deployment tests were not rerun for this infrastructure change; the new checks exercised the actual CDC route.
Ignored local receipts:
| Receipt | SHA-256 |
|---|---|
clickpipe-discovery-preflight-v1.json | 69b698598f5937bc1d12bdd4a4dd63b2d881e794858c532228b76d792909ddb4 |
clickpipe-add-discovery-journal-v1.json | 265abe9c0f8c52007411dd7c0e57e91847f93e85a7212520c52ba69b813bdfa3 |
clickpipe-discovery-verify-v1.json | bcdf09b6d30371d50ddbed35597be080ecd6140d8dc4b1c039b1ef031fe32869 |
clickpipe-independent-confirmation-v1.json | 66645cbb3c2b200144810ef57abb835a54e515cef56797c9c6e4f2fee7353371 |
clickpipe-live-discovery-canary-v1.json | d845a932e1c12909399ae554666b64d1c252a2ca3e2853311dbd371e76d4c513 |
clickpipe-final-audit-v1.json | b7fde156600ff7087b1240197819885308560faf437d77cb329aa248ea500b7c |
This proves one real replication, archive, idempotency and erasure sequence. The importer ran explicitly from the reviewed candidate using existing worker credentials; this is not evidence that the production worker deployment has adopted that candidate or runs the importer on schedule. The final audit measured the operator machine about 22.5 seconds ahead of Postgres, so client timestamps and importer lag fields must not be used as calibrated end-to-end latency measurements. This run does not establish sustained throughput, complete capture, long-outage recovery, production concurrent capacity, a full visit-TTL cycle or reader-outcome improvement. These remaining gates still prevent broad feature activation. Experimental semantic ranking remains disabled after its failed quality evaluation.