Story evidence and feed storage development results
This follow-up addresses two measured weaknesses from the readiness investigation. It does not promote a descriptor model, enable compact storage, deploy product changes or establish reader uplift.
Meaning-based evidence review
The serving descriptor checks require a recognized cue in each literal quote. A separate experimental path proposes only the six interest and eight experience facets, checks exact source provenance, and asks an independent model to judge each quote against its full source context and a fixed rubric. Verdicts are supported, contradicted, insufficient evidence or production instruction. Every quote requires exactly one decision bound to the canonical input and proposal hashes. Model confidence is not used as a calibrated probability.
Experimental proposals/reviews have separate versions and shapes. They cannot pass the production descriptor validator and are not imported by the serving worker. Production extraction reuses the same bounded transport, preserving its request format, model identity, deadline, response limits, cancellation and sanitized errors. Caller options are now snapshotted so mutation during a request cannot change its accepted model identity.
A live diagnostic exposed occasionally rewritten/concatenated quotes. Curation removes nonliteral quotes and then empty facets; it cannot repair malformed envelopes, duplicates, wrong hashes or exceeded budgets. Retained evidence is revalidated. Provenance does not prove correct interpretation.
Development protocol
- Twelve public, published, family-distinct worlds: three each with English, Chinese, Japanese and Spanish metadata. Fixed hash sampling excluded every family in the earlier pilot and frozen evaluation set. This is a small development sample, not population-weighted or human-labeled ground truth.
- Corpus SHA-256:
d7858c1de657cb3cfa4f789a614ca85b3bc80f2f281c8a69c406ac0f01c56746; canonical source text totaled 142,291 bytes. Sources remain in ignored local artifacts; no private chats or secrets are committed. - A separate blind source-only AI annotation produced 77 facets. All hashes and literal quotations were verified. The annotator did not inspect predictions or the cue vocabulary.
- Exact proposer:
google/gemini-3-flash-preview; exact reviewer:anthropic/claude-sonnet-4.6. The baseline used the proposer with the existing production prompt/validator. All response identities were checked; no silent retries or fallback models. - Initial run: 32 requests, $0.313054 reported usage; four proposals failed on nonliteral evidence. One diagnostic request cost $0.0060865. After provenance curation, all 12 proposals/reviews completed in 24 requests costing $0.339308. The unchanged baseline was reused. These are observed costs, not forecasts.
| Method | Predicted facets | Agree with reference | Reference coverage | Agreement among predictions | Worlds with experience facets |
|---|---|---|---|---|---|
| Serving extractor and cue validator | 26 | 19 | 24.7% | 73.1% | 6/12 |
| Experimental proposal before review | 91 | 62 | 80.5% | 68.1% | 12/12 |
| Same proposals, cue validator | 25 | 16 | 20.8% | 64.0% | 7/12 |
| Same proposals, semantic reviewer | 70 | 54 | 70.1% | 77.1% | 12/12 |
The final two rows compare validation on identical proposals. The baseline comparison also changes the proposal prompt/schema and includes stochastic generation. Neither is a causal reader-outcome comparison. Review rejected 21 facets: 13 absent from the reference, but also eight reference-supported facets. Sixteen accepted facets disagreed with the reference, and 23 reference facets were missing. Disagreement is not automatically model error; references and taxonomy also need adjudication.
Semantic reference coverage by language was 72.2% English, 82.4% Chinese, 68.2% Japanese and 60.0% Spanish. Prediction agreement was 81.3%, 82.4%, 71.4% and 75.0%. Three worlds per language cannot certify language parity.
The final proposal step retained 91/99 facets and 148/163 literal quotes. Proposal p50/p95 was 6.796/8.930 seconds; review p50/p95 was 7.679/8.655 seconds. This belongs in background processing, never the feed request path.
Sixteen authored multilingual contrasts checked consequential actions without cue words, words cut out of denials, classifier instructions and incidental franchise mentions. The live reviewer matched all 16 expected support decisions, costing $0.091002. The cases are committed in story-evidence-contrast-fixture.ts; passing simple development contrasts is not a general semantic accuracy guarantee.
Total reported provider cost was $0.7494505 across 73 requests. No embeddings backfill or promotion occurred. The frozen 24-world evaluation set was not used to tune or evaluate this new path.
Reproduction and decision
From packages/server, with the saved local artifacts:
pnpm exec tsx scripts/evaluate-story-evidence.ts ../../.local-artifacts/semantic-development-corpus.json ../../.local-artifacts/semantic-development-reference.json ../../.local-artifacts/semantic-development-curated-results.jsonThe scorer validates canonical sources, quote provenance, serving baseline descriptors, proposal hashes and complete review decisions, then recomputes lexical curation. It makes no network calls and does not trust saved aggregate counts. Public source catalogs are deliberately not repository fixtures.
Decision: continue development, not promotion. Adjudicate disputed facets/quotes and improve ambiguous taxonomy and quote selection on development data. Then freeze the configuration, use a separate evaluation set once, compare retrieval with existing/content-only baselines, and run the reader-outcome experiment. Coverage gains alone are insufficient.
Lossless feed storage
Rich v2 snapshots repeatedly serialize the same feature names. Optional compact-v1 stores each ordered name list once and numeric values in arrays. It reconstructs the original session, including zero-valued keys, ordering, delivery features, replay pages and unknown surrounding fields. No rounding, inventory truncation, extra Redis keys or compression dependency is introduced. Lua-visible ownership, revision and expiry fields remain intact.
DISCOVERY_SESSION_ENCODING defaults to raw. Upgrade every reader before any compact writer. To roll back to an older binary, switch all writers to raw, then wait a full one-hour session TTL before returning old readers. New binaries read both formats. Compact is not enabled in production.
A full 1,000-card offline traversal reduced maximum body size from 1,245,766 to 380,678 bytes. Total serialized reads fell from 34,925,321 to 10,669,903 bytes; writes from 35,323,237 to 10,839,060 bytes. These are deterministic fixture byte counts, not production throughput. Compact encoding/decoding costs more CPU.
Paired Railway probes
A temporary Node 22 service in Railway testing ran both modes with 1,000 rich v2 cards, 200 attempted page requests, two Redis connections, a ten-second traffic deadline and unchanged 32 MiB planned/sampled memory ceilings. Three pairs per actor count included reversed mode order. Each run validated replay, ownership and full decoded fixtures; this instrumentation adds CPU work, particularly decode/validation for compact writes. It is a session/Redis probe, not full-origin HTTP/SQL/ranking or production capacity.
| Concurrent actors | Raw p95, median of three runs | Compact p95, median of three runs | Raw sampled peak memory, median | Compact sampled peak memory, median |
|---|---|---|---|---|
| 1 | 96.105 ms | 114.977 ms | 459,496 B | 197,352 B |
| 4 | 329.331 ms | 466.866 ms | 4,196,973 B | 1,313,389 B |
| 8 | 810.638 ms | 819.170 ms | 9,180,275 B | 2,626,675 B |
These are medians of per-run p95s, not pooled percentiles. Small sample size, variable CAS contention, runtime CPU and guard instrumentation limit inference. At eight actors, median serialized reads fell from 244,023,593 to 70,252,366 bytes and writes from 217,965,921 to 62,556,878 bytes. Feature values were reconstructed and checked; nothing was discarded to achieve smaller payloads.
All 18 runs completed: 3,600 attempts, 3,430 successful pages and 170 explicit admission rejections, with no unexpected errors. Every run cleaned its exact owned keys. The temporary service b3a54db2-2c2e-40a4-85f5-1f26748b8947 was deleted from testing and absence verified. Shared testing data and production services/flags were not modified.
Decision: keep raw writes as the default. Compact materially reduces memory and wire bytes but did not establish a latency improvement, so it is not enabled by this work. Before choosing a production representation, profile full-origin processing and consider separating immutable inventory from small mutable page state. That design must preserve atomic revisions, replay, bounded retention and erasure without turning new storage limits into premature feed exhaustion.
Independent review also identified a correctness-probe compatibility issue: that test tool parsed compact bodies without decoding them. The follow-up uses the codec before fixture validation and verifies both modes. Actual Redis tests and tombstone/replay coverage are part of the final verification below.
Verification and scope
Hosted build and workspace typecheck passed. The focused content/model/evidence suite passed 216 tests. A broader affected discovery/personalization run passed 427 with two local Redis-dependent skips before the final additional replay check. After review fixes, the affected storage/measurement suite passed 44 with two Redis-dependent skips, and capacity scripts passed 19 with one Redis-dependent skip. These overlapping counts are not additive.
CI installs its own disposable Redis and explicitly runs raw and compact correctness/load cases. The compact measurement regression uses the real session decoder/replay and PostgreSQL-compatible tombstone transaction; after erasure it issues no receipt and recreates no events. Independent review found no remaining production-code defects. The offline scorer reproduced the development counts after validating provenance.
Final exact-commit CI status is maintained on PR #179, avoiding a documentation-only commit changing the revision being verified. No browser check, production deployment, private-chat analysis or reader-outcome experiment was performed in this follow-up. The existing stack remains sufficient for the next experiments.