Published-story extraction pilot
Two bounded runs used the final story-content-v2 / story-descriptor-v3 code on September 19, 2026. Both read the same eight eligible, published public worlds from the production read replica. Their canonical input hashes matched for all eight worlds. No database writes, private messages, embeddings backfill or production flag changes occurred. Existing OpenRouter credentials were supplied in memory. Each run made eight extraction requests with the production adapter's 30-second deadline, 4,096-token completion cap and response bounds.
| Diagnostic | Gemini 3 Flash Preview | Claude Sonnet 4.6 |
|---|---|---|
| Valid, nonempty curated descriptors | 8 / 8 | 8 / 8 |
| Retained / proposed facets | 64 / 92 | 29 / 85 |
| Retained / proposed evidence quotes | 81 / 145 | 36 / 128 |
| Provider-reported cost for this final run | $0.049998 | $0.284271 |
| Mean elapsed extraction time | 6.38 s | 21.51 s |
| Maximum elapsed extraction time | 11.16 s | 25.46 s |
Exact model IDs were google/gemini-3-flash-preview and anthropic/claude-sonnet-4.6; response identities were checked. Cost totals cover these successful final runs only, excluding earlier diagnostic requests and the separate embedding probe. They are measured small-sample costs, not a catalog budget estimate. Current Sonnet list pricing was verified against the provider page; the table uses actual response usage rather than multiplying that list price by estimates.
The sampled worlds were 恋与深空模拟器, 萬靈天下, RPG Engine & World Sim., Poison in the Bottle · Battle Royale, Sakura Season, Marvel RPG, SEVENTEEN Simulator and Jujutsu Kaisen. Six rows have English language metadata and two Chinese; some English rows contain Chinese tags or mixed-language source sections. This is a purposive sample of audited popular/representative worlds, not a random sample or a Spanish/Japanese coverage benchmark.
What the pilot established
The live provider rejected the initial nested schema with HTTP 400. A controlled short-input probe reproduced that rejection; removing only wire maxItems constraints made the same request acceptable. Local facet/quote limits, strict shape validation, token/response bounds and provider parameter enforcement remain in place. This is consistent with the provider's documented JSON-schema subset and complexity limits.
The next live run exposed all-or-nothing evidence handling: a weak additional quote could reject an otherwise useful descriptor. Curation now removes unsupported individual quotes and then empty facets, reports proposed/retained counts, and validates the result with the unchanged strict storage validator. Malformed envelopes, wrong versions/hashes, duplicates and exceeded budgets still fail rather than being repaired. A subsequent pilot exposed AI-narrator roles mistaken for player roles; the final prompt and multilingual role guard reject those clear production roles while preserving tested fictional reader roles.
A separate real OpenAI batch embedded both content and experience text for the first sampled world. It returned two finite, nonzero 1,536-dimensional vectors with the exact expected text-embedding-3-small response identity. This verifies the adapter contract, not retrieval quality or the optimal embedding model.
Quality limits and promotion decision
Neither model is promoted by this pilot. Retention counts are diagnostics, not accuracy: a model can propose more correct facets, more irrelevant facets, or evidence that a conservative validator fails to recognize. The more expensive model did not remove the coverage problem simply by replacing the cheaper one.
Manual inspection still found incomplete interests for Jujutsu Kaisen, missing experience coverage for the battle-royale story, and loose open-text labels such as a role classified as a tone. Literal quotation establishes provenance, not correct interpretation. Model confidence values are not calibrated probabilities. The current experience embedding uses accepted experience facets; free-form tone, setting and player-role facets are stored but do not enter that vector or the eight numeric reader/story features. The content vector independently uses bounded published prose, so missing explicit facets do not erase all matching information for a story.
Before broad descriptor-driven ranking or interest onboarding, freeze an independently labeled, language/category-stratified benchmark. Measure supported facet precision, omitted important facets, retained quote quality, empty experience coverage, stability, latency and cost on the same inputs. Review source truncation and conservative vocabulary as well as model choice. Keep benchmark worlds separate from examples used to tune the prompt or validator. Only then select a descriptor configuration, backfill a testing catalog and measure retrieval against content-only and existing collaborative baselines.
This pilot validates an operational extraction path and identifies remaining quality work. It provides no recommendation uplift, user-personality inference, calibrated-confidence result or basis for a full production rollout. See the rollout gates.