Published-content embedding implementation — 2026-09-19

Task 1 only, implemented directly in the codex/discovery-foundations worktree at C:/Users/jefra/.codex/worktrees/yumina-discovery. Read the worktree's AGENTS.md and the discovery-foundations plan before changes. This task made no production reads/writes, paid API calls, browser visits, deploys, pushes, or commits, and used no subagents or self-spawned reviewer.

Result and root cause

Published schemas represent openings as enabled entries with role: "greeting". The old embedding hooks and backfill looked only for schema.firstMessage, so current published stories lost their openings. The backfill also duplicated the text builder. Initial approval embedded from the earlier moderation-queue SELECT, not the rows actually published by the transaction.

embedding-content.ts now provides the single DB/env-free builder used by both the application service and backfill. The application re-exports the builder for compatibility. All four callers pass a published schema rather than extracting firstMessage themselves.

Input contract

• buildWorldEmbeddingText({ name, description, tags, announcement, schema }) produces the payload. schema is required and must be the published material. No private messages, session variables, player-specific interpolation, system entries, or working-copy material are read.
• Enabled greeting entries are ordered by numeric position ascending, absent positions last, with stable ties, matching PromptBuilder.buildGreetingEntries / entrySort. The regression directly compares the engine's output order. Entries and the input document are not mutated.
• Non-greeting entries, disabled entries, non-string greeting content, and empty greetings do not contribute. Empty/malformed documents are safe. An existing entries field is authoritative even if empty, malformed, or containing only disabled greetings.
• Only when entries is absent does the builder accept legacy schema.firstMessage, as a string or the first string in its historical array representation. Legacy content cannot override real entries or resurrect a deliberately empty opening.
• The combined opening retains the established 2,000 UTF-16-character maximum and gains a 3,000-byte UTF-8 ceiling. Metadata byte budgets are title 256, description 3,000, tags 512, announcement 1,000; headings and separators fit within an overall 8,000-byte cap. These separate budgets reserve room for openings despite oversized metadata. Truncation preserves Unicode code points. This is a byte/character bound, not a measured tokenizer count.
• The input version is published-greetings-v1. buildWorldEmbeddingInput returns { text, version, hash }, where hash = SHA256_UTF8(version + "\n" + text). Hashes identify the exact bounded input, not vector bytes or a world-version-history ID. World IDs and irrelevant schema fields do not change the hash. Bump the version when extraction, ordering, labels, or bounds change.

Published-source audit

Caller	Source of embedding material	Commit ordering
lib/approve-review.ts	actualUpdates from the publication transaction's UPDATE ... RETURNING, rather than the prior variants SELECT	Existing after-commit mechanism retained; only rows actually published are embedded
lib/pending-edit.ts	Rows returned by the UPDATE that commits approved held edits to worlds	Returned rows collected for embedding; effects run after the transaction/context commits
routes/admin-worlds.ts	updated.schema from the committed admin force-publish mutation	Existing post-transaction force-publish hook retained
routes/world-versions.ts	result.world.schema returned by makeVersionLive	Only after a changed, successful live switch; draft restoration has no embedding hook
scripts/embed-worlds.ts	Current worlds.schema with both is_published = true and status = 'published'	Does not join pending edits or saved versions

worlds.schema is the current public material for a published world; pending edits and saved/restored drafts are stored separately. A previously published saved version must actually be made live before it can become the embedding source.

Storage and failure handling

Hooks and backfill condition their writes on the world remaining published and its schema, name, description, tags, and announcement still matching the embedded snapshot. JSONB equality avoids dependence on object key order. A delayed response cannot overwrite a newer content snapshot or write after unpublication. The backfill's missing-only mode also rechecks embedding IS NULL at write time.

Vectors must have 1,536 finite values. Hook failures remain non-fatal. Logs carry world IDs, version/hash, and stored/superseded/failure outcomes; provider and SQL error messages are not logged because they can include submitted story bodies. No story text or title is emitted by dry-run. Existing vectors remain available if a refresh fails; this task does not add a retry queue or clear stale vectors.

Backfill contract and future operator commands

These commands are documentation only; no real database backfill was run during implementation. The operator must explicitly select the intended DATABASE_URL. The script does not load .env or the application's environment validator.

# Safe default: count/hash published rows missing vectors, no API key needed.
pnpm --filter @yumina/server exec tsx scripts/embed-worlds.ts

# Inspect all published inputs, including existing vectors that need migration.
pnpm --filter @yumina/server exec tsx scripts/embed-worlds.ts --dry-run --all --input-version published-greetings-v1

# Future authorized apply: requires both API credentials and explicit version.
pnpm --filter @yumina/server exec tsx scripts/embed-worlds.ts --apply --all --input-version published-greetings-v1

Dry-run is the default and also an explicit option. It never invokes the provider or issues UPDATE statements; the CLI additionally sets the database session's default_transaction_read_only = on. --apply without --input-version, unknown versions/options, and --apply --dry-run are rejected before database connection. Only the current supported input version can be targeted; historical saved world versions cannot be embedded into the current world's vector.

The script keyset-pages by ID with a 64-row maximum and sends at most 64 bounded payloads per request. Each JSONL row contains worldId, inputVersion, inputHash, inputBytes, and outcome. The final JSON line counts candidates, missing/existing vectors, stored vectors, failures, and superseded inputs. Save these lines to compare hashes across runs without saving story bodies. Apply exits nonzero if any input failed or was superseded. Provider failures do not trigger an implicit paid retry; a missing-only rerun resumes unfinished rows, while --all explicitly re-embeds successful rows as well.

Regression evidence

Tests were written before the corresponding behavior changes:

1. Against the old builder, the published-greeting regression returned only Title: Published story. The six-test content suite had five expected assertion failures and one pass, including missing openings and the absent versioned-hash API.
2. Before adding conditional writes, storage tests demonstrated stale-content and unpublication writes succeeding incorrectly, and provider error text leaking into warnings: three failures and one pass.
3. Before implementing the backfill runner/options contract, all six backfill tests failed their feature assertions.

Focused command after implementation:

pnpm --filter @yumina/server test src/lib/embedding-content.test.ts src/lib/embeddings.test.ts src/lib/embedding-backfill.test.ts

Coverage: six content tests, four hook-storage tests, six backfill tests. Local PGlite with pgvector executes the real conditional SQL. The provider is always fake. The isolated launcher strips credentials, prevents .env loading, and blocks unmocked external fetches. The explicit temporary API-key string in the hook test is synthetic and the original typed value is restored after each test.

Final focused rerun after the cleanup/type correction: 16 passed, 0 failed, 0 skipped, approximately 12.1 seconds. Scoped git diff --check also passed (only Git's existing LF/CRLF conversion notices).

Related publishing/version regression command:

pnpm --filter @yumina/server test src/lib/embedding-content.test.ts src/lib/embeddings.test.ts src/lib/embedding-backfill.test.ts src/lib/approve-review-publish-date.test.ts src/lib/pending-edit-variables.test.ts src/lib/world-version-history.test.ts src/lib/review-transaction.test.ts

Result: 41 passed, 0 failed, 0 skipped, seven suites plus the standalone review transaction test; observed total duration about 28.1 seconds. The 131-world backfill fixture stored all 131 vectors in exactly three requests, each at most 64 inputs; its observed test duration was about 116 ms in that run. These are local synthetic correctness checks, not production latency/capacity measurements. The draft fixture contains a distinct unpublished opening that never enters the canonical request. Dry-run checks observed zero API calls and zero UPDATE statements.

pnpm --filter @yumina/server typecheck was run and found the new test cleanup's undefined assignment to the string-valued OPENAI_API_KEY field. Fixed by restoring the captured original string. Per the coordinating agent's instruction not to duplicate full checks, final full-project/typecheck verification belongs to integration; this report does not claim a fresh full typecheck pass. The test importing the backfill script also brings that otherwise script-only file into TypeScript's compilation graph.

Changed paths owned by this task

• packages/server/src/lib/embedding-content.ts — new shared content, bounds, input version/hash.
• packages/server/src/lib/embedding-content.test.ts — engine-order and legacy/bounds/hash regressions.
• packages/server/src/lib/embeddings.ts — canonical input, snapshot-guarded persistence, safe logs.
• packages/server/src/lib/embeddings.test.ts — local pgvector storage/race/error regressions.
• packages/server/scripts/embed-worlds.ts — shared builder, versioned dry-run/apply, bounded paging and guarded writes.
• packages/server/src/lib/embedding-backfill.test.ts — CLI and local PostgreSQL backfill regressions.
• packages/server/src/lib/approve-review.ts — committed publication rows as source.
• packages/server/src/lib/pending-edit.ts — committed edit rows as source.
• packages/server/src/routes/admin-worlds.ts — published schema passed to embedding hook.
• packages/server/src/routes/world-versions.ts — newly live schema passed to embedding hook.
• docs/recsys/2026-09-19-embedding-implementation.md — this evidence/handoff.

No changes were made by this task to recommendations.ts, routes/worlds.ts, pagination files, schema/migrations, package manifests, or the shared implementation plan. Other worktree changes belong to the coordinating agent.

Concerns and integration follow-up

• No schema change means there is no durable per-vector input version/hash. Existing vectors cannot be reliably classified by input version from database state alone. Use explicit --all --input-version published-greetings-v1 for the initial migration after reviewing dry-run output. Hash logs make current-input changes reproducible, but do not automatically skip identical existing inputs or prove how an old vector was built.
• Metadata-only edits outside the existing four publish hooks are not newly wired here. In particular, owner-world routes and recommendations are outside this task's scope. A metadata change or nonsemantic schema change during an API call conservatively skips the write; a future hook/backfill is needed to refresh it.
• Ordering is mirrored locally to keep the builder DB/env-free and tolerant of malformed historical documents. The parity test protects the current engine contract; future engine sorting changes must update the builder and bump its version.
• The 2,000-character/3,000-byte opening budget can truncate later alternate greetings. Multibyte openings may hit the byte ceiling sooner. This deliberately keeps the embedding input bounded rather than including every full alternate opening.
• Unit/SQL regressions do not validate a live OpenAI response, production query plans, ranking uplift, or every HTTP publish endpoint end to end. Caller data flow was inspected, with existing approval/draft/version transaction regressions run locally. The planned independent reviewer and integration checks remain with the main agent.