Long-term recommendation architecture and spending decision
The recommended path is a custom, measurable retrieval/ranking system on the existing managed services. No new vendor subscription is required to implement the current work. Existing usage charges can grow; that is different from requiring a new product. The September 19 read-only Railway inventory confirmed a testing environment with Redis and the app, and a production analytics service. No service was provisioned or resized.
Architecture decision
Use Postgres for authoritative publication, access, Library state and committed outcomes; pgvector for content retrieval; Redis for low-latency session state; ClickHouse for durable observations, features and temporal evaluation; PostHog for product diagnosis. Train offline and publish versioned, evaluated artifacts. Keep inference bounded and independent from third-party API calls on each page.
flowchart LR
P[Neon: catalog, eligibility, vectors] --> C[Candidate retrieval]
C --> R[Versioned ranking]
R --> D[Variety and exclusions]
D --> F[Discover]
S[Recommendation Redis] <--> D
F --> E[Canonical observations and confirmed outcomes]
E --> W[ClickHouse archive]
E --> H[PostHog diagnostics]
W --> T[Temporal datasets and evaluation]
T --> G[Experiment and promotion gate]
G -. validated future model .-> RThe promotion arrow is a release gate, not an automatic training loop enabled by this change. PostHog mirrors are never added again as independent training events.
Retrieval should combine semantic similarity, collaborative behavior, session intent, quality/freshness and controlled exploration. Rank multiple positive outcomes (qualified new discovery, save, later return) with explicit negative preferences. Apply audience, publication, blocks and Library/group exclusions as hard constraints; apply creator/topic variety as a separate policy. A single average user vector is not sufficient for someone with several distinct interests. Introduce multiple interest representations as an ablation once the corrected observation stream is usable.
Meta describes retrieval followed by progressively more expensive ranking, and emphasizes a model registry, repeatable performance tests, controlled launch and prediction-quality monitoring. Those engineering principles are appropriate here; its fleet size is not a template for Yumina's 1,725-row catalog. Meta engineering.
Spending priorities
| Resource | Decision now | Evidence that justifies expansion |
|---|---|---|
| Redis for recommendation sessions | Support a dedicated instance on the existing Railway account. Prefer isolation from authentication, rate limits and other critical Redis consumers when enabling material cursor traffic. This is additional capacity, not a new vendor requirement. | Measure full session bytes, active actors, admission rejection, p95/p99 latency and memory under concurrency. Admission caps are load controls, not capacity estimates. |
| Neon / pgvector | Keep as the initial semantic retrieval store. Check filtered-query recall against exhaustive search and inspect real query plans. | A dedicated vector service is justified when measured recall/latency or isolation targets fail after indexing/query tuning, not at an arbitrary catalog size. |
| ClickHouse / ClickPipes | Extend the existing allowlisted CDC path and archive canonical events. Increase compute/storage if ingestion or training freshness falls behind. | Source-to-archive reconciliation, backlog, query cost, deduplication and measured retained bytes. Archive feature snapshots separately from private content. |
| Training compute | Keep CPU-based baselines and a separate scheduled training job. Store manifests and artifacts in existing object storage. | Rent a GPU job only when a multi-interest/two-tower experiment has sufficient data and a defined comparison, or CPU training misses its refresh target. No always-on GPU fleet is needed for current serving. |
| Feature store (e.g. Feast) | Implement versioned serving snapshots and shared feature definitions now; keep the boundary replaceable. | Add a feature-store service when multiple models/teams need managed feature materialization, online freshness and repeated point-in-time retrieval. It does not repair missing or incorrectly attributed events. |
| Kafka / Redpanda / stream processing | Existing PostgreSQL CDC is the first transport. | Add a durable event bus when independent high-throughput producers/consumers, replay, fan-out or latency requirements exceed the measured CDC design. Do not run two unrelated event truths. |
The pgvector project documents filtered-search indexing and iterative scans; Qdrant offers payload indexes and filter-aware vector retrieval. Both still require a representative benchmark and correct eligibility checks. A vector database retrieves candidates; it cannot infer the product objective, distinguish a failed Play click from a successful session, or fix biased labels. pgvector, Qdrant indexing.
Feast's online/offline architecture and point-in-time joins address feature reuse and temporal correctness. The current snapshot contract deliberately preserves those concepts so a later adoption is possible without reinventing event identity. Feast architecture, point-in-time joins. ClickPipes already supports managed PostgreSQL ingestion; a new message broker is not a prerequisite for this stage. ClickHouse integrations.
Cost and scale gates
The audit observed roughly 749,000 Recommended impressions over seven days; served opportunities are a different, potentially larger volume. Estimate cost from measured bytes per serve/event, daily actual serves, retention and index/replication overhead. Do not multiply impressions by a guessed row size and call that a capacity plan. The implementation reports snapshot bytes and coverage status on serving telemetry, preserves immutable IDs across retry, and bounds client buffers and Redis admissions.
The store benchmark includes pending feature snapshots and retry pages, not only short IDs. It still excludes real primary-DB ranking, multi-instance HTTP traffic and browser layout. Validate these on an isolated staging deployment before selecting the production cap or increasing traffic. Use at least expected peak and a defined overload case; record error rate, p95/p99 latency, memory growth, queue saturation and successful recovery.
Data quality precedes model promotion: reconcile source/warehouse IDs, preserve occurrence time and actor identity, validate confirmed outcomes, union overlapping play intervals, count distinct return days, require label maturation, and split chronologically with gaps. New features/models need ablations and stable actor-randomized experiments. D7 uplift cannot be known until enough real follow-up has elapsed. New subscriptions cannot remove that observation requirement.
Release boundary
This decision authorizes no purchase and makes no production capacity or user-uplift claim. The code supports incremental rollout and measurable upgrade decisions. The most likely near-term spend is isolated Redis/staging capacity and additional warehouse storage within existing accounts; exact sizing follows load/volume evidence. Managed ML platforms or a replacement vector database remain options, not dependencies.