Architecture Roadmap (Current Workspace, Next Boundary Steps)

Purpose

This roadmap exists to guide architectural refactoring in a practical, low-risk way so PopChoice stays maintainable as it grows. The repository already uses a workspace-based layout, so the focus now is not "move to a monorepo someday" but "make the current workspace boundaries explicit, enforceable, and easier to extract over time."

Current State

The repository is maintainable today and productive for ongoing work.

What is already true:

• the repo already uses npm workspaces with apps/*, packages/*, and services/*
• the web app and supporting apps already live under apps/
• background jobs and sync processes already live under root services/*
• a shared package already exists in packages/shared
• recommendation orchestration now lives in reusable feature-owned modules outside route ownership
• account, password reset, recommendation feedback, and movie-memory flows exist for signed-in users

The main remaining risks are:

• boundary clarity across app/domain/infrastructure concerns
• extractability of logic into independently owned modules over time
• some API boundaries still retain more orchestration awareness than the target boundary model

Status Snapshot

Done Already

• Workspace layout exists: apps/, packages/, and services/ are already in place.
• Shared package extraction has started with packages/shared reused by root services.
• Background service responsibilities are partially separated into services/movie-discovery, services/movie-seed, and services/movie-backfill.
• Recommendation orchestration is extracted into reusable feature-owned modules instead of living only inside route handlers.
• Some recommendation thresholds/constants are separated into dedicated helper modules instead of being embedded directly in route handlers.
• Ownership boundaries for src/app, src/integrations, src/lib, src/utils, and src/clients are documented in BOUNDARIES.md.
• README and development docs reflect the current workspace layout.
• src/features is documented as the home for cross-route product behavior.
• App-local external API wrappers now live under src/integrations, distinct from root services/* runtimes.
• Shared recommendation input screening and async job orchestration are extracted behind the feature layer.
• Recommendation thresholds and limits are centralized in a single feature config module.
• Recommendation persistence and DB row mapping are extracted behind a feature-owned adapter.
• More-picks persistence and quiz-data loading are extracted behind a feature-owned adapter.
• The reusable more-picks pipeline is extracted into the recommendation feature layer.
• The more-picks route and worker now share feature-owned orchestration instead of duplicating queue fallback and processing logic.
• Movies catalog data access and response types are extracted behind a feature module instead of living in the route file.
• The register route now delegates user-creation workflow to a feature module instead of owning DB writes directly.
• Poster utility routes now delegate TMDB fetch and proxy logic through the integration layer instead of owning raw fetch behavior.
• Recommendation feature modules now import concrete clients directly instead of relying on the broad @/clients barrel.
• A stable movie identity helper exists for recommendation memory, preferring TMDB ids and falling back to normalized title plus release year.
• Favorite/reference movies mentioned in the quiz are excluded from recommendation candidates while still contributing to taste matching.
• Recommendation feedback is captured from the results page and stored separately from generated recommendation payloads.
• Signed-in feedback is persisted into user_movie_interactions with upsert semantics for watched, liked, not_interested, and wrong_mood.
• Feedback-derived recommendation memory now filters watched/not-interested movies and down-ranks wrong-mood movies for signed-in users.
• Liked movie memory now applies a conservative positive ranking boost for matching signed-in recommendation candidates.
• Account recommendation history is deduplicated by movie identity so repeated recommendation attempts do not appear as separate discoveries.
• Result pages expose a share action that creates/copies a stable recommendation URL.
• New users are signed in automatically after registration when the session secret is configured.
• Password reset request and confirmation routes exist, with Resend delivery in production and non-production reset URL exposure for local testing.
• A dedicated /account/movie-memory experience exists with candidate cards and catalog search backed by /api/account/movie-memory.
• Account movie-memory API orchestration now lives behind a feature-owned service module instead of the route handler.
• Movie-memory candidate cards keep session state locally, submit completed choices in one batched request, and flush pending choices on page hide or unmount.
• The account movie-memory view supports large histories with paginated loading, virtualized rendering, total counts, and poster-aware fallbacks.
• The available-movies catalog now supports exact-title escape-hatch search with optional year-range filters across the API and page UI.
• The available-movies table now has actionable empty states for empty catalogs and filter searches with no matches.
• Expensive recommendation/poster POST routes now reject oversized JSON bodies before validation, moderation, queue creation, TMDB fetches, or OpenAI work.
• OpenAI calls now use per-call timeout options and AbortSignal cancellation, with legacy recommendation requests mapping upstream OpenAI timeouts to HTTP 504.
• tmdb_match_reviews persists ambiguous TMDB/local matches and runtime mismatches for later manual review.
• PR CI has a consolidated services-ci pass for service workspaces and CodeQL runs for Actions and JavaScript/TypeScript.

Issues Still Present

• The current quiz is still a first-generation guided flow. It should evolve toward a signal-based recommendation model with a shorter "tonight" quiz, a swipe-based mode for movie-heavy users, and a TMDB-first catalog strategy. See RECOMMENDATION-ROADMAP.md.
• Route-local compatibility re-export files still exist under src/app/api/movie-recommendation; future recommendation changes should continue moving real logic into src/features/recommendation.
• Account settings/profile/provider identity remain intentionally thin.

Reference: use BOUNDARIES.md as the current ownership baseline.

Backlog Hygiene

• Create a GitHub issue for every actionable roadmap ticket before or alongside adding it to this document.
• If a roadmap item is too large for one PR, keep the original issue as an epic/umbrella and create focused child issues for implementation-sized work.
• Link roadmap items to concrete issues whenever possible so completed work can be checked off and future agents do not have to rediscover context.
• Keep unlinked bullets for direction-setting only; convert them into issues once they become actionable.

Target Direction

PopChoice should continue evolving within the current workspace layout toward clearer ownership and easier extractability, for example:

• apps/web
• apps/bull-board
• services/movie-discovery
• services/movie-seed
• services/movie-backfill
• packages/domain
• packages/config
• packages/clients
• packages/shared
• optional: packages/ui

This is an extraction direction, not a mandate for large-scale file moves right now. Current work should optimize for clean boundaries inside the existing workspace first.

Guiding Principles

• Prefer explicit boundaries over convenience imports.
• Keep route handlers thin; move orchestration into reusable modules.
• Separate domain logic from infrastructure details (DB, API clients, queues).
• Minimize coupling and avoid hidden dependencies through broad utility barrels.
• Standardize shared tooling only after boundaries are stable.
• Refactor in small, reversible steps with existing tests and CI gates.

Phased Plan

Phase 1: Stabilize boundaries in the current workspace

• Define and document ownership for app, domain, infrastructure, and shared helpers.
• Reduce cross-layer imports that bypass intended boundaries.
• Keep API routes focused on validation, orchestration calls, and response mapping.
• Align docs with the real workspace structure before making larger architectural claims.

Phase 2: Refactor for extractability

• Move reusable domain logic into cohesive modules that can be lifted out later.
• Isolate external integrations behind stable client/service interfaces.
• Centralize configuration and constants to reduce manual synchronization.
• Prefer extracting recommendation flows out of src/app/api into clearer domain-owned modules.
• Refactor the quiz submission lifecycle so recommendation creation is modeled explicitly instead of coordinated through route-local useEffect, refs, and navigation timing.
• Keep account/movie-memory orchestration behind feature-owned modules as the API surface grows beyond the current service extraction.

Phase 3: Extract intentional packages from the existing layout

• Use the existing apps/, services/, and packages/ layout more intentionally as boundaries become clearer.
• Move modules incrementally to preserve behavior and delivery speed.
• Keep migration steps explicit and reversible.

Phase 4: Standardize shared tooling/config/testing

• Consolidate shared TypeScript, lint, formatting, and test conventions.
• Reuse config packages where duplication exists across app/services.
• Align CI checks to the new boundaries and ownership model.

CI/CD and Deployment Track

• Build production container images in GitHub PR/CI once, publish them with commit/PR metadata, and document preview or downstream deploys running those already-built images instead of rebuilding the monorepo in each deployment environment.
• Preserve provenance between a PR check, container digest, deployed preview, and /api/build metadata through GHCR labels, digest artifacts, runtime image metadata, and Docker-baked fallbacks.
• Run Coolify from GHCR images via a single IMAGE_TAG release bundle instead of compiling PopChoice services on the VPS.
• Add an optional Coolify deploy webhook path after successful development image publishing.
• Define the staged local -> development -> production deployment model in #556, including DNS wildcard usage, Coolify service-domain mapping, development vs production resources, immutable production image tags, and preview cleanup/certificate rate-limit guidance.
• Keep deployment-time work focused on migrations, health checks, runtime configuration validation, and compatibility checks rather than application compilation.
• Add a dedicated migration release gate so database migrations are visibly completed before rolling long-running web/worker services.
• Add release compatibility checks that verify all running PopChoice services report the same commit/image tag.

Operational Observability Track

• Build the self-hosted observability stack under #498. This is intentionally a learning track: SaaS tools would be simpler, but self-hosting teaches uptime checks, log shipping, metrics, traces, alerts, retention, and backups.
• Add Uptime Kuma health and synthetic monitoring in #502 for /api/health, build metadata, and cheap smoke checks before deeper instrumentation. See Uptime Kuma Monitoring.
• Add Grafana Loki log aggregation in #499 so Coolify/Docker logs are searchable by service, level, request id, recommendation id, queue, job id, and stage. See Observability Logs.
• Add Prometheus metrics and Grafana dashboards in #501 for container health, Postgres, Redis, BullMQ queues, recommendation latency, provider timeouts, and failed jobs. See Observability Metrics.
• Add OpenTelemetry traces with Tempo in #500 once the low-noise uptime/logs/metrics foundation exists. See Observability Traces.
• Add alert routing, retention, backups, and incident runbooks in #503 so the monitoring stack stays useful and recoverable. See Observability Alerts and Observability Runbooks.
• Deploy the self-hosted observability stack to production in #508, including Coolify wiring, secrets, access control, alert contact points, backup coverage, and post-deploy verification.
• Improve Telegram alert readability in #525 so mobile notifications show concise firing/resolved state, service, instance, severity, runbook, and silence links instead of raw Grafana expression payloads.
• Tune deploy-sensitive metrics target alerts in #526 so scrape visibility gaps during ordinary redeploys are not treated like confirmed user-facing P1 outages.
• Add deployment-aware alert silences and post-deploy verification in #527 so normal Coolify redeploys do not create avoidable alert noise while failed or unrecovered deploys still notify clearly.
• Enable Coolify notifications for failed deploys, failed backups, server disk/resource warnings, and unhealthy or restarting containers where supported.
• Add external uptime monitoring for https://pop-choice.shchilkin.dev/api/health, then consider a deeper synthetic recommendation smoke check.
• Add application error tracking for frontend, API routes, and workers so browser errors, API exceptions, and background job failures are visible outside Coolify logs.
• Improve structured log correlation by carrying requestId, recommendationId, recommendationSlug, jobId, and stage through web and worker logs.
• Make BullMQ job failures easier to diagnose by logging retry count, final failure state, recommendation identifiers, and stage at failure.
• Ship Docker/Coolify logs to a searchable log store once local log scrolling becomes painful.
• Track lightweight operational metrics: recommendation success/failure counts, average recommendation duration, queue depth, failed jobs, OpenAI/TMDB timeout counts, and DB/Redis health failures.

Security and Reliability Track

• Add per-call OpenAI timeout handling with cancellation where supported, and map upstream timeout failures to clear 504-style API responses.
• Add request body size limits for externally facing routes before expensive parsing, moderation, embedding, or recommendation work begins.
• Add retry/backoff and circuit-breaker behavior for expensive external dependencies where retrying is safe.
• Sanitize client-facing error responses so internal exception details, upstream payloads, and infrastructure hints stay out of API responses.
• Validate required environment variables on application startup for web, workers, and root services so misconfigured deployments fail early.
• Clarify idempotency and retry behavior for recommendation creation, worker retries, more-picks jobs, and failed queue recovery.
• Add a shared operator login model for operational apps in #548. apps/bull-board now supports shared Basic Auth with optional fail-closed behavior, and future backoffice routes should reuse the same operator-auth contract instead of embedding admin access in the user-facing web app.
• Add dependency/security scanning, static security checks, and periodic security review expectations to CI or maintenance workflows.

Data Quality Track

• #471: add a catalog metadata model for cast, directors, genres, and keywords before expanding search beyond title/year.
• #472: extend TMDB backfill/discovery to populate people, genre, and keyword metadata, with catalog-health visibility for missing coverage.
• Split the backoffice/catalog-health epic #493 into implementation-sized child issues before broad implementation starts.
• Define the dedicated backoffice app boundaries, routing, deployment, and ownership model in #547. The backoffice is planned as apps/backoffice, deployed like apps/bull-board, not as UI inside apps/web.
• Build the protected catalog-health overview in #549, covering duplicate movie identities, missing posters, missing localized names/overviews, missing runtimes, stale TMDB metadata, and missing cast/director/genre/keyword coverage.
• Persist TMDB backfill review cases in tmdb_match_reviews, then expose a protected TMDB match review queue in #550 for ambiguous matches, missing metadata, and rejected runtime/year confidence cases.
• Add safe review actions and audit/history rules in #551 before allowing operators to apply, reject, or defer catalog fixes.
• Periodically refresh TMDB-backed metadata for older records without destabilizing existing recommendation history.
• Make seed, discovery, and backfill responsibilities explicit enough that data-quality fixes do not duplicate or fight each other.
• Define migration/versioning expectations for schema changes in #494, including production migration safety, rollback notes, and seed/backfill coordination.

Account Platform Track

• Add a user profile model for display name, avatar, and account settings metadata.
• Add account settings APIs and UI for profile edits, saved recommendation edits, and taste-profile management.
• Design a provider identity model before adding magic-link or social login so local credentials and external providers can coexist cleanly.
• Add saved-recommendation mutations for rename, annotate, remove, and organize actions without leaving the account page.
• Make taste memory inspectable and editable so users can correct watched, liked, not-interested, and wrong-mood signals.
• Plan scalable account memory views before the list grows: search, signal filters, pagination or virtualized lists, and compact rows for large watched/liked histories.

Accessibility and UI Quality Track

• Add recurring accessibility checks for keyboard navigation, focus states, labels, color contrast, and reduced-motion behavior.
• Add focused tests or visual checks for the quiz, loading/results handoff, account pages, and feedback controls.
• Keep design-system examples aligned with production components so UI regressions are easier to spot before release.

Testing and Evaluation Track

• #474: add a Playwright e2e harness with an isolated migrated test database and deterministic seed fixtures.
• #475: cover auth, catalog, quiz, recommendation, and feedback smoke flows through product-level e2e tests.
• #476: add an AI recommendation eval harness with deterministic fixtures by default and optional live model/provider runs.
• #490: add a scheduled or manually triggered real-data recommendation eval workflow with seeded database data and real catalog retrieval.
• Keep Storybook/component tests separate from full product e2e tests so UI component regressions and app-flow regressions fail with clear ownership.
• Keep AI evals separate from normal e2e smoke tests because recommendation quality gates need fixture scoring, model controls, and optional API cost.

Product Feedback Track

• Expand the explicit movie-memory experience so watched/not-seen setup feels complete for users with large histories.
• Expand liked feedback beyond exact candidate boosts into a richer positive taste signal once the canonical signal model exists.
• Consider a separate "worth rewatching" angle for watched movies so strong matches can still appear intentionally, with copy that frames them as rewatch candidates instead of new discoveries.
• Make the reason for reused titles transparent when feedback history intentionally allows a repeat.
• Manual watched-list management, rewatch mode, richer preference editing, and gamified taste history can follow after the core memory behavior is stable.
• Continue polishing the dedicated movie-memory experience around exact-title search, empty states, and large-history review now that deck state and batched submission are in place.
• Keep manual movie search as a secondary escape hatch for exact titles. Movie memory and available-movies now both expose catalog search; extract a shared catalog-search component if another surface needs the same controls.
• Treat posters and localized metadata as first-class data quality requirements for movie memory. Candidate cards should degrade gracefully, but missing poster coverage should be visible in catalog-health reporting.
• Add a stable way to avoid recommending movies the user just marked as watched, not-interested, or wrong-mood, while still allowing an intentional "rewatch" recommendation mode later.

TMDB and Catalog Expansion Track

• Pivot recommendation retrieval toward TMDB-backed catalog coverage instead of relying primarily on the embedded/course-sized movie database.
• Keep the local movies table as a cache/index of known titles, embeddings, localized names, poster URLs, and TMDB ids rather than the full source of truth.
• Backfill TMDB ids for existing local movies using exact title/year matches first, then persist ambiguous or low-confidence matches for manual review.
• Add cast, director, genre, and keyword metadata as first-class catalog data before implementing actor/director/genre search.
• Move TMDB discovery, backfill, and metadata refresh into shared rate-limited BullMQ catalog workers in #492 before growing catalog volume. The worker enforces one configurable TMDB request budget across catalog-maintenance jobs, honors 429 with backoff, dedupes jobs by stable tmdbId/movieId keys, and exposes queue depth/failures in Bull Board.
• Add a back-office review queue for ambiguous TMDB matches, missing posters, duplicate identities, and metadata conflicts in #493 before applying risky automatic merges.
• Prefer TMDB ids for all cross-feature identity checks. Fall back to normalized title plus year only when TMDB identity is unavailable.
• Design future discovery flows around dynamic TMDB candidate sets: "I have watched many films" deck mode, quiz-assisted mode, and later a preference/taste-training mode.

Account Experience Track

• Add magic-link style login options, with rate limits and email delivery observability.
• Add social login/provider linking after the provider identity model is stable.
• Add profile editing for display name, avatar, and basic preferences.
• Add account achievements, taste progress, and gamified memory-building only after the underlying movie-memory signals are reliable.
• Add feedback loops that explain how recommendation feedback changes future recommendations.

Recommendation Experience Track

• Treat quiz answers, swipe reactions, account memory, and result feedback as inputs into a shared taste-signal model.
• Rework the guided quiz around "what do you want tonight?" instead of relying on a favorite movie, broad genre labels, and optional actor input.
• Add an alternate taste-swipe mode for users who have watched many films and prefer to react to concrete movie cards instead of answering abstract questions.
• Move toward TMDB-first candidate generation: use TMDB for broad discovery and keep the local database as a cache/enrichment/reranking layer rather than the whole movie universe.
• Keep TMDB ids as the preferred movie identity and log ambiguous title/year matches for later admin/back-office review.
• See RECOMMENDATION-ROADMAP.md for the staged plan.

Group Recommendation Rooms Track

• Keep #359 as the umbrella for the group-room milestone instead of treating it as a single implementation PR.
• #467: build room persistence, TTL, cleanup, and participant storage first.
• #468: add share links, participant join flow, and readiness state.
• #469: run the recommendation pipeline from completed room answers and persist a stable shared result.
• #470: add QR invite and projector mode only after the core room flow works.
• Preserve the current same-device group mode until room-backed group mode is complete enough to replace it intentionally.

Priority Items for the Next 30 Days

1. Complete #81 with available-movies runtime, score, and age-rating filters on the existing catalog fields.
2. Refactor the quiz submit/results handoff in #484 so navigation state is explicit and the quiz page does not need short-lived reset guards.
3. Start #474 so full e2e work has a real isolated DB foundation before adding many browser scenarios.
4. Add #475 auth, catalog, quiz, and recommendation smoke flows on top of the isolated e2e harness.
5. Add #476 deterministic recommendation eval fixtures and scoring.
6. Add #490 scheduled/manual real-data recommendation evals for seeded DB and catalog-retrieval changes.
7. Decide the catalog metadata model in #471 before expanding #82 into actor/director/genre search.
8. Populate the catalog metadata model in #472 from TMDB backfill/discovery before expanding #82 into actor/director/genre search.
9. Expand available-movies search in #473 across title, actor/director, and genre metadata populated by #472.
10. Move TMDB discovery/backfill/metadata refresh into shared rate-limited BullMQ catalog workers in #492 before increasing catalog expansion volume.
11. Plan and split the #493 backoffice/catalog-health epic, including shared login protection for it and apps/bull-board.
12. Clarify production migration/versioning expectations in #494 for schema changes, rollbacks, and preview volume recreation.
13. Complete the first self-hosted observability track in #498 with uptime, logs, metrics, traces, alerts, retention, backups, and runbooks.
14. Deploy the self-hosted observability stack to production in #508 and verify metrics, logs, traces, alerts, access control, and backups on the VPS.
15. Improve Grafana Telegram alert formatting in #525 after the first production alert examples.
16. Reclassify deploy-sensitive app metrics target alerts in #526 so P1 remains reserved for user-facing or core dependency outages.
17. Plan deploy-aware alert silences and post-deploy verification in #527.
18. Define the local -> development -> production deployment model in #556, then split implementation follow-ups if production promotion, domain layout, or rollback automation grows beyond documentation.

Working Checklist

Boundary Cleanup

• Route handlers validate input, call orchestration, and map responses without owning business logic in the current API surface.
• Domain orchestration does not depend on route-local module placement.
• Infrastructure concerns stay behind clients, repositories, or queue adapters.
• Shared exports remain intentional and do not hide ownership in cross-boundary code paths.

Recommendation Flow

• A reusable recommendation pipeline exists.
• Recommendation pipeline ownership is no longer tied to src/app/api/movie-recommendation.
• Queueing, DB writes, and response mapping are separated cleanly from recommendation decision logic.
• Similarity thresholds and related calibration docs point to the same source of truth.
• Positive user memory (liked) influences ranking.
• Quiz submit handoff no longer depends on route-local reset timing.

Documentation Alignment

• README reflects the current apps/web/src structure.
• Development docs reflect the current apps/, packages/, and services/ layout.
• Service docs and architecture docs use the same terminology for boundaries and ownership.
• Agent guidance exists in root AGENTS.md.

Non-Goals (For Now)

• No large-scale restructuring just to make the repo look more "monorepo-like".
• No large-scale file moves that mix structural and behavioral change.
• No broad rewrite of working features.
• No premature package splitting before clear ownership boundaries exist.