ROADMAP.mdROADMAP.md← root20,967 B← Back to the app

ROADMAP.md — Development Phases

The project is built in 16 sequential phases. Never jump between features. Each phase maps to a task file in docs/tasks/ containing the full TASK-XXXX spec sheets. A phase is done when every task in its file meets the Definition of Done in AGENTS.md.

Flow:

SPECIFICATION → ARCHITECTURE → DATABASE → BACKEND → FRONTEND → REALTIME
→ AI → TESTING → SECURITY → PERFORMANCE → DEPLOYMENT

Phase 0 — Foundation ✅ scaffold created

Task file: docs/tasks/phase-00-foundation.md

[ ] Repository initialized (git, branches, hooks)
[ ] Monorepo (pnpm workspaces + Turborepo)
[ ] CI/CD (lint → typecheck → unit → integration → build → security scan → E2E)
[ ] Docker dev environment (postgres, redis, minio, opensearch)
[ ] Environment configuration (.env.example, zod-validated env)
[ ] Linting (ESLint + Prettier)
[ ] TypeScript (strict, shared tsconfig)
[ ] Testing bootstrap (Vitest/Jest/Playwright configs)
[ ] PostgreSQL connection + first migration
[ ] Redis connection
[ ] Object storage (MinIO) bootstrap
Exit criteria: pnpm install && pnpm docker:up && pnpm db:migrate && pnpm dev works from a clean clone; CI is green; all apps/packages build and typecheck.

Phase 1 — Authentication

Task file: docs/tasks/phase-01-authentication.md

[ ] Registration (email + password, validation, email verification)
[ ] Login / logout
[ ] Sessions (access JWT + rotating opaque refresh tokens, session listing/revocation)
[ ] Password reset (token, expiry, single-use)
[ ] User profile (view, update, avatar, security settings)
Exit criteria: full auth flow E2E-tested; refresh rotation proven; audit events for login/logout/reset; rate limiting on auth endpoints.

Phase 2 — Workspace

Task file: docs/tasks/phase-02-workspace.md

[ ] Workspace create / rename / delete, icon upload
[ ] Members + roles (Owner/Admin/Member/Guest), role changes
[ ] Invitations (email, accept/decline, revoke, expiry)
[ ] Teams / teamspaces + team membership
[ ] Sidebar (Home, Inbox, Search, Favorites, Teamspaces, Private, Shared, Trash)
[ ] Workspace settings (profile, billing placeholder, security, integrations placeholder)
Exit criteria: multi-user workspace flow E2E-tested; permissions enforced server-side; sidebar renders tree from API.

Phase 3 — Page Engine

Task file: docs/tasks/phase-03-pages.md

[ ] Page CRUD, nested pages, page tree with fractional positions
[ ] Page metadata: icon, cover, title, description
[ ] Trash (soft delete, restore, purge), archives
[ ] Favorites
[ ] Page permissions (inherit + override), sharing scopes
[ ] Page version history
Exit criteria: 100k-block page tree loads in < 2 s (virtualized); trash/restore round-trip E2E-tested; unauthorized access rejected with audit event.

Phase 4 — Block Editor

Task file: docs/tasks/phase-04-editor.md

[ ] Block types: paragraph, headings, lists, todo, quote, callout, code, image, file, video, divider, toggle, columns
[ ] Drag & drop (nested, cross-page, multi-block, indicators, auto-scroll)
[ ] Slash command menu (searchable, dynamic database creation)
[ ] Right-click context menus (block + page)
[ ] Full keyboard shortcut set
[ ] Undo/redo, copy/paste (internal + external markdown/HTML)
[ ] Fractional positioning engine (packages/editor positions module)
Exit criteria: all Phase-4 interactions E2E-tested; reorder latency < 16 ms locally; ordering survives reload; WAI-ARIA paths for all mouse operations.

Phase 5 — Database Engine

Task file: docs/tasks/phase-05-databases.md

[ ] Databases + typed properties (22 property types)
[ ] Records (each record = a page), create/update/archive
[ ] Table view (virtualized, 100k+ records)
[ ] Filters (nested AND/OR/NOT), sorting, grouping
[ ] Views: board, calendar, gallery, timeline, list (+ view settings: visible properties, column width, card layout, color, density)
[ ] Chart + dashboard views (base)
Exit criteria: 100k-record database renders with virtualization; filter/sort/ group engine unit-tested; all views open a record's page.

Phase 6 — Relations

Task file: docs/tasks/phase-06-relations.md

[ ] Relations (1:1, 1:N, M:N, bidirectional) + relation links
[ ] Rollups (COUNT, SUM, AVERAGE, MIN, MAX, PERCENT, UNIQUE)
[ ] Formula engine (safe AST evaluator; if/and/or/not/sum/average/min/max/ dateAdd/dateBetween/now/concat/replace/contains/length + prop())
[ ] Database buttons (set property, create/update/archive/duplicate page, webhook, trigger automation)
[ ] Sub-items & dependency tracking for computed properties
Exit criteria: formula engine 100% function coverage, zero eval; rollups recompute on relation change (event-driven); button actions E2E-tested.

Phase 7 — Collaboration

Task file: docs/tasks/phase-07-collaboration.md

[x] Realtime WebSocket server (rooms, auth, heartbeat, resume) — TASK-0091 (2026-09-27): the handshake (IP rate → JWT → room-permission order, the workspaces[] membership check, the REALTIME_TRUST_PROXY per-VU identity), the rooms (page/database doc rooms + the control room), the R4 heartbeat (30 s ws.ping, 90 s silence → 4000) + the 90 s presence prune, the connect-limiter (30/min/IP), the R5 resume (the per-room update log — a separate structure from the room, the warm sync.request {lastSeq} → missing-binary-delta + ack, the state-vector fallback), the R6 SIGTERM drain (4010 retryAfterMs + flush ≤ 10 s). Docker-free-green: realtime unit 109/109 + integration 59/59 (the resume log in tests/integration/realtime/resume.test.ts); the browser-side proof (the 7-scenario S13 suite's server-restart-recovery + reconnect-resume scenarios) is committed CI-ready (Docker-gated).
[x] Yjs CRDT integration (block tree ↔ Yjs doc codec) — packages/collaboration/src/codec.ts (blocksToYDoc/yDocToBlocks, minimal-op diff, two appliers); property-tested: 200-forest lossless round-trip, two-codecs-agree over random ops, 100-run mutation round-trip (19/19 codec tests, typecheck clean). Protocol wire contract + close codes frozen in packages/collaboration/src/types.ts.
[x] Presence (avatars, "who is here"), live cursors — TASK-0093 (under the (b) decision: JSON control frames, not the binary 0x02 awareness relay — that's TASK-0136). Server fan-out (join re-broadcast, the 50/51-peer avatars-only cap, the 90 s prune, the leave) + the web UI (the avatar bar with the +n overflow, the cursor overlay with the mandatory name label, the role=status live region ≤ 1/s, the join/leave toast toggle default-on, the database presence) are committed; Docker-free-green: shared 106/106, realtime unit 109/109, realtime integration 59/59 (incl. the < 500 ms cursor move + the cap boundary), web 85/85. The 3-client BROWSER E2E (tests/e2e/specs/presence-cursors.spec.ts) is committed CI-ready (Docker-gated; the join→avatar delta is measured against the 500 ms exit metric).
[x] Realtime block edits + realtime comments — block edits: under the (b) decision (per-block editor + the block:changed refetch overlay — the CRDT provider is TASK-0136), a committed edit flushes via blockApi.batch (the 500 ms debounce) + the realtime block:changed refetch-hint converges the OTHER tab (Docker-free-proven: tests/integration/realtime/convergence.test.ts; the browser proof — the S13 7-scenario concurrent-editing scenario, 20 interleaved ops zero loss + the Postgres at-rest assert — committed CI-ready, Docker-gated). Realtime comments: TASK-0094 (2026-09-27) — the comment.event room frame + the mention→notification fan-out (Docker-free-proven: tests/integration/realtime/task-0094-fanout.test.ts; the 300 ms UI E2E + the 3-user seam re-run in the S13 comments-300ms scenario — committed CI-ready, Docker-gated).
[x] Notifications (in-app + email + webhook channels) — TASK-0094 (2026-09-27): the in-app bell (the API surface unread-count/list/read/read-all + the realtime notification.push control-room fan-out — Docker-free-proven: tests/integration/realtime/task-0094-fanout.test.ts; the browser bell E2E committed CI-ready, Docker-gated, + the S13 7-scenario re-runs the live bell update); the email channel (the worker's email-decision.test.ts unit-proves the send/suppress/digest decision; the queue-level E2E is a recorded gap — no queue test exists); the webhook channel (the worker's webhooks.processor.ts + webhook-sign.test.ts — the HMAC-signed delivery; automation + admin-notification delivery). The bell's R3.7 a11y contract (the full Connection: … chip label + the banner role=status/aria-live=polite) is in the S13 a11y-offline scenario (committed CI-ready, Docker-gated).
[x] Offline mode (IndexedDB queue, reconnect, conflict resolution) — TASK-0095 S12 (2026-09-28): the outbox (a raw-IndexedDB factory-outbox store + the OutboxQueue worker: per-resource FIFO, the 500-item cap, the failed list, the 24 h GC — 20 unit tests, Docker-free-green) + the idempotency interceptor (the 6-case matrix — 11 unit tests) + the mutatingApiFetch offline gate (every mutation → isOffline() → the outbox enqueue; the online event drains with the Idempotency-Key) + the offline banner/chip/failed-dialog UI. Conflict resolution: under the (b) decision it is the LWW split (a title 409 → LWW-with-consent; a record property → plain LWW — 8 unit tests in packages/collaboration/src/lww.ts); the CRDT variant (y-indexeddb + the Yjs provider in the editor) is TASK-0136 (recorded, not landed). The browser proof (the S13 7-scenario offline-reconnect-sync: a block edit enqueued offline, drained on the online event, at-rest in Postgres, reconnect < 3 s + the a11y-offline R3.7 contract) is committed CI-ready (Docker-gated).
Exit criteria (evidence — TASK-0095, 2026-09-28): two-browser concurrent edit E2E (the S13 7-scenario suite, committed CI-ready/Docker-gated; the convergence loop is Docker-free-proven in convergence.test.ts); realtime update latency < 200 ms p95 (the S14 1k k6 baseline, thresholds enforced in-script, CI-gated); offline edit → reconnect sync (the S13 offline-reconnect-sync, committed CI-ready/Docker-gated; the outbox/idempotency mechanics Docker-free-green). All three are committed + CI-wired; the "passes / verified" evidence is the first green CI run (the S15 3-runs gate — Docker-gated, not run locally).

Phase 8 — Search

Task file: docs/tasks/phase-08-search.md

[x] Indexer (domain events → OpenSearch; permission metadata per document) — TASK-0096 (581b8ad5 + the TASK-0100 S0–S5 fixes that made it run): the event→doc→index pipeline (500-doc _bulk, idempotent docIds), the grant snapshot stamped at index time (grants — public/workspace/ teams/users + the creator-implicit-viewer model), deletes on trash/purge, the search_documents mirror, and the 15-min reconcile backstop. Proven live by the TASK-0100 S5 lag e2e.
[x] Full-text + prefix + fuzzy search — TASK-0097 (70060e58): POST /api/v1/search (multi_match BM25 + prefix + fuzzy, cursor pagination, highlights, fail-closed scoping, 400/503 SEARCH_UNAVAILABLE); the integration test proves tenant isolation (another workspace's doc never appears; a non-member workspaceId → 404 no-leak). (TASK-0100 S6: the route returns 200, matching docs/api/search.md.)
[x] Filters (type, workspace, team, updated date, creator), recent searches — TASK-0097 filters (type/workspace/team/date/creator) + TASK-0098 recent searches (persisted, per-user).
[x] Search UI (global search page, Cmd/Ctrl+K integration, suggestions) — TASK-0098 (4f61275a): the ⌘K dialog (225 ms debounce, keyboard nav, suggestions) + the /search page (grouped results, deep links, the loading/empty/error states).
Exit criteria (evidence — TASK-0100, 2026-09-29): search response < 500 ms p95 (full k6 profile over the 100k fixture: p95 49.9 ms / p99 92.8 ms — a ~10× margin; the CI perf-search lane gates reduced on merge-to-main + full nightly); index lag < 30 s (the 10-run E2E: create-side mean 3.7 s / max 4.3 s, delete-side 2.1 s); permission- filtered results proven by the integration test (tenant isolation) + live (e2e: a second user finds 0 private-page rows, the creator finds theirs). The "Full CI green" evidence is the first green CI run (CI-gated, Phase-7 close-out convention).

Phase 9 — Files

Task file: docs/tasks/phase-09-files.md

[ ] S3-compatible storage integration + presigned upload/download
[ ] File pipeline: virus scan → metadata → thumbnails
[ ] Previews (images, PDF, video/audio) + file block UI
[ ] File metadata, versions, permissions, trash
Exit criteria: upload of 500 MB file works (multipart); malware sample blocked; preview for each supported kind E2E-tested.

Phase 10 — Forms

Task file: docs/tasks/phase-10-forms.md

[x] Form builder (questions: type, required, validation, DB property mapping)
[x] Public form pages (slug, branding, closed state)
[x] Submission → database record (+ automation trigger)
[x] Form responses view
Exit criteria: public submission without auth creates a record and fires form.submitted; validation rejects bad input server-side. ✅ proven by tests/e2e/specs/forms.spec.ts (full lifecycle a–g, desktop + mobile).

Phase 11 — Automation

Task file: docs/tasks/phase-11-automation.md

[x] Automation CRUD (triggers, conditions, actions)
[x] Trigger matcher + condition evaluator
[x] Action executors (update record, create page, email, webhook, notify, assign, status, integration, AI action)
[x] Queue (BullMQ) + workers, retries with backoff, dead-letter
[x] Execution history + logs UI
Exit criteria ✅ (2026-09-30, see docs/tasks/phase-11-automation.md “Verification Result + Gap Report”): record-created → status-change automation E2E-tested (tests/e2e/automation.spec.ts, 14/14); failed action retries 8× then dead-letters (integration scenario 6, incl. admin notification — seam-driven hermetic, the only way to observe 8 backoff hops); every run logged (run + per-step automation_logs verified in E2E and integration).

Reserved task IDs: TASK-0115 … TASK-0120 are intentionally unallocated. They are reserved for tasks that may be added between Phase 11 and Phase 12 without renumbering the stable Phase 12+ IDs (TASK-0121 … TASK-0135).

Phase 12 — AI

Task file: docs/tasks/phase-12-ai.md

[x] AI gateway (OpenAI, Anthropic, Google, OSS, custom OpenAI-compatible) — TASK-0121
[x] Chat (Ask AI) with workspace context — TASK-0122
[x] Page AI (summarize, rewrite, improve, translate), writing tools — TASK-0123
[x] Database AI (generate database, extract data, generate formula) — TASK-0124
[x] Workspace AI search (RAG: chunker → embeddings → vector → hybrid → LLM) — TASK-0125
[x] AI page/template/task generation via structured tool calls — TASK-0126 (POST …/ai/generate: ref-based plan executed through domain services under the requester's permissions, rollback + 120 s timeout + cancel + Idempotency-Key; golden E2E ai-generate.spec.ts 2/2 + full Phase-12 AI suite 36/36)
Exit criteria: provider swap changes zero app code (TASK-0121 capability suite); RAG answer cites sources (TASK-0125 ai-rag.spec.ts); tool-call generation creates valid pages/databases (TASK-0126 ai-generate integration 6/6 + golden E2E); per-workspace token budget enforced (TASK-0121 + TASK-0126 budget suite, all four AI endpoints 402).

Phase 13 — API

Task file: docs/tasks/phase-13-api.md

[x] Public REST API v1 (pages, databases, records, search, files, comments, workspaces) — TASK-0127 (error envelope, per-key rate limits, auth, OpenAPI + bidirectional contract test)
[x] API keys (scoped, hashed, per-key rate limit) + OAuth2 (auth code) — TASK-0128 (key UI, one-time reveal, E2E) + TASK-0129 (auth-code + PKCE + consent, refresh rotation + reuse family-kill, RFC 7009 revoke; integration 20/20 + E2E 4/4)
[x] Webhooks (events, signatures, retry queue, delivery log) — TASK-0130 (13-event catalog single source, AES-GCM signing secret, X-Factory-Signature HMAC, worker-owned 30 s·2^(n−1) backoff → 8 attempts → DLQ + admin notification, resubmit, incoming-webhook signature/timestamp/replay gate, 5-tab developer hub with delivery inspector; integration 15/15 + E2E exit scenario 2/2)
[x] OpenAPI spec + interactive docs — TASK-0127 spec + docs surface; TASK-0130 added the webhook + incoming + OAuth operations — contract test 6/6 (SPEC ⊆ LIVE and LIVE ⊆ SPEC)
Exit criteria: OpenAPI contract matches live endpoints (CI check); webhook retry with backoff proven; API-key auth E2E-tested.

Phase 14 — Enterprise

Task file: docs/tasks/phase-14-enterprise.md

[x] SSO: SAML + OIDC — TASK-0131 (7be54855)
[x] SCIM user/group provisioning — TASK-0132 (30299a5f)
[x] Audit log viewer (filter, export) — TASK-0133
[x] Organization policies (enforced security settings) — TASK-0133
[x] Data retention policies (auto-purge, hold) — TASK-0133
[x] Admin console (org overview, usage, members, policies) — TASK-0133
[ ] Advanced permissions (policy expressions, guest restrictions) — no TASK in the phase file; out of scope for TASK-0131/0132/0133 (recorded: the guest restriction shipped is the policy's disableGuestPublicShare enforcement point; policy-expression permissions are a future task)
Exit criteria: SSO round-trip E2E; SCIM sync tested; retention job purges per policy (integration test); audit log covers all mandatory events. — all met.

Phase 15 — Production Hardening

Task file: docs/tasks/phase-15-production.md

[x] Security audit (OWASP ASVS L2) + dependency scanning — TASK-0134 (audit + hardening + tests/security suites; license allowlist)
[x] Performance testing (targets in docs/product/non-functional-requirements.md) — top-5 profiles + fixes in docs/development/perf-report.md; dev k6 lane report-only, the §26 p95 hard gates run in the CI staging lane
[x] Load testing (10k WS clients, 100k-record queries, 1k rps API) — tests/performance k6 matrix + seed-100k.mjs; staging-scale shapes gated in the CI staging lane (release.yml perf stages)
[x] Database optimization (indexes, partitioning for blocks/audit, query plans) — EXPLAIN-proven indexes via migration + query-shape fixes (record open ≤ 4 queries, batch save flat); partitioning NOT done (no EXPLAIN case at dev scale — recorded rule 29 in the task file)
[x] Caching strategy (Redis: page trees, search, permissions) — version-key tree cache (race test), permission TTL 60 s + event bust, search 5 s TTL (permission-keyed), column metadata 60 s
[x] Backup/restore + disaster recovery runbook — wal-g PITR + versioned daily base backups, OpenSearch snapshot repo, deployment.md §8 runbook + scratch-namespace drill (tests/integration/drill/); the executed-drill evidence is a staging act (deployment.md §8.8, release pipeline)
[x] Observability: OpenTelemetry, Prometheus, Grafana, Loki, alerting — OTel in all 4 services (correlation + span scrubber, unit-tested), infrastructure/observability/ scrape + 6 dashboards + 6 alert rules + alertmanager; dashboards-rendering + alert-firing proven in staging (Loki optional per spec — pino→stdout is primary)
Exit criteria: all perf targets met in tests/performance suite; restore drill documented + executed; dashboards + alerts live in staging. — everything locally verifiable is met and verified (k6 dev lane + EXPLAIN + query-count/cache tests green; DR runbook + drill tooling committed; observability as code with the dashboard-completeness check; the final full-flow E2E is green on desktop + mobile). The staging-gated evidence (k6 §26 p95 hard gates, the executed PITR drill, dashboards rendering live data + a fired alert) runs in the release pipeline against the staging cluster before the v1.0.0 tag — release.yml gates the tag on it.

MVP Milestone (first production release)

Authentication + Workspace + Pages + Block Editor + Drag/Drop + Slash Commands + Right-click menus + Database + Table View + Board View + Relations + Search + File Upload + Permissions + Comments — i.e. Phases 0–9 (partial).

Then: Calendar, Timeline, Gallery, Forms, Charts, Automation, Realtime, AI, API, Enterprise.