ARCHITECTURE.md — AI Software Factory

Technical architecture. Detailed deep-dives live in docs/architecture/ (system, frontend, backend, database, realtime, search, ai, security). If this file conflicts with PRODUCT-SPEC.md, this file wins and the conflict must be recorded (see AGENTS.md).

1. System Overview

USERS
  │
  ▼
┌───────────────────┐
│ Next.js Web App   │
└─────────┬─────────┘
          │  HTTPS / WebSocket
          ▼
┌───────────────────────────┐
│ API / BFF (NestJS)        │  modular monolith
└─────────────┬─────────────┘
              │
   ┌──────────┼──────────────────────┐
   ▼          ▼                      ▼
PostgreSQL  Redis            Object Storage (S3)
   │          │                      │
   │    ┌─────┴─────────┐            │
   │    ▼               ▼            │
   │  BullMQ        Realtime        │
   │  Worker(s)     (WebSocket)     │
   │    │               │           │
   └────┴───────┬───────┴───────────┘
                ▼
        Events → Automations / Webhooks / Search indexer / AI

Services:

App	Runtime	Responsibility
apps/web	Next.js 14 (App Router)	UI, block editor, database views, offline client state
apps/api	NestJS	REST API v1, auth, all domain modules, event bus (NATS-free: domain events → BullMQ/Redis streams)
apps/worker	Node + BullMQ	Automation runs, file pipeline (scan/thumbnails), search indexing, webhooks, AI jobs, email
apps/realtime	Node + ws	Yjs CRDT sync, presence, cursors, realtime comments/notifications

Architectural Invariants

1. Layering: Editor Model → Domain Model → Application Service → Repository → PostgreSQL. The editor never touches PostgreSQL. Business logic never imports Prisma directly outside repositories.
2. Modular monolith: NestJS modules — AuthModule, UsersModule, OrganizationModule, WorkspaceModule, PageModule, BlockModule, DatabaseModule, RelationModule, FileModule, CommentModule, SearchModule, NotificationModule, AutomationModule, AIModule, IntegrationModule, AuditModule. No microservice split until scaling justifies it; module boundaries are the future service boundaries.
3. Authorization is server-side, always. Every handler passes through the permission guard in packages/permissions. The frontend is a convenience layer only.
4. Ordering is fractional. Block/page/property/record ordering uses LexoRank-style string positions. Array indexes are never persisted as ordering.
5. Events, not calls, for side effects. Domain operations emit domain events (page.created, record.updated, …) onto the event bus; workers consume for search indexing, webhooks, automations, notifications. This keeps write paths fast and makes the automation/webhook systems decoupled.

2. Frontend Architecture (apps/web)

apps/web
├── app/            # Next.js App Router: (auth)/, (app)/[workspace]/, /share/[slug], /api (route handlers for BFF)
├── components/     # app-level composites (top bar, dialogs, command palette)
├── editor/         # block editor wiring: Tiptap instances, slash menu, context menus, DnD
├── database/       # view renderers: table, board, calendar, gallery, timeline, chart, form, dashboard
├── workspace/      # sidebar, page tree, teamspaces, settings UI
├── collaboration/  # presence avatars, cursors, Yjs client bootstrap, offline indicator
├── search/         # search page, suggestions, filters
├── settings/       # workspace / user / billing / security settings
├── hooks/          # usePage, useBlocks, useDatabase, useRealtime, useOffline
├── stores/         # Zustand: editorState, uiState, presence, offlineQueue
├── lib/            # api client (fetch + TanStack Query), ws client, utils, formatters
└── types/          # frontend-only types (DTOs come from @factory/shared)

Key decisions:

• TanStack Query owns server state (pages, databases, records, search, notifications). Zustand owns client-only state (editor caret/selection, UI panels, presence, offline queue).
• React Hook Form + Zod for all forms (forms, settings, auth).
• Tiptap (ProseMirror) powers the block editor; each Notion block type maps to a Tiptap extension/node defined in packages/editor. The document is a block tree, serialized to our canonical Block JSON — not raw HTML.
• dnd-kit for all drag & drop (blocks, columns, kanban rows, dashboard widgets, sidebar).
• Virtualization (@tanstack/react-virtual) for large pages and databases (100k+ records).
• Offline: Yjs local-first document + IndexedDB persistence (@factory/collaboration). Edits apply locally, sync on reconnect; CRDT resolves conflicts.
• Theme: design tokens in packages/ui (CSS variables) — light/dark/system, no hard-coded colors.
• Accessibility: WCAG 2.2 AA. Every interactive block exposes a keyboard path and ARIA semantics; drag & drop has non-drag alternatives (context menu "Move to", arrow-key move).

3. Backend Architecture (apps/api)

NestJS modular monolith:

apps/api/src
├── main.ts
├── app.module.ts
├── common/           # guards (auth, permission, rate limit, api key), interceptors, filters, pipes
├── config/           # env validation (zod), feature flags
├── modules/
│   ├── auth/         # register, login, logout, refresh rotation, password reset, email verification
│   ├── users/
│   ├── organizations/
│   ├── workspaces/   # workspace CRUD, members, invitations, teams, settings
│   ├── pages/        # page CRUD, nesting, tree, trash, favorites, versions
│   ├── blocks/       # block CRUD, batch operations, fractional positioning, synced blocks
│   ├── databases/    # databases, properties, views, records (record = page), query engine
│   ├── relations/    # relations, relation links, rollups, formulas, buttons
│   ├── files/        # presigned upload/download, pipeline status
│   ├── comments/     # threads, replies, resolve/reopen, reactions, mentions
│   ├── search/       # query API over OpenSearch, suggestions
│   ├── notifications/
│   ├── automations/  # automation CRUD, manual trigger, run history
│   ├── ai/           # AI gateway endpoints, chat, RAG, generation (structured tool calls)
│   ├── integrations/
│   ├── api-keys/
│   ├── webhooks/
│   ├── share/        # public share links + password + expiry
│   ├── templates/
│   └── audit/
└── events/           # domain event bus (Redis streams), event producers/consumers

• DTOs + Zod validation shared with the frontend via packages/shared — the same schemas validate API input and define client types.
• Repository pattern: each domain has *Repository interfaces (Prisma implementations live in common/repositories). Services depend on interfaces, never on PrismaClient directly.
• Domain events: EventBus.publish('page.created', payload) → Redis stream → workers index into search, dispatch webhooks, run automations, notify.
• Error model: typed ApiError with stable codes (PERMISSION_DENIED, NOT_FOUND, VALIDATION_ERROR, …) mapped to consistent JSON errors (see docs/api/).
• Rate limiting (Redis sliding window) per user / API key / IP.
• Audit: AuditService.log(actor, action, resource, meta) in the same transaction where possible; mandatory for the event list in docs/security/.

4. Database Architecture (prisma/)

• PostgreSQL as the system of record. Single Prisma schema: prisma/schema.prisma.
• Migrations are the only way to change schema (prisma migrate dev / migrate deploy).
• Block model: blocks table, tree via parentId, fractional position, content JSONB + properties JSONB. block_versions for history.
• Database model: databases → database_properties (typed schema) → database_records (each row 1:1 with a pages row) → database_views (per-view filter/sort/group/visibility settings as JSONB).
• Relations: relations (typed, cardinality) + relation_links (m2m join table). Rollups and formulas are property-level records evaluated in packages/database-engine (safe AST evaluator — never eval).
• Permissions: RBAC (workspace roles) + permissions table (grantee × resource × level) evaluated by packages/permissions.
• Multi-tenancy: every tenant-scoped table carries workspaceId (and organizationId where relevant); all queries are scoped by tenant in the repository layer. Row-level filtering is enforced by the application, never left to the client.
• Search: search_documents mirror table + OpenSearch index (built by the indexer worker, not by the API).
• AI: ai_conversations, ai_messages, ai_embeddings (chunk references into the vector store).
• Full model reference: docs/database/database-architecture.md.

5. Realtime Architecture (apps/realtime)

Browser (Yjs provider)
   │  WebSocket (token-authenticated)
   ▼
Realtime Server
   ├── Rooms per (workspace, page) / (workspace, database)
   ├── Yjs awareness protocol → presence + cursors + avatars
   ├── Document updates → applied to Yjs Doc (server authoritative)
   │        └── on update: persist to Postgres (debounced) + emit domain events
   └── Comments / notifications fan-out

• Yjs CRDT is the source of truth while editing; the persisted blocks table is the source of truth at rest. On connect, the server hydrates the Yjs doc from Postgres.
• Server-side presence heartbeat; stale peers pruned.
• Backpressure + per-connection rate limits; reconnect with Yjs update resume (document update log kept in Redis, truncated after persistence).
• Realtime comments and notification pushes ride the same socket.

6. Search Architecture (packages/search)

PostgreSQL (source)
   │  domain events + periodic reconciliation
   ▼
Search Indexer (worker)
   │  chunking, embedding (AI), permission metadata
   ▼
OpenSearch (full-text + vector + metadata filters)
   ▲
   │
Query API (apps/api/modules/search)

• Full-text, prefix, fuzzy (via OpenSearch analyzers + fuzzy queries).
• Hybrid AI search: keyword BM25 + vector semantic + metadata filters → reciprocal rank fusion → optional LLM answer with citations (RAG).
• Results are permission-filtered at query time (grants stored per document).

7. AI Architecture (packages/ai)

App / Worker
   │
   ▼
AI Gateway (provider abstraction: unified chat/embed/tools API)
   ├── OpenAI
   ├── Anthropic
   ├── Google
   ├── Open-source (vLLM / Ollama / OpenAI-compatible)
   └── Custom OpenAI-compatible endpoint
   │
   ▼
Capabilities: chat, summarize, rewrite, translate, extract, generate
   (page / database / formula / template / tasks) — via STRUCTURED TOOL CALLS,
   never raw SQL or raw block mutation.

• Workspace AI search (RAG): content chunker → embedding service → vector store → hybrid retrieval → LLM answer with source citations.
• All AI calls logged (model, tokens, latency) for observability + cost control.
• Provider config is per-workspace (enterprise) with platform default fallback.

8. Storage & File Pipeline (packages/storage)

Upload (presigned S3 PUT)
   ↓
File record (status=UPLOADING)
   ↓
Virus scan (ClamAV worker)
   ↓
Metadata extraction (mime, dimensions, page count, duration)
   ↓
Thumbnail / preview generation
   ↓
Object storage (S3-compatible, MinIO in dev) + metadata in Postgres
   ↓
file.uploaded event → search index + permissions

• Private buckets; downloads via short-lived signed URLs.
• File versions via file_versions.

9. Automation & Events

Domain op → DomainEvent (Redis stream)
   → Automation trigger matcher → condition evaluator → action executor (BullMQ)
   → Webhook dispatcher (retry queue with exponential backoff + signatures)
   → Notification fan-out (in-app / email / webhook)

• Triggers: record created/updated, property changed, page created, form submitted, schedule (cron), webhook, manual button.
• Actions: update record, create page, send email, send webhook, notify, assign user, change status, call integration, run AI action.
• Every run is recorded (automation_runs + automation_logs): status, input, output, error, timing. Retries with exponential backoff, dead-letter queue.

10. Security Architecture (summary — see docs/security/)

• AuthN: email+password (bcrypt), JWT access tokens (15 min) + rotating refresh tokens (opaque, hashed at rest, session table), email verification, password reset. Enterprise: SAML/OIDC SSO, SCIM provisioning.
• AuthZ: RBAC (Owner/Admin/Member/Guest) + resource permissions (None/View/Comment/Edit/Full Access) at organization/workspace/teamspace/page/ database/record/file. Evaluated server-side on every operation.
• Transport: HTTPS, secure cookies (HttpOnly, SameSite=Lax), strict CORS.
• Abuse: rate limiting, CSRF tokens for cookie-authenticated state changes.
• Data: input validation (Zod everywhere), parameterized queries (Prisma), output sanitization, file validation + malware scanning, API key hashing, at-rest secret encryption (envelope encryption, key from env).
• Accountability: audit log for all security-sensitive operations (actor, workspace, resource, action, timestamp, IP, UA, metadata).
• Full checklist: docs/security/security-architecture.md + SECURITY.md.

11. Deployment & Infrastructure (infrastructure/)

• Local dev: infrastructure/docker/docker-compose.yml — postgres, redis, minio (+minio console), opensearch, and (optionally) api/web/worker/realtime.
• Production:
  • infrastructure/nginx/ — reverse proxy (TLS termination, WS upgrade, gzip, security headers).
  • infrastructure/kubernetes/ — manifests: deployments (api ×N, worker ×N, realtime ×N, web), HPA, ingress, PDBs; secrets via external secret store.
  • infrastructure/terraform/ — cloud provisioning reference (RDS/ElastiCache/S3 or self-hosted equivalents).
• Observability (Phase 15): OpenTelemetry traces/metrics → Prometheus + Grafana; Loki for logs; alerting on API latency, queue depth, WS connection count, error rate.
• Backup/DR: Postgres PITR + S3 versioning; documented restore runbook.

12. Testing Architecture

Layer	Tool	Location
Unit	Vitest (packages), Jest (api)	co-located *.test.ts / *.spec.ts
Integration	Jest + Testcontainers (PG, Redis, OpenSearch, MinIO)	tests/integration
E2E	Playwright	tests/e2e
Performance	k6 / Artillery	tests/performance

Critical unit-tested engines (must have coverage): formula engine, permission engine, block operations (fractional positions), database filtering/sorting/grouping, relations, rollups, Yjs doc ↔ block tree codec.

13. Key Third-Party Decisions

Concern	Choice	Why
Editor	Tiptap (ProseMirror)	Block-level nodes, extension system, first-class collaboration (Yjs)
Sync	Yjs + y-prosemirror	Battle-tested CRDT, awareness protocol for presence
DnD	dnd-kit	Accessible, composable, multi-container (columns, pages, kanban)
Server state	TanStack Query	Caching, retries, offline mutations
Client state	Zustand	Tiny, no boilerplate, devtools
Forms/validation	React Hook Form + Zod	Shared schemas between FE/BE via packages/shared
DB	Prisma + Postgres	Type safety, migrations, JSONB for flexible block/property content
Queue	BullMQ + Redis	Simple, reliable, delayed jobs, rate limits, retries
Search	OpenSearch	BM25 + kNN vectors + metadata filters in one engine
Storage	S3-compatible (MinIO dev)	Self-hostable, standard SDK