Development Guide — AI Software Factory

How to get a local environment running, what lives where, and the daily developer workflows. Reads: README.md (overview), AGENTS.md (rules), ROADMAP.md (phases), coding-standards.md (conventions), testing.md (test strategy), deployment.md (environments & infra).

---

1. Prerequisites

Tool	Version	Why	Check
Node.js	≥ 20 (22 recommended)	Runtime for all apps/packages	node -v
pnpm	≥ 9 (packageManager: pnpm@9.15.0)	Package manager (corepack-managed)	pnpm -v
Docker + Compose	any recent	Local Postgres, Redis, MinIO, OpenSearch	docker compose version
Git	≥ 2.40	Version control + hooks	git -v
(Optional) k6	latest	Performance tests	k6 version
(Optional) Artillery	latest	WS load tests	artillery -v

Environment constraints: Linux/macOS/WSL2 (native Windows works via WSL2). Rough local footprint: ~6 GB RAM for infra + ~2 GB for dev servers.

2. Local setup, step by step

# 1. Clone and enter
git clone <repo-url> ai-software-factory
cd ai-software-factory

# 2. Lock the toolchain (corepack reads packageManager from package.json)
corepack enable
pnpm -v                      # → 9.15.0

# 3. Install all workspace dependencies (root + apps/* + packages/*)
pnpm install

# 4. Configure environment
cp .env.example .env         # then fill in the "change-me" secrets:
#    JWT_SECRET, REFRESH_TOKEN_SECRET, SESSION_SECRET,
#    MASTER_ENCRYPTION_KEY (openssl rand -hex 32)
#    For pure local dev the .env.example defaults are fine otherwise.

# 5. Start infrastructure (postgres, redis, minio, opensearch)
pnpm docker:up
pnpm docker:up --wait        # wait for healthchecks

# 6. Create the Prisma client + apply migrations
pnpm db:generate
pnpm db:migrate              # first run: creates the initial migration
pnpm db:seed                 # contracted demo fixture (verified at the end)

# 7. Run everything (web, api, worker, realtime via Turbo)
pnpm dev

Open http://localhost:3000 and log in with the seeded account: demo@factory.local / Factory!2345.

Expected console output (abridged):

• web        ready on http://localhost:3000
• api        NestJS v10 — listening on http://localhost:4000 (health: /api/v1/health)
• worker     BullMQ worker — queues: search, files, webhooks, automation, email, ai (health: :4003)
• realtime   Yjs gateway — ws://localhost:4001 (health: :4002)

2.1 Docker Compose services (local)

Reference file: infrastructure/docker/docker-compose.yml. Services:

Service	Image	Ports (host→container)	Volume	Healthcheck	Purpose
postgres	postgres:16-alpine	5432→5432	pgdata:/var/lib/postgresql/data	pg_isready -U factory	System of record (db factory, user factory/factory)
redis	redis:7-alpine	6379→6379	redisdata:/data	redis-cli ping	Cache, queues (BullMQ), event bus (streams), presence, rate limits
minio	minio/minio	9000→9000 (S3 API), 9001→9001 (console)	miniodata:/data	mc ready	Object storage (bucket factory-uploads, created by init)
minio-init	minio/mc	—	—	—	One-shot: creates factory-uploads bucket
opensearch	opensearchproject/opensearch:2.x	9200→9200	osdata:/usr/share/opensearch/data	curl /_cluster/health	Search + vector index (single-node, security plugin disabled locally via DISABLE_SECURITY_PLUGIN=true; never in prod)
clamav (optional)	clamav/clamav	3310→3310	—	clamd --ping	Virus scanning for the file pipeline (enable with --profile files)

Optional dev containers (profile apps): api (4000), web (3000), worker (4003), realtime (4001) — used by CI and by developers who prefer containerized apps; local dev runs apps on the host for hot reload.

2.2 Port table (local development)

Service	Port	Protocol	Purpose
web	3000	HTTP	Next.js app (incl. /api/v1/health, /health/ready)
api	4000	HTTP	NestJS API (/api/v1/health, /api/v1/health/ready)
worker	4003	HTTP	Health only (/api/v1/health, /health/ready) — no business surface
realtime	4001	WS	Yjs gateway (auth via ?token=)
realtime	4002	HTTP	Health (/health, /health/ready) + Prometheus /metrics
postgres	5432	TCP	System of record (db factory, user factory/factory)
redis	6379	TCP	Cache, BullMQ queues, factory:events streams, presence
minio	9000 / 9001	HTTP	S3 API / console (bucket factory-uploads)
opensearch	9200	HTTP	Search + vector index
clamav (optional)	3310	TCP	Virus scanning (profile files)

Deviation (recorded per AGENTS.md rule 29, also in docs/tasks/phase-00-foundation.md TASK-0009): TASK-0009 item 4 names the realtime health routes "on 4001", but the gateway deliberately serves WebSocket and HTTP on separate ports — the health server binds 4002 (HEALTH_PORT) because ws's noServer upgrade model already owns 4001's upgrade path. All three routes exist (/health, /health/ready, /metrics); only the port differs from the task text.

Useful one-liners:

pnpm docker:up                      # infra only
pnpm docker:up --profile files      # + clamav
pnpm docker:logs -f postgres        # follow a service
pnpm docker:down                    # stop (volumes kept)
pnpm docker:down -v                 # stop AND delete data (nuclear)
docker compose -f infrastructure/docker/docker-compose.yml exec postgres \
  psql -U factory -d factory        # psql shell

3. Monorepo map

pnpm workspaces (apps/*, packages/*) + Turborepo for task orchestration.

3.1 Apps

App	Stack	One-line purpose
apps/web	Next.js 14 (App Router) + React 18, Tiptap, dnd-kit, TanStack Query, Zustand, Tailwind	The entire user interface: auth screens, workspace shell, block editor, all 9 database views, search, settings, public/share routes, and the BFF route handlers under app/api
apps/api	NestJS 10 (modular monolith)	REST API v1 + all domain modules (auth, workspace, page, block, database, relation, file, comment, search, notification, automation, AI, integration, api-keys, webhooks, share, templates, audit), the Redis event bus, guards, and OpenAPI generation
apps/worker	Node + BullMQ	Consumes all domain queues: search indexing, file pipeline (scan/metadata/thumbnails), webhook dispatch + retries, automation runs, email, AI jobs, import/export jobs, retention/purge jobs
apps/realtime	Node + ws + Yjs	WebSocket gateway: Yjs CRDT document rooms, presence/awareness, cursors, realtime comments & notification fan-out, per-connection rate limiting

3.2 Packages (11)

Package	One-line purpose	Consumed by
packages/ui	Design system: shadcn/ui components + theme tokens (CSS variables), icon set	web
packages/editor	Tiptap block editor: all block-type extensions, slash menu, context menus, DnD wiring, the fractional positions module, block tree ↔ Yjs codec	web
packages/database-engine	Pure domain engine: property types, filter/sort/group evaluator, view query, relations/rollups, the safe formula parser/AST evaluator, button action planner — zero I/O, shared client+server	web, api, worker
packages/auth	AuthN primitives: JWT create/verify, refresh-token rotation logic (hashed, family tracking), password hashing, session helpers, SSO (OIDC/SAML) claim mapping	api, realtime (token verify)
packages/permissions	AuthZ engine: RBAC + resource-level levels, inheritance walk, decision caching contract, the NestJS permission guard implementation	api (and worker for actor-context execution)
packages/collaboration	Client-side Yjs: provider bootstrap, awareness/presence types, offline queue (IndexedDB), reconnect/resume logic	web
packages/search	Search I/O: OpenSearch client, index document builders (permission snapshots), query translation (UI filter → OpenSearch DSL), hybrid (BM25 + kNN) fusion	api, worker
packages/ai	AI gateway: provider abstraction (OpenAI/Anthropic/Google/OSS/custom), unified chat/embed/tools API, RAG pipeline (chunker, retriever, citations), tool-call validation	api, worker
packages/storage	S3 client wrapper, presign helpers, file pipeline stage implementations (metadata, thumbnails, scan client), signed-URL service	api, worker
packages/shared	The contract package: Zod schemas for every DTO, error types (ApiError codes), event names/payloads, constants (block types, property types), API path map, id prefixes	everyone
packages/config	Shared tooling presets: tsconfig base (strict), ESLint flat config, Prettier config, Vitest/Jest shared setup	everyone (dev only)

Dependency direction (enforced by lint — see coding-standards §3): web → {ui, editor, database-engine, collaboration, search, ai, storage, shared}; api → {auth, permissions, database-engine, search, ai, storage, shared}; worker → {same as api, minus UI-only}; realtime → {auth, shared, collaboration(codec)}. shared and config depend on nothing internal. No package imports an app.

4. Development workflows

4.1 Dev servers

Command	What runs	Notes
pnpm dev	web + api + worker + realtime (Turbo persistent)	Hot reload everywhere
pnpm dev --filter=web	web only	Next.js dev, HMR on :3000
pnpm dev --filter=api	api only	NestJS --watch (ts-node), :4000
pnpm dev --filter=worker	worker only	Prints each processed job with queue + id
pnpm dev --filter=realtime	realtime only	ws :4001

Hot reload behavior:

• web: Next.js HMR; packages/ui|editor|... source changes hot-reload too (workspace deps are link: and the apps' build watches them).
• api: NestJS watch mode restarts on apps/api/src change; package changes require a restart (stop/start) — package code is compiled to dist in the monorepo build; for fast iteration run api with --preserveWatchOutput.
• worker/realtime: restart-on-change as api.

4.1a Source viewer (demo: see how the project is developed)

The demo deployment serves a read-only browser of this repository at https://notion.filps.com/source/ so visitors can read the task specs, roadmap, changelog and code. Links to it live in the topbar ("Source") and on the login page.

• Service: source-viewer.service (systemd) → node server.mjs on 127.0.0.1:8090 (loopback only; Caddy proxies /source* for notion.filps.com). The viewer code is deployed at .dsh-scratch/source-viewer/server.mjs on the demo host (scratch area, intentionally not part of this repo — it is demo-host tooling, not product code).
• Read-only by construction: the process only ever opens files for reading; unknown/unsafe paths 404; path traversal is resolved against the repo root with a realpath containment check.
• Secret handling: live env files (.env, .env.bak-*, infrastructure/docker/.env.docker, … — everything except .env.example) are visible with every value redacted server-side (blurred bars; the raw endpoint substitutes (redacted)). Variable names and comments are shown, but the secret values never leave the process, so "view source" reveals nothing. A token-pattern scrubber additionally strips high-entropy strings from the shown text. gitleaks-baseline.json (holds literal secret strings from scan history) and build/deps caches (node_modules, .next, .turbo, dist, build, .git) are never served. gitleaks.toml (the config) and .env.example (the template) are safe and stay visible.
• Ops: systemctl restart source-viewer · logs: journalctl -u source-viewer · Caddy route: the handle /source* block in the notion.filps.com server block of /etc/caddy/Caddyfile.

4.2 Seeding & fixtures

• pnpm db:seed — idempotent, and ends by running the contracted-fixture verification (prisma/verify.ts), failing on any broken invariant. Seeds: 2 verified users (demo@factory.local / admin@factory.local), the non-personal demo workspace (owned by the demo user), exactly 2 personal workspaces (slugs personal-<4 random chars>, never hardcoded), 2 teams, 6 pages ("Getting started" + 2 children + 3 record pages), 1 Tasks database (title, 2× select, date, person; 3 records each 1:1 with a page; 1 TABLE view), 1 comment on "Getting started" (by admin), and the demo user's favorite on "Getting started". In non-production it prints the demo credentials block + the personal-workspace slugs. Safe to re-run.
• prisma/seed-large.ts (added in Phase 3) — 100k-record database + 100k-block page for performance work: pnpm tsx prisma/seed-large.ts --records=100000.
• Test fixtures live in tests/integration/fixtures/ (JSON + generator scripts) and are separate from the dev seed.

4.3 Environment variables

All config comes from environment (.env locally; validated with Zod in apps/api/src/config/env.ts — the app refuses to boot with an invalid env). Full reference from .env.example:

Variable	Default (dev)	Description
NODE_ENV	development	development / test / production
PORT	4000	API listen port
WEB_URL	http://localhost:3000	Origin of the web app (CORS allow-list, cookie domain)
API_URL	http://localhost:4000	API origin (used by web BFF, webhooks docs)
REALTIME_URL	ws://localhost:4001	WebSocket gateway origin
PUBLIC_BASE_URL	http://localhost:3000	Base URL for absolute links (share URLs, emails)
DATABASE_URL	postgresql://factory:factory@localhost:5432/factory?schema=public	Postgres connection
REDIS_URL	redis://localhost:6379	Redis connection
JWT_SECRET	— (required)	Access-token signing key (HS256; ≥ 32 bytes)
JWT_EXPIRES_IN	15m	Access token TTL
REFRESH_TOKEN_SECRET	— (required)	Refresh-token signing (rotation integrity)
REFRESH_TOKEN_TTL_DAYS	30	Refresh token lifetime
SESSION_SECRET	— (required)	Session cookie encryption
BCRYPT_ROUNDS	12	Password hash cost
S3_ENDPOINT	http://localhost:9000	S3-compatible endpoint (MinIO dev)
S3_ACCESS_KEY / S3_SECRET_KEY	minioadmin	S3 credentials
S3_BUCKET	factory-uploads	Uploads bucket (private)
S3_REGION	us-east-1	S3 region
S3_FORCE_PATH_STYLE	true	Path-style URLs (required for MinIO)
SEARCH_URL	http://localhost:9200	OpenSearch endpoint
SEARCH_USERNAME / SEARCH_PASSWORD	(empty)	OpenSearch basic auth (prod: always set)
SEARCH_INDEX_PREFIX	factory	Index name prefix
AI_PROVIDER	openai	openai | anthropic | google | oss | custom
AI_BASE_URL	https://api.openai.com/v1	Provider base URL (OSS/custom OpenAI-compatible)
AI_API_KEY	(empty)	Provider key (leave empty in dev to disable AI → graceful 503)
AI_MODEL	gpt-4o	Default chat model
AI_EMBEDDING_MODEL	text-embedding-3-small	Embedding model (RAG)
SMTP_HOST / SMTP_PORT / SMTP_USER / SMTP_PASSWORD	localhost:587	Transactional email (dev: use Mailpit — see §7)
SMTP_FROM	AI Software Factory <no-reply@example.com>	Envelope + header sender
REALTIME_PORT	4001	WS gateway port
REALTIME_HEARTBEAT_MS	30000	Presence heartbeat interval
YJS_STORAGE_PREFIX	yjs	Redis key prefix for Yjs update logs
WEBHOOK_RETRY_MAX	8	Webhook delivery attempts
WEBHOOK_TIMEOUT_MS	10000	Per-attempt HTTP timeout
RATE_LIMIT_WINDOW_MS	60000	Sliding-window length
RATE_LIMIT_MAX	300	Default per-user API requests per window (auth endpoints have their own stricter tiers)
OTEL_ENABLED	false	OpenTelemetry export on/off (NFR-H5: off by default)
OTEL_EXPORTER_OTLP_ENDPOINT	http://localhost:4318	OTLP/HTTP collector
MASTER_ENCRYPTION_KEY	— (required in prod)	64-hex-char DEK for at-rest secret encryption (envelope)

4.4 Prisma workflow

Task	Command
Create/apply migration in dev (after editing schema.prisma)	pnpm db:migrate (prompts for name)
Regenerate client only (no DB)	pnpm db:generate
Apply committed migrations (CI/staging/prod)	pnpm db:deploy
Seed	pnpm db:seed
Browse data	pnpm db:studio (Prisma Studio on :5555)
Validate schema	pnpm prisma validate

Rules (from prisma/migrations/README.md + AGENTS.md):

1. Migrations are the only schema change path — never hand-edit SQL in a committed migration, never reset the prod DB.
2. Every schema change ships in the same task/PR as the feature that needs it.
3. Destructive changes (drop column/table, type narrowing) require review and are recorded in the task file; prefer two-step (add → migrate data → drop).
4. JSONB columns hold flexible payloads (block content, property values, view settings, formula ASTs) — their internal shape is versioned in packages/shared Zod schemas, not in the DB.

5. Testing commands

Full strategy in testing.md. Quick reference:

# Unit (Vitest in packages, Jest in apps/api) — co-located *.test.ts / *.spec.ts
pnpm test:unit                          # all workspaces
pnpm --filter @factory/database-engine test:unit
pnpm --filter api test:unit

# Integration (Testcontainers: Postgres, Redis, OpenSearch, MinIO)
pnpm test:integration                   # tests/integration (needs Docker)

# E2E (Playwright) — boots the full stack (docker infra + api/worker/realtime)
pnpm test:e2e                           # tests/e2e, chromium by default
pnpm test:e2e -- --project=firefox
pnpm test:e2e -- --grep "MVP flow 1"

# Performance (k6 / Artillery)
pnpm test:perf                          # runs the default load profile
pnpm k6 run tests/performance/api-load.js --env DATA_DIR=tests/performance/fixtures

CI runs the whole pyramid in order (lint → typecheck → unit → integration → build → security scan → e2e) — see the mapping in testing.md §8.

6. Package manager & build pipeline

• pnpm 9 with workspaces; strict-peer-deps on; no npm/yarn in CI (corepack enforces the version from packageManager).
• Turborepo orchestrates cross-workspace tasks (turbo.json):
  • build — dependsOn: ^build (builds dependencies first), caches outputs (.next/**, dist/**, build/**);
  • lint / typecheck / test / test:unit — depend on ^build (packages must be built for apps to typecheck against their dist);
  • dev — persistent, no cache (runs all four apps concurrently);
  • test:integration — no cache (always fresh containers).
  • Cache is content-hashed per task; pnpm build on a clean clone is slow (full build), incremental after.
• Monorepo conventions: apps expose no public API; packages export from src/index.ts in dev (turborepo builds to dist for prod/CI).

7. Debugging

What	How
API devtools	API exposes Swagger UI at http://localhost:4000/docs (OpenAPI from the same generator as the public spec). Enable request logging: DEBUG=factory:* pnpm dev --filter=api (pino debug level, structured JSON). Correlate with the web by copying x-request-id from response headers into the log grep.
Database	pnpm db:studio (GUI); docker compose exec postgres psql -U factory -d factory for SQL. Watch a query live: EXPLAIN (ANALYZE, BUFFERS) <sql>; slow queries: enable log_min_duration_statement=200 on the dev postgres via docker compose exec postgres psql -c "ALTER SYSTEM SET log_min_duration_statement='200ms'; SELECT pg_reload_conf();".
Redis	docker compose exec redis redis-cli → INFO, MONITOR (dev only), XINFO GROUPS <stream> / XLEN <stream> for the event bus; BullMQ queues: ZRANGE bull:search:wait 0 -1 WITHSCORES, LLEN bull:webhooks:failed.
OpenSearch	curl -s localhost:9200/factory_pages/_search -H 'content-type: application/json' -d '{"query":{"match_all":{}},"size":1}'; _cat/indices?v for sizes; _cat/shards for health; mapping: _mapping.
MinIO console	http://localhost:9001 (minioadmin/minioadmin) — browse factory-uploads, inspect object metadata/versions. docker compose exec minio mc alias set local http://minio:9000 minioadmin minioadmin && mc ls local/factory-uploads from the CLI.
Realtime WS tester	wscat -c ws://localhost:4001/ws?token=<JWT> (get a JWT from POST /auth/login). Send {"type":"join","doc":"page:<id>"}; watch presence and update frames. Or use the in-app devtools: localStorage.debug = 'factory:ws' in the web console logs every frame.
Worker jobs	pnpm dev --filter=worker logs each job (queue, jobId, status, ms). Fail a job on purpose: set DEBUG=factory:bull and throw in the processor; failed jobs land in bull:<queue>:failed — reprocess with IORANGE or the admin endpoint POST /admin/queues/:name/reprocess.
Web	Next.js dev overlay; Zustand devtools (open at web → DevTools → Redux tab via the @factory/ui devtools mount); TanStack Devtools for query cache; React Profiler for render spikes.
Email	Dev: Mailpit (docker compose --profile mailpit up mailpit) → http://localhost:8025 (all SMTP captured, no real mail).
AI calls	Gateway logs every request (provider, model, tokens, latency) at info; set AI_DEBUG=1 to dump full prompts/responses (dev only — never in prod, NFR-P2).

8. Git workflow & commits

• Branches: main (always releasable), develop (integration target), feature/<task-id>-<slug>, fix/<slug>, refactor/<slug>. Branch from develop, PR into develop; develop → main for releases (tag vX.Y.Z).

• Commits (Conventional Commits, scope = package or app):

  feat(editor): add draggable blocks
  feat(database): add board view
  fix(auth): rotate refresh tokens on reuse
  test(database): add filter golden datasets
  docs(api): document page endpoints
  chore(infra): bump postgres image to 16.3
  

• PR requirements (template in .github/PULL_REQUEST_TEMPLATE.md): linked task (docs/tasks/phase-XX-…md), description, screenshots for UI, migration note if schema changed, checklist: tests written + passing, typecheck, lint, docs updated, DoD items from AGENTS.md ticked.

• CI contract (every PR must pass, in order): install → lint → typecheck → unit → integration → build → security scan (SCA + gitleaks) → E2E. Only green PRs merge; main additionally runs the perf suite.

9. Common issues

Symptom	Cause / fix
pnpm dev starts but web shows "Failed to fetch"	API not up yet (it boots slower). Check pnpm docker:ps; the web BFF retries 3×. If persisting: verify API_URL matches the port.
Port already in use (3000/4000/4001/5432…)	lsof -i :4000 → kill the squatter, or override in .env (PORT=4010 etc.) and in the web env.
Prisma: P1001: Can't reach database	Infra not up: pnpm docker:up --wait. Or .env DATABASE_URL host mismatch (use localhost on host, postgres inside containers).
OpenSearch bootstrap failure in compose logs (max_map_count)	Linux kernel setting: sudo sysctl -w vm.max_map_count=262144 (add to /etc/sysctl.d/ permanently).
MinIO: AccessDenied on upload	Bucket not created yet — run the minio-init one-shot (automatic on docker:up); or check S3_ACCESS_KEY/S3_SECRET_KEY match compose env.
Login works but WebSocket fails (401)	WS auth uses the access JWT; check the token isn't expired (15 min) — the client should refresh before joining; dev: set JWT_EXPIRES_IN=2h in .env to stop this while debugging.
Worker processes no jobs	Consumer group missing or worker not connected to the same Redis as the API: verify both read the same REDIS_URL. redis-cli → XINFO GROUPS factory:events should show lag decreasing.
E2E flaky "realtime" tests	Two browsers must share one API instance with WS enabled; run pnpm test:e2e (which starts the full stack) rather than against a manually started dev server with different env.
corepack says pnpm version mismatch	corepack enable && corepack prepare pnpm@9.15.0 --activate.
TS errors only in api after editing a package	Package dist is stale: run pnpm build --filter=<pkg> then restart api (watch mode doesn't rebuild linked packages).
Seed says "upsert failed: unique constraint invitations"	You re-ran seed after changing invitation data; it's idempotent by design — if it truly fails, pnpm db:migrate reset (dev only!) or delete the demo workspace rows.
Search returns nothing after editing a page	Index lag (≤ 30 s by design) or the indexer queue is down: `docker compose logs worker
High memory in dev (Chrome + Next + 4 nodes)	Stop worker when not testing pipelines; run opensearch with OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g.

10. Where things live (cheat sheet)

Need	Path
Task spec for the current phase	docs/tasks/phase-XX-*.md
Requirement IDs	../product/product-requirements.md
API endpoint docs	docs/api/ (generated from packages/shared + Nest decorators)
DB schema (source of truth)	prisma/schema.prisma + ../database/database-architecture.md
Design tokens/components	packages/ui + ../ui/design-system.md
Security rules	../security/security-architecture.md
Env reference	.env.example (+ §4.3 above)
Compose (local) / K8s / nginx / terraform	infrastructure/{docker,kubernetes,nginx,terraform} + deployment.md
Test suites	tests/{integration,e2e,performance} + testing.md