Testing Strategy — AI Software Factory

The testing contract for the whole monorepo. Strategy: a pyramid — many fast unit tests around the pure engines, a meaningful integration layer against real infrastructure, and a small number of end-to-end flows that prove the product works as a user experiences it. Coverage and engine mandates are NFRs (NFR-M1…M4 in non-functional-requirements.md §9). Related: development-guide.md §5 (commands), ../product/functional-requirements.md (what is being tested), AGENTS.md Definition of Done (every feature needs unit + integration + e2e where applicable).

---

1. Strategy pyramid

            ┌──────────────┐
            │     E2E      │  Playwright, tests/e2e
            │  (13 MVP     │  full stack, 3 browsers + mobile viewport
            │   flows +    │  nightly + every PR (chromium), full matrix
            │   regressions)│ on release branches
           ┌───────────────────────┐
           │     Integration       │  Jest + Testcontainers, tests/integration
           │ API + real PG/Redis/  │  permission matrix, cross-tenant, audit
           │ OpenSearch/MinIO/WS   │  completeness, events, pipelines
         ┌───────────────────────────────┐
         │            Unit               │  Vitest (packages) + Jest (apps/api)
         │ engines, services, codecs,    │  co-located *.test.ts / *.spec.ts
         │ schemas, guards (mocked deps) │  run on every commit, < 2 min
         └───────────────────────────────┘
        (below the pyramid: typecheck, lint, SCA, secret scan — CI gates)

Allocation rule: the harder something is to observe, the lower in the pyramid it must be tested. A formula bug is caught in a 5 ms unit test, not a 30 s e2e test. E2E proves wiring, unit tests prove logic.

Layer	Tool	Location	Runtime target	Fails CI when
Unit (packages)	Vitest + @testing-library/react (ui/editor)	packages/*/**/*.test.ts(x), co-located	< 90 s all packages	any failure or coverage below NFR-M1/M2 thresholds
Unit (api)	Jest + supertest (controller level, mocked repos)	apps/api/**/*.spec.ts, co-located	< 60 s	any failure or coverage < 70 % (apps)
Integration	Jest + Testcontainers (postgres, redis, opensearch, minio) + real Nest app	tests/integration/**	< 10 min	any failure
E2E	Playwright	tests/e2e/**	< 15 min (chromium)	any failure (chromium); other browsers: release gate
Performance	k6 (HTTP) + Artillery (WS)	tests/performance/**	< 20 min	pass-criteria breach (nightly + Phase 15 gate)
Security	SCA (pnpm audit + OSV), gitleaks, ZAP baseline, axe (in e2e)	.github/workflows + tests/e2e/a11y	in CI	critical SCA / secret / axe critical/serious

2. Unit testing

2.1 Tooling & config

• packages → Vitest: config extends packages/config/vitest.shared.ts (environment: 'node', globals: true, coverage.provider: 'v8', coverageThreshold per NFR-M1). React-testing packages (ui, editor, collaboration) set environment: 'jsdom' + @testing-library/react.
• apps/api → Jest: apps/api/jest.config.ts (testEnvironment node, moduleNameMapper for @factory/* → packages/*/src/index.ts so the api unit-tests the packages' source without a build step).
• apps/worker, apps/realtime → Jest (same pattern as api).
• apps/web → Vitest for component-level unit tests (jsdom) — the bulk of web verification is e2e; web unit tests target hooks, stores, and pure UI logic (not pixels).

Run: pnpm test:unit (Turbo, all workspaces). Single file: pnpm --filter @factory/database-engine exec vitest run formula.

2.2 Where tests live

Co-located, mirroring source: src/formula/evaluator.ts → src/formula/evaluator.test.ts. Shared fixtures per package under src/**/*.fixtures.ts or test/fixtures/ inside the package. No test file may import from another package's test/ directory (fixtures are re-used via packages/shared/test-utils when shared).

2.3 Mocking rules (the contract)

Rule	Statement
U1	Unit tests of services mock repository interfaces, never Prisma directly. Every domain service depends on a *Repository interface (implemented in apps/api/src/common/repositories/*Prisma). In a unit test the service receives a stub/mock of that interface (Vitest vi.fn() / Jest jest.fn()). No PrismaClient import appears in any unit test file (enforced by eslint no-restricted-imports in *.test.ts files).
U2	Real Postgres is the job of integration tests (Testcontainers). If a unit test needs "the database", that is a design smell — refactor to a repository interface.
U3	Engines are pure: database-engine (filters, formulas, positions, rollups, view query) and permissions (decision logic) have no mocks at all — they take plain data and return plain data. Their tests are golden/property tests on real inputs.
U4	External I/O is mocked at the client boundary: OpenSearch client, S3 client, SMTP, AI provider, HTTP for webhooks. Mock the interface (SearchIndexer, FileStorage, MailSender, AiGateway, HttpClient), not fetch/axios.
U5	Time is injected (now: () => Date parameter or a test clock via vi.useFakeTimers) — no test may depend on wall-clock timing except where the SUT is a timer.
U6	No network in unit tests, ever (Vitest server.deps off + a CI guard that unit jobs run with networking disabled).
U7	Async: no setTimeout-based sleeps in assertions; use vi.waitFor/Jest waitFor on state.
U8	Error paths: every service method has a test for its typed ApiError cases (NOT_FOUND, VALIDATION_ERROR, PERMISSION_DENIED, CONFLICT) — the error code is part of the contract.

2.4 Coverage gates (NFR-M1/M2, enforced in CI)

Scope	Line	Branch	Function
packages/database-engine, packages/permissions, packages/editor (positions, codec), packages/shared	90 %	85 %	90 %
apps/api	70 %	60 %	70 %
apps/web (components+hooks)	60 %	50 %	60 %
apps/worker, apps/realtime	60 %	50 %	60 %

3. Engine test mandates

These are named suites with their own acceptance criteria; they are the load-bearing logic of the product and get the strictest treatment.

3.1 Formula engine (packages/database-engine/formula)

• Function coverage = 100 % (ROADMAP Phase 6 exit): every function in if, and, or, not, sum, average, min, max, dateAdd, dateBetween, now, concat, replace, contains, length + the prop() accessor has:
  • happy-path cases (typed: number, string, boolean, date, array, null),
  • edge cases (empty lists, mixed types → type error, unicode strings, date timezones, now() determinism pinned via injected clock),
  • error cases (wrong arity, unknown function, unknown property, division by zero → #DIV/0!-style error value, not an exception).
• Parser: table-driven token/AST tests (every literal kind, operator precedence matrix: 1 + 2 * 3, and(a, or(b, c)) == 1, ternary nesting).
• Fuzzing (nightly job + 1 000-expr smoke on PR): 10 000 random expressions (grammar-based generator: valid expressions + deliberately malformed: unbalanced parens, truncated strings, 10k-deep nesting, 1 MB input). Assertions: 0 crashes, 0 unhandled rejections, 0 evaluations > 100 ms (budget enforced in the evaluator), and for valid expressions the result is either a value or a typed error — never a hang. Seed corpus is committed (formula.fuzz.seeds.json); new crashes are added to it.
• Zero eval: a static grep in CI fails the build if eval(, new Function(, or vm.runIn* appears anywhere in packages/database-engine (AGENTS.md rule 18).

3.2 Permission engine (packages/permissions)

• Decision matrix suite: the full cartesian sample — roles {Owner, Admin, Member, Guest} × resource levels {None, View, Comment, Edit, Full Access} × resource kinds {workspace, teamspace, page (top), page (nested ×3), database, record, file} × grant scenarios {no explicit grant, explicit grant on resource, explicit grant on ancestor, explicit None on ancestor (deny-override), team grant, conflicting grants (max wins)} — every cell has an expected decision; the suite renders the matrix as a table in the test report. 100 % pass is a CI gate (NFR-SE2).
• Property-based tests (fast-check): random page trees (depth ≤ 8, random grants) → invariants: (1) access is monotone down the tree (a grant on an ancestor is visible to all descendants unless denied), (2) effective(grantee, resource) equals the hand-computed walk, (3) removing a grant never increases access, (4) workspace role floor holds (Guest can never exceed its grants). 10 000 generated cases.
• Guard integration (unit, api): PermissionGuard with a mocked engine — asserts it calls engine.can(actor, resource, level), throws PERMISSION_DENIED (403) on false, and writes the audit hook.

3.3 Fractional positions (packages/editor/positions)

• Property-based tests (fast-check) for the LexoRank-style alphabet:
  • mid(a, b) is strictly between a and b (string order) for random valid pairs,
  • mid(a, b) never produces a string longer than the cap (512 chars) for any input pair where a valid midpoint exists,
  • rebalance: when mid(a, b) hits the cap (no representable midpoint), the algorithm returns a special marker and the re-batcher assigns fresh positions to the affected sibling run — property: after rebalance, all positions are distinct, sorted consistently, and the relative order of the whole run is preserved,
  • order isomorphism: applying N random inserts (each mid of neighbors) to a sorted list yields positions that sort back to the insertion-order list,
  • determinism: same inputs → same outputs (no clock/randomness).
• Examples in tests (readable, committed): mid("a0", "a1") === "a0V" (midpoint between adjacent base positions); mid("a0V", "a1") === "a0Vg"; inserting 30 items between "a0" and "a1" exercises the short-alphabet tier; a 60-deep-insertion chain triggers a rebalance of that sibling group (asserted).

3.4 Filter / sort / group (packages/database-engine/query)

• Golden datasets: 6 committed fixture databases (small, hand-verified: filters.golden.json, sort.golden.json, group.golden.json) — each with records (nulls, unicode, dates at DST boundaries, empty selects, nested multi-select) and cases: single condition, AND/OR nesting (depth 4), NOT, me/today tokens, every operator per property type, multi-key sort with nulls-first/last, multi-level group with counts. Expected outputs are hand-computed and committed — a change to expected output requires review (it is the spec).
• Property-based: random record sets + random valid filter trees → evaluate with the engine AND with an independent reference implementation (a simple records.filter(jsPredicate) written directly from the tree) → results identical (10 000 cases). This catches drift between the engine and the documented semantics.
• Client/server parity: the same golden files run in both the Vitest (package) suite and an api-level Jest suite that calls the query engine via the repository — outputs must match byte-for-byte.

3.5 Yjs ↔ block-tree codec (packages/editor/codec + packages/collaboration)

• Round-trip property: random block tree → Yjs doc → block tree is identity (content, positions, properties, nesting); 10 000 trees.
• Convergence: two random edit scripts applied to two docs, synced → both derive the same final tree (CRDT property, 1 000 pairs).
• Update-log resume: apply edits, persist at a random point, replay the remaining updates onto a fresh doc → identical final tree.

3.6 Other mandated unit suites

• Zod schemas (packages/shared): every schema round-trips its own fixture (valid) and rejects a documented invalid set (the "contract" of the API).
• Auth (packages/auth): refresh rotation state machine (issue → rotate → replay → family revoked) as a finite-state test; JWT claims exactness.
• Webhook signing (packages/shared + worker): signature vector test (fixed secret/body → expected v1= hex) — interop-stable, do not change without bumping the scheme.
• Import parsers (markdown/html/csv/notion-zip): fixture files → expected block trees (golden), including hostile inputs (deeply nested HTML, 50k-row CSV, malformed Notion manifest → typed errors, no crash).

4. Integration testing

4.1 Infrastructure: Testcontainers

tests/integration/testcontainers.ts starts, per test module (reuse via a module-scoped setup):

Container	Image	Used for
Postgres	postgres:16-alpine	every test (migrations applied via prisma migrate deploy on a temp db)
Redis	redis:7-alpine	event bus, rate limits, permission cache, queues
OpenSearch	opensearchproject/opensearch:2.x (security disabled)	search indexer + query tests
MinIO	minio/minio	file pipeline tests
(ClamAV — Phase 9)	clamav/clamav	scan-stage tests (EICAR sample)

The Nest app is booted for real (Test.createTestingModule with the real modules, Testcontainer URLs injected via env) — guards, pipes, interceptors, event bus, and repositories all live in this layer. The web app is not booted (that's e2e).

4.2 Fixtures & harness

• tests/integration/helpers/api.ts — API harness: an authenticated request() wrapper (creates a real user via the API, logs in, returns scoped as(user) clients for multiple users; asApiKey(scope), asWebhookReceiver() spin up a local HTTP receiver for outbound tests).
• tests/integration/helpers/db.ts — clean-slate helpers: freshWorkspace(), seedDbWith({records, properties}) (uses the engine's golden generators), waitForEvent(stream, predicate, timeout).
• All fixtures are generated inside the test (unique slugs per run) — no shared mutable state between test files (parallel-safe; Jest maxWorkers=2 while containers are shared).
• Events: tests that assert side effects (search indexed, webhook fired, automation ran) use waitForEvent/waitFor with a 10 s budget — never fixed sleeps.

4.3 Mandatory integration suites

Suite	Proves (maps to)
auth/	register→verify→login→refresh-rotation→replay-revocation, password reset, sessions, rate limits (NFR-SE5), audit events (PR-001…010)
tenancy/	cross-tenant matrix: 2 workspaces × 50 random (actor, resource) request pairs → 0 leaks (NFR-P1, CI gate); personal workspace auto-creation; invitation lifecycle; last-owner protection
permissions/	the decision matrix executed against the live API (every cell: allowed ↔ 2xx, denied ↔ 403 + audit row) (NFR-SE2)
pages/	CRUD, nesting, fractional order across concurrent moves (2 clients interleaving inserts — final order consistent), trash/restore/purge cascade, versions, public share link (password/expiry)
databases/	records-as-pages (1:1 invariant), 100-record batch create, query endpoint = golden outputs, view CRUD, property type validation (22 types × valid/invalid value), relation link/unlink both sides + cardinality enforcement, rollup recompute on record.updated event (event-driven, no poll), formula recompute on dependency change, button action execution + audit
files/	full pipeline happy path (upload→scan→metadata→thumb→READY), EICAR → VIRUS_DETECTED + not servable (NFR-SE6), signed-URL expiry, mime-mismatch rejection, 50 MB multipart, versioning, per-file permission override
search/	event→index lag < 30 s, full-text/prefix/fuzzy, permission-filtered results (grant revoked → doc disappears from results within 30 s), reindex job, 503 graceful degradation (NFR-R3)
comments/	threads on page/block/record/selection, resolve/reopen, reactions dedup, mention → notification + mention_notifications read state
forms/	public unauthenticated submission → record + form.submitted event, validation rejection (server-side), closed form → 410, rate limit 10/15 min/IP
automations/	record-created → status change E2E-ish (trigger→condition→action), retry/backoff (force 3 failures → succeeds on 4th), dead-letter after exhaustion, run history rows + logs, paused automation ignored, schedule trigger with fake clock
webhooks/	signature correctness (recomputed in test), retry with backoff (flaky receiver: fail 2× then 200), delivery log rows, SSRF rejection (private IP URL), event fan-out to multiple subscriptions
api-keys/	key create (raw shown once) → hashed at rest (DB assert), scope enforcement (READ key → 403 on write), revocation, per-key rate limit
audit/	run the audit scenario script → assert all mandatory events present with actor/ip/ua/resource fields (NFR-SE7)
retention/ (Phase 14)	policy set → job run → purged/kept per scope; HOLD blocks purge
realtime/	two WS clients join a doc room → edit converges; presence heartbeat + stale prune (fake time); resume after reconnect (update log); permission: guest with VIEW can join read-only (updates rejected)

4.4 File-upload fixture conventions (Phase 9, TASK-0105)

The files security suite has four non-obvious fixture rules:

• Sparse 500 MB upload — tests/fixtures/files/make-sparse.ts builds a 500,000,000-byte file by writing one 4 KB marker block then truncate-ing to length (sparse: ~one block on disk, zero disk cost on CI runners). The integration suite streams it to the real presigned PUT and asserts the object's exact size end-to-end (the ROADMAP "500 MB multipart" exit criterion). Never commit the sparse file — it is generated on the fly.
• EICAR — the standard 69-byte anti-virus test string (a known, non-malicious marker, safe to ship in a repo). The integration pipeline spec (TASK-0102) and the security suite (TASK-0105, S5) drive runFilesPipeline in-process with env.CLAMD_SOCKET pointed at the host ClamAV unix socket (zCONTSCAN) — the worker's production scan path. (The TCP zINSTREAM on :3310 is broken in the pinned ClamAV image — UNKNOWN COMMAND / size-limit — so the socket path is the one under test. The virus finalize writes scanReport to the top-level File column, and metadata.pipeline.error = "virus_detected".)
• ClamAV in CI — the CI e2e profile runs the ClamAV container (unix socket) + MinIO; the e2e "Blocked by virus scan" card (TASK-0103) depends on the real worker scanning via that socket. Integration assertions that need the scanner are gated on the socket existing (skip, not fail, when absent — e.g. a laptop without the ClamAV container).
• Report artifact + budget — the security suite writes tests/e2e/reports/files-security.json (scenario → pass/fail + timing) and enforces the 15-minute suite budget with a guard test (the 500 MB upload is included in the budget — a CI-regression canary). CI uploads it as an artifact; any failure blocks the phase (TASK-0105 exit criterion).

4.5 Local dev-stack relaunch gotchas (Phase 9, recorded 2026-09-30)

If the local pnpm dev (turbo) stack is killed and relaunched app-by-app (rather than via turbo run dev), four env/infra traps silently break the e2e suite — none show up in the API's own logs:

• CORS — the API's strict CORS whitelist is CORS_ORIGINS (default http://localhost:3000, apps/api/src/config/env.ts). The web dev server in this repo is pinned to :3100 (next dev -p 3100, because :3000 is taken by a different local app). If the API is relaunched without CORS_ORIGINS=http://localhost:3100,..., the browser page's cross-origin calls to :4000 fail preflight (no Access-Control-Allow-Origin) — so e2e tests that wait for a rendered file-card-*/preview time out while API-only assertions still pass (the login fixture logs in via node fetch, bypassing browser CORS). Symptom: "timed out waiting for file-card- to be visible" on every render test, yet the worker log shows file finalized READY.
• Realtime gateway (apps/realtime, ws :4001 / health :4002) — it must be running or page-live updates (file status, presence) never reach the browser. It reads PORT for its ws port (default 4001) — so when relaunching app-by-app with a shared env sourced from the root .env (which sets PORT=4000 for the API), unset PORT first or realtime crashes with EADDRINUSE :::4000.
• Host deps are processes, not containers (on this dev host) — MinIO is a host process (/home/ubuntu/.local/bin/minio server /home/ubuntu/factory/minio-data) and ClamAV is a host clamd on /run/clamav/clamd.ctl (the worker's CLAMD_SOCKET), not the factory-clamav container (whose clamd listens on /tmp/clamd.sock). If either was SIGTERM'd, restart them as before: MinIO with MINIO_ROOT_USER/PASSWORD=minioadmin; clamd (1.5.x has no --localsocket flag — use -c <cfg> with LocalSocket /run/clamav/clamd.ctl) as the clamav user. The integration suite gates on MinIO/search (skip, not fail) and the EICAR scan needs the host clamd socket live.
• Disk → OpenSearch flood-stage — the S1 500 MB upload lands as real bytes in the MinIO factory-files bucket; repeated runs fill the root disk and OpenSearch sets factory-content-v1 to read_only_allow_delete (bulk writes → 429, silently — see the TASK-0105 recorded note). Purge the stale files/**/big.bin objects and clear the block before re-running the search-presence assertions.

5. E2E testing (Playwright)

5.1 Project config (tests/e2e/playwright.config.ts)

Project	Browser / viewport	Runs
chromium	Chrome, 1440×900	every PR (CI) + nightly
firefox	1440×900	nightly + release branch
webkit	1440×900	nightly + release branch
mobile	chromium, 390×844 (touch)	nightly + release branch
a11y	chromium, 1440×900, axe-core injected	every PR (MVP flows only)

Common config: testDir: 'tests/e2e/specs', workers: 4, retries: 1 (CI only), timeout: 30 s, expect.timeout: 5 s, trace: 'retain-on-failure', screenshots: 'only-on-failure', video: 'retain-on-failure'.

Stack boot (tests/e2e/global-setup.ts): docker infra up (idempotent), prisma migrate deploy + a dedicated e2e seed (small: 2 workspaces, 3 users, 1 database with 20 records, 1 public page, 1 form), api + worker + realtime started as child processes (or containers) with NODE_ENV=test, OpenSearch pre-indexed. Teardown kills the processes. The e2e suite never touches the dev seed.

5.2 Auth via storageState

• tests/e2e/helpers/auth.setup.ts (Playwright setup project, runs first): performs real registration + email verification (verification token read from the Mailpit API in dev / from the DB in CI) + login for the four fixture users: owner, admin, member, guest; plus one API-key context; writes tests/e2e/state/<role>.json (storageState: cookies + localStorage tokens).
• Feature specs consume via test.use({ storageState: 'state/member.json' }).
• Role-specific specs (permissions) run the same flow under two states and assert the divergence (e.g. member sees the page, guest gets the 403 screen) — this is the e2e face of the permission matrix.

5.3 The 13 MVP flows

These 13 specs are the MVP acceptance set (each ≈ one user journey; all must pass on chromium for the MVP release; AGENTS.md DoD requires an E2E test per feature, so feature specs are added on top of these):

#	Flow (spec file)	Proves (PR refs)
1	01-register-verify-login.spec.ts — register → verify email (token from outbox) → land in personal workspace → login again → session appears in security settings	PR-001…005, 013
2	02-workspace-invite.spec.ts — owner creates workspace → invites member's email → member registers/accepts → sees workspace + role; invitation revoke path	PR-014…018
3	03-page-tree-trash.spec.ts — create page → nest child → rename → favorite → delete → restore (position preserved) → purge (owner)	PR-025…028, 030
4	04-editor-blocks-dnd.spec.ts — type text → slash menu: heading, todo, quote, callout, divider, code → drag-reorder 2 blocks → nest into toggle → multi-block drag → undo/redo → reload → order persists	PR-034…036, 045, 050, 051, 043
5	05-editor-menus-keyboard.spec.ts — right-click block menu (turn into, duplicate, color), page menu (duplicate, move), keyboard: ⌘B/I/U, ⌘⇧1/2/3, ⌘Z/⇧⌘Z, ⌘A ladder, ⌘K palette navigation	PR-041, 048, 049
6	06-editor-paste-external.spec.ts — copy markdown from an external textarea (clipboard API) → paste → blocks created (heading/list/todo/table); copy block → paste into another page	PR-043
7	07-database-table-crud.spec.ts — slash /database → add properties (number, select, date, checkbox) → add 3 records inline → edit cells (commit/revert) → archive → duplicate	PR-057…061, 062, 074
8	08-database-board.spec.ts — add Board view → columns from Status → create card in "To Do" → drag to "Done" (property updated, row in table reflects) → filter on board	PR-064, 053, 071
9	09-database-filter-sort-group.spec.ts — table: nested filter (OR inside AND) → row set matches expected ids → 2-key sort (nulls) → group by select → group counts correct	PR-071…073
10	10-relations.spec.ts — create Tasks db → add Relation property (1:N, bidirectional) → link 2 tasks to a project from both sides → reverse list updates → unlink	PR-075, 076
11	11-search.spec.ts — edit page title/content → wait ≤ 30 s → global search finds it (highlight) → type filter → guest (no grant) does not see the page in results	PR-098…104
12	12-files.spec.ts — upload image via OS-drag simulation (input fallback) → pipeline to READY (progress UI) → insert into block → resize → download (signed URL) → file trash	PR-105…110
13	13-comments-mentions.spec.ts — comment on a block → @mention second user → second user's inbox shows notification → reply → resolve → react	PR-091…096

Additional standing e2e specs: public-share.spec.ts (public URL, password, expiry, duplicate), offline.spec.ts (Phase 7: edit offline → reconnect → sync), realtime-two-browsers.spec.ts (Phase 7), audit-visual.spec.ts (Phase 14), a11y/ (axe over the 13 flows: 0 critical/serious).

5.4 AGENTS.md DoD → test matrix

Every feature task's Definition of Done maps to tests as follows (checked in the task file):

DoD item	Where it's proven
Backend + frontend implemented	unit (service) + feature e2e
Database migration	integration (migrate deploy + schema assertion)
API validation	unit (schema) + integration (400 with field errors)
Authorization	integration permission matrix cell + e2e role pair
Error handling	unit (ApiError cases) + e2e (error states)
Loading / empty state	e2e (expect skeleton → content; empty-state text)
Mobile behavior	mobile project spot-checks (≥ 1 spec per phase touching layout)
Accessibility	a11y project (axe) + manual checklist for new components
Unit / integration / E2E tests	this document's gates
Documentation / logging / audit	docs CI check; integration (log field assertions); audit suite

6. Performance testing

6.1 Scripts (tests/performance/)

Script	Tool	Scenario
api-load.js	k6	80/20 read/write mix over the recorded endpoint set (pages, db query, search, comments); ramp 0 → 1 000 rps over 5 min, hold 30 min
search-latency.js	k6	1 000 queries against the 1M-doc fixture index; per-analyzer variants (full/prefix/fuzzy)
db-query.js	k6	POST /databases/:id/query on the 100k-record fixture: paged reads, filtered reads, 100-record batch writes (NFR-11)
upload.js	k6	50 MB multipart × 20 VUs (NFR-9)
ws-load.js	Artillery	10 000 WS connections across 100 rooms (100/room); 10 % of clients edit at 2 updates/s; 30 min (NFR-S2); measures p95 apply latency + connection stability
realtime-latency.js	Artillery + client probe	2-peer per room, 500 edits, latency A→B (NFR-6)
event-bus.js	k6 (producer) + metrics	5 000 events/s for 5 min → consumer lag assertion (NFR-S4)

Fixtures: tests/performance/fixtures/ — generator scripts produce the 100k-record database, 100k-block page, and 1M-doc OpenSearch index (same generators as the integration fixtures, larger scale).

6.2 Load profiles & pass criteria

Profile	When	Pass criteria (from the NFR doc)
smoke (2 min)	every PR (api only, 50 VUs, 2 min)	error rate < 0.5 %, p95 < 2× budget (soft gate, advisory)
standard (35 min)	nightly	NFR-5 (p95 < 300 ms @ 1k rps), NFR-4 (search p95 < 500 ms), NFR-9 (upload), error < 0.1 %
scale (60 min)	release branch + Phase 15 gate	NFR-S2 (10k WS, p95 < 200 ms, pod mem < 2 GB), NFR-S3, NFR-S4, NFR-11
soak (4 h)	Phase 15 gate + quarterly	no memory growth > 10 % over the window (leak check), no connection drops, lag stable

Reports: k6/Artillery JSON + HTML to artifacts/perf/ (CI upload); a comparison against the previous run is posted to the PR (regression > 10 % on any target metric = failing check on the scale profile).

7. Security testing

Check	Tool	Gate
Dependency SCA	pnpm audit --audit-level=critical + OSV API on lockfile	blocking (critical)
Secrets in repo/history	gitleaks + trufflehog	blocking
Web attack baseline	ZAP baseline scan against staging (auth flow scripted)	blocking on High+ findings
Payload suite	tests/e2e/security/: SQLi (every string input), XSS (block content, comments, titles, share tokens), path traversal (file names), CSRF (state change without token), IDOR (cross-workspace ids)	blocking
Permission matrix	integration permissions/ suite (NFR-SE2)	blocking
Accessibility	axe-core in the a11y project (0 critical/serious)	blocking on MVP flows
ASVS L2	manual + tool-assisted audit (Phase 15)	release gate

8. CI pipeline mapping

Repo: GitHub Actions (.github/workflows/ci.yml). Jobs run in a fixed order (dag), on a ubuntu-latest runner with Docker (for Testcontainers + e2e stack):

install ──► lint ──► typecheck ──► unit ──► integration ──► build ──► security ──► e2e ──► (merge)
                │            │           │            │              │           │           │
                │            │           │            │              │           │        perf-smoke
                │            │           │            │              │           └──► (main/nightly: perf-standard)
                └── shared cache (pnpm store, turbo cache)

Stage	Runs	Fails PR on
install	corepack enable, pnpm install --frozen-lockfile	any
lint	eslint (all workspaces, incl. import-boundary + i18n + no-eval + no-secrets rules) + prettier check	any
typecheck	turbo run typecheck (strict)	any
unit	turbo run test:unit with coverage → coverage gates (NFR-M1/M2)	any failure / below threshold
integration	pnpm test:integration (Testcontainers, workers=2)	any
build	turbo run build (all apps + packages)	any
security	SCA + gitleaks + (on staging deploy) ZAP baseline	critical SCA / secret / High ZAP
e2e	pnpm test:e2e chromium + a11y project, full stack from global-setup	any chromium/a11y failure
perf-smoke	tests/performance smoke profile (advisory)	— (comment only)
nightly (schedule)	e2e full browser matrix (firefox/webkit/mobile) + perf-standard + formula fuzz (10k)	posts report; blocks release branch promotion
on main (post-merge)	release build (docker images tagged sha-<short>), deploy to staging, SLO dashboard refresh	deploy failure pages on-call
release branch	perf-scale + soak + DR drill checklist (quarterly)	Phase-gate (Phase 15 exit)

Artifact retention: e2e traces/videos 14 d, perf reports 90 d, docker images 30 d.

9. Test authoring rules

1. Name by behavior: describe("formula: dateAdd") → it("adds months crossing year and leap day"). No test1, no "works".
2. One behavior per it; arrange-act-assert in one screen; no more than 2 assertions per it (split otherwise) — except matrix suites which iterate the matrix in a loop.
3. Stable IDs in fixtures (cuid or literal) so failure reports are debuggable; generated ids are logged with the failing assertion.
4. No test may depend on test order or wall clock (fake clocks, unique slugs, clean fixtures).
5. Fix the code, not the test: a failing golden test means either a bug or an intended behavior change — the latter requires updating the golden file + the functional requirement in the same PR (with reviewer sign-off on the diff of expectations).
6. Flaky = P1 bug: any test with ≥ 2 CI flakes in a week gets an issue; flaky e2e specs are quarantined (test.fixme with issue link) and tracked in docs/tasks/ until fixed.
7. Every bug fix ships with a regression test at the lowest pyramid level that catches it (AGENTS.md rule 9).
8. Coverage is a floor, not a goal: 90 % with meaningless assertions is worse than 80 % with the matrix suites — reviewers reject assertion-free coverage padding.