Tests
The test pyramid for AI Software Factory. Every feature lands with the matching layer(s) filled in — this is part of the AGENTS.md Definition of Done, not a nice-to-have.
┌────────────┐
│ E2E │ Playwright — full browser, full stack
│ (fewest) │ tests/e2e
┌┴────────────┴┐
│ Integration │ NestJS + real Postgres/Redis (Testcontainers)
│ │ tests/integration
┌─┴──────────────┴─┐
│ Performance │ k6 — load & latency budgets
│ │ tests/performance
┌─┴──────────────────┴─┐
│ Unit │ Vitest/Jest — formula engine, permission
│ (most) │ engine, block ops, filtering, sorting
└──────────────────────┘ packages/*/src (alongside the code)
Suite map
| Suite | Location | Runner / tool | What it owns | Command (from repo root) | CI stage (.github/workflows/ci.yml) |
|---|---|---|---|---|---|
| Unit | packages/*, apps/api (colocated) |
Vitest (packages) / Jest (api) | pure logic: formula AST evaluator, permission engine, block ops, database filter/sort/group, relations, rollups | pnpm test:unit |
unit |
| Integration | tests/integration |
Vitest + Testcontainers | API contract over real Postgres/Redis: auth, workspaces, pages, blocks, databases, comments, files, webhooks, permissions (tenant isolation + audit) | pnpm test:integration |
integration |
| E2E | tests/e2e |
Playwright (Chromium desktop + mobile) | user journeys through the real web app against the full dev stack | pnpm test:e2e |
e2e |
| Performance | tests/performance |
k6 (Go binary) | API p95 < 300 ms @ 1k rps; realtime < 200 ms @ 10k conns; 100k-record DB < 500 ms | pnpm --dir tests/performance <api|realtime|database> |
manual (staging), not in PR CI |
Passing criteria (the budgets)
From PRODUCT-SPEC §Performance — enforced as thresholds in the perf scripts (the single source of truth), hard waits in E2E, and soft (logged) budgets in E2E perf specs (dev hardware is too noisy for hard numbers):
| Metric | Budget | Enforced by |
|---|---|---|
| Page load (FCP) | < 2 s (prod) | e2e performance.spec.ts (soft ×1.5) + api.js p95 |
| Editor interaction | < 50 ms | e2e performance.spec.ts (soft ×3) |
| Local block move | < 16 ms | e2e (UI event loop — soft) |
| Search | < 500 ms | api.js search_ms p95; e2e search spec (5 s availability budget) |
| API (mixed load) | p95 < 300 ms | api.js thresholds (hard, staging) |
| Realtime round-trip | p95 < 200 ms | realtime.js thresholds (hard, staging) |
| 100k-record DB queries | p95 < 500 ms | database.js thresholds (hard, staging) |
| Record create @ 100 rps | p95 < 300 ms | database.js thresholds (hard, staging) |
Selector convention (E2E)
Playwright specs locate elements by data-testid only (text/role locators
are banned in specs — they couple tests to copy and break on i18n). The
canonical ids (kept in sync with apps/web):
| testid | What it is |
|---|---|
editor-root |
the editor container |
block-{id} |
a top-level block row (also carries data-block-id) |
drag-handle |
THE block's drag handle — one id, appears on hover |
new-block |
trailing "keep typing" area (click → new focused paragraph) |
slash-menu / slash-menu-item-{name} |
slash command palette + its items |
todo-checkbox-{id} |
the checkbox on a TODO block |
toggle-chevron-{id} |
a toggle's open/close caret (controlled <details>) |
column-list / column-{n} |
a columns container + its n-th cell |
code-block-pre |
the rendered (non-edit) code block |
block-context-menu |
a block's right-click menu (role="menu") |
context-menu-turn-into / context-menu-turn-{type} |
"Turn into…" submenu + target types |
context-menu-delete |
the Delete item in a block menu |
sidebar / sidebar-page-{id} |
workspace sidebar + a page node |
sidebar-new-page |
"New page" button |
page-title / page-icon / page-cover |
page header elements |
page-context-menu / context-menu-{action} |
page right-click menu + items |
trash-section / trash-item-{id} / trash-restore-{id} / trash-purge-{id} |
trash UI |
favorite-item-{id} |
favorites section item |
page-versions / page-version-entry |
version history panel + entries |
workspace-switcher / workspace-dialog / workspace-settings |
workspace UI |
settings-tab-{profile|billing|security|integrations} |
settings tabs |
member-row-{userId} / member-role-{userId} |
member list |
auth-email / auth-password / auth-submit / auth-error / user-menu / user-menu-logout |
auth UI |
database-view / db-table / db-board / db-row-{recordId} / db-cell-{recordId}-{propName} / db-header-{propName} / db-add-row / db-filter-* / db-group-* / db-board-column-* / db-board-card-{recordId} |
database UI |
search-button / search-dialog / search-input / search-result-{id} / search-result-type-{id} / search-filter-type / search-recent / search-suggestion |
search UI |
file-card-{fileId} / file-name-{fileId} / file-download-{fileId} / file-dropzone / file-input / toast-error |
files UI |
comment-popup / comment-input / comment-thread-{id} / comment-{id} / comment-replies-{id} / comment-reply-input / comment-resolve-{id} / mention-badge-{userId} / mention-picker / reaction-bar / reaction-picker |
comments UI |
share-button / share-dialog / share-password-input / share-revoke / public-page |
sharing UI |
automation-list / automation-new / automation-row-{id} / automation-runs-{id} / automation-run-{runId} |
automation UI |
When apps/web adds a new interactive element, it ships with its testid in
the same PR (see AGENTS.md DoD: "E2E test").
Environment per suite
| Suite | Env it needs | Where it comes from |
|---|---|---|
| Unit | none (pure JS) | — |
| Integration | Docker (Testcontainers pulls postgres:16-alpine, redis:7-alpine); Prisma client generated |
pnpm db:generate + Docker daemon |
| E2E | full stack: pnpm docker:up + pnpm db:deploy + pnpm db:seed + pnpm dev |
local dev / CI e2e job (services + turbo dev) |
| Performance | staging-shaped stack; 100k-record DB pre-seeded (tests/performance/helpers/seed-100k.mjs) |
staging (infrastructure/terraform) |
Variable names are the canonical .env.example set (DATABASE_URL,
REDIS_URL, SEARCH_URL, S3_ENDPOINT, JWT_SECRET, …). No suite reads
secrets from anywhere else; integration/perf harnesses generate random
secrets at runtime.
How to add a test (checklist)
- Unit: colocate
foo.unit.test.tsnext tofoo.tsin the package; cover the decision table (all branches, the error cases). - Integration: new
<feature>.integration.test.tsintests/integration— drive everything throughcreateTestApp()+ the HTTP surface; use Prisma only to assert side effects (audit rows, delivery rows). Seetests/integration/README.md. - E2E: extend the matching spec in
tests/e2e/specs/(or add<feature>.spec.ts); use thetest/expectfrom../fixtures(authPage,api,secondUser(),freshWorkspace()); locate bydata-testidonly; clean up via the fixtures' teardown. - Performance: only when a PRODUCT-SPEC budget is in play; add a k6
script + thresholds + the
pnpmscript intests/performance. - Skip markers: later-phase features are written fully and marked
test.skip(true, "lands in Phase N — <reason>")— never a bareTODO. When the phase lands, delete the skip line; the test is ready.
Known limitations
- Realtime E2E flakiness: WebSocket delivery assertions are timing
sensitive. The realtime delivery tests are skipped until Phase 7
(comments.spec.ts); the perf suite owns realtime under load. Where a spec
must wait for a WS update, poll an explicit UI state (never raw
networkidle, which a live socket defeats — seetests/e2e/utils/wait.ts). - Perf in dev is report-only: dev hardware (laptop, CI runner) cannot hit the absolute budgets; the hard thresholds only mean something on the staging instance class. Don't chase a local p95 — trend it instead.
- Integration = real Postgres, mocked object store:
filesintegration tests exercise the full upload state machine but not a real S3/MinIO (the dev-stack e2efiles.spec.tscovers the presigned-PUT loop against real MinIO). OpenSearch is likewise e2e-only (slow to boot in containers per spec). - 10k/100k seeds are direct-DB fixtures: the seed helpers
(
seedRecordsin integration,seed-100k.mjsin perf) bypass the API on purpose (bulk fixture speed). They must never be used to bypass authorization in a spec — API-level access checks stay the test target. - E2E mobile project: a coverage pass (375×812), not the primary surface — it retries 3× in CI and is expected to be green but is not gated more strictly than desktop.
Running everything locally
pnpm install
pnpm db:generate
pnpm test:unit # fastest loop
pnpm test:integration # needs Docker
pnpm docker:up && pnpm db:deploy && pnpm db:seed && pnpm dev
pnpm test:e2e # in a second terminal (stack must be up)
pnpm --dir tests/performance api # k6 — staging only for real numbers