Performance Tests (k6)

Load and performance suites for AI Software Factory, written in k6. k6 is a Go binary, not an npm dependency — install it once:

# macOS
brew install k6
# Linux (Debian/Ubuntu)
curl -s https://packagecloud.io/install/repositories/grafana/k6/script.deb.sh | sudo bash
sudo apt-get install k6
# Anywhere (pinned release):
# https://github.com/grafana/k6/releases

The pnpm check script verifies k6 is on PATH.

k6 version pin: v0.57.0. CI installs exactly this (a pinned release tarball). v0.57 is the first release whose goja engine parses this suite's ES2021 numeric separators (1_000). The archive-parse gate (pnpm --dir tests/performance check:load) makes a script that fails to parse fail the lane before the load run. Several v0.57 behaviors the suites rely on are non-obvious — see k6 v0.57 quirks before editing the scripts.

---

The TASK-0135 suites (the Phase-15 §26 gates)

Phase 15 ("production") requires the four §26 load criteria to be enforced in CI: report-only in dev, hard in staging. These four suites are the enforcement. Each has a smoke profile (seconds, mechanics — runs in dev/PR) and a full profile (the real §26 shape — runs in staging).

Script	§26 criterion	Smoke profile	Full profile	Hard threshold (staging)
api-load.js	API 1k rps p95 < 300 ms	5 rps / ~11 s	1k rps (400 tree + 300 list + 200 search + 100 create) / 9 min	http_req_duration p95 < 300 ms; per-scenario p95 < 300 ms; http_req_failed < 0.1%
realtime-load.js (via run-realtime-load.sh)	Realtime 10k conns / 200 msg/s p95 < 200 ms + zero dropped Yjs ops	100 conns / 10 s ramp / 20 s hold	10,000 conns / 5-min ramp / 15-min hold	op_ack_ms p95 < 200 ms; connect_ms p95 < 1000 ms; join_failures < 0.1%; protocol_errors = 0; ws_unexpected_closes < 0.5%
database-load.js	100k-record table p95 < 500 ms + 100 rps record create	~1 rps / ~2 min	100k-table scroll/open + 100 rps create / ~4 min	table load p95 < 500 ms; record-create p95 < 500 ms
editor-local.mjs	Editor interaction p95 < 50 ms + block move < 16 ms (local)	20 type / 10 create / 10 move	500 type / 100 create / 200 move	interaction p95 < 50 ms; block-move p95 < 16 ms

The spec's record-create is "100 vusers / ~5% of traffic"; 100 of 1000 is 10%. api-load.js follows the explicit 1k-rps profile (400+300+200+100) and records the discrepancy in its header rather than guessing.

The seeders + helpers

File	Role
seed-100k.mjs	Builds the 100k-record fixture: K6_DBS databases (default 1 = one true 100k table) × K6_RECORDS records (default 100,000), 200 hub pages, one 10k-block page, in a dedicated perf-load workspace. Idempotent (a seed-100k-v1 marker row in test_seeds; a no-op when the marker matches). The API must be up (it registers/logs in the perf user, then inserts the records directly via Prisma). ~4–5 min for 100k.
extract-100k-env.mjs	Reads the seed-100k-v1 marker back and prints shell-exportable K6_WORKSPACE_ID / K6_DB_ID / K6_PAGE_IDS / K6_BIG_PAGE_ID — the ids the suites' setup() require (they throw when empty). CI: eval "$(node tests/performance/extract-100k-env.mjs 2>/dev/null)".
run-realtime-load.sh	One command = the whole realtime DoD check: seed the load identities (helpers/seed-1k-load.mjs, idempotent) → run realtime-load.js → the host-side zero-dropped-ops check (verify-zero-op-loss.mjs). Needs K6_JWT_SECRET + REALTIME_TRUST_PROXY=1 on the server.
verify-zero-op-loss.mjs	The "zero dropped Yjs ops" gate. k6's VM has no Yjs, so it replays the verify room's ops host-side and compares the final doc hash to a sequential-replay hash. Must run immediately after k6 (inside the room's ~5-min GC window). Exit 1 on a mismatch.
threshold-breach-demo.js	The red-proof: a throwaway suite that hits /api/v1/health against an impossible p95 budget (K6_P95_MS=1) under K6_STRICT=1, so k6 exits 99. It proves the hard tier actually blocks on a breach (a lowered threshold turns the step red) without touching the real fixtures. The CI lane asserts rc == 99.

The legacy suites (pre-TASK-0135, still wired to CI)

Script	What it measures
api.js	Mixed API load to 1,000 rps (GET page / GET blocks / POST block batch / POST search) — p95 < 300 ms, failures < 0.1%.
realtime.js	10,000 concurrent WebSocket connections, Yjs update every 500 ms — RTT p95 < 200 ms, < 0.5% disconnects.
realtime-1k.js	1,000 concurrent WebSocket connections (the TASK-0095 S14 baseline) — RTT p95 < 200 ms, 0 protocol errors.
database.js	100k-record database: table page load, board grouping, filtered query, record create.
search-search.k6.js	100k-doc search query performance (the dedicated search lane — see the K6_SKIP_SEARCH note).

The new *-load.js suites supersede the legacy api.js / realtime.js / database.js for the §26 gates, but the legacy scripts remain the perf / perf-search CI jobs' targets. They are not removed.

---

The report-only vs hard tiers

The single switch that decides whether a run can fail the pipeline is K6_STRICT:

• K6_STRICT=1 (default — the hard tier, staging) → the script's real options.thresholds are active. A breach makes k6 run exit 99, which fails the CI step and blocks the release pipeline.
• K6_STRICT=0 (report-only — local + dev/PR) → every threshold is replaced with a trivially-true expression (avg<1e15, rate<=1, count<1e15 — see the v0.57 quirks for why those exact bounds). The run never fails on a threshold, but the k6 summary still prints the real p95/p99/error values, which CI uploads as the PR artifact.

editor-local.mjs is report-only by construction (it never fails on a budget — see its header), and run-realtime-load.sh derives K6_STRICT from K6_PROFILE (smoke→0, full→1) unless you override it.

The two CI tiers

Tier	Runner	Trigger	Profile	K6_STRICT	Zero-op-loss verify
dev / report-only	ubuntu-24.04 (GitHub-hosted)	every PR (pull_request) + push to main/develop + nightly	smoke	0	skipped (K6_VERIFY=0)
staging / hard	[self-hosted, factory-staging] (4 vCPU)	workflow_dispatch → perf_mode=staging-hard	full	1	on (K6_VERIFY=1)

The dev tier's job is to prove the suites + endpoints work end-to-end and to print real numbers into the PR artifact; it is not sized for the full profile (a GitHub-hosted runner is not a 1k-rps / 10k-conn machine), so dev numbers are mechanics, not budgets. The staging tier is where a §26 breach blocks the release.

CI job map (.github/workflows/ci.yml)

Job	Suites	Tier	When
perf	realtime-1k.js (legacy) + run-realtime-load.sh SMOKE (the new 10k suite)	report-only	every PR (smoke), push to develop/main (1k baseline)
perf-load	api-load.js + database-load.js SMOKE + the threshold-breach red-proof	report-only	every PR + push + nightly
perf-editor	editor-local.mjs SMOKE (full dev stack)	report-only	every PR + push + nightly
perf-search	search-search.k6.js (100k-doc corpus)	reduced/full	push to main (reduced) + nightly (full)
perf-staging-hard	all four FULL + the realtime zero-op-loss verify	hard (blocking)	workflow_dispatch → staging-hard

The staging-hard job is self-contained on the self-hosted runner: it boots the repo's dev compose infra (docker compose — GitHub services: are unavailable on self-hosted runners) + the full app stack from source (pnpm dev), seeds the 100k fixture (K6_DBS=1 → one true 100k table) and the 100k-doc search corpus (perf:seed:search, so api-load runs the true 1k-rps matrix including the 200-rps search scenario — no K6_SKIP_SEARCH), then runs all four suites in full + hard.

The Phase-15 release pipeline's stage 8 performance (infra/pipeline/release.yml — absent in this repo) is the target that reads the deployed staging after the rollout is Ready. The ci.yml perf-staging-hard job is the in-repo, self-contained equivalent and what release-pipeline stage 8 invokes (or a target-staging variant of it).

---

Environment contract

All vars are optional unless marked required. Defaults live in the scripts.

Var	Default	Purpose
K6_API_URL	http://localhost:4000	API base URL (api-load / database-load / editor-local / breach-demo)
K6_REALTIME_URL	ws://localhost:4001	WebSocket endpoint (realtime-load)
WEB_BASE_URL	http://localhost:3000	Web base URL (editor-local only)
K6_EMAIL / K6_PASSWORD	demo@factory.local / Factory!2345	Auth for the suites (the seeded demo user)
K6_WORKSPACE_ID	— (required by api-load / database-load / editor-local)	The seeded workspace. From extract-100k-env.mjs, else the user's first workspace.
K6_PAGE_IDS	— (required by api-load)	Comma-separated hub page ids (200 from the seeder marker).
K6_DB_ID	— (required by api-load / database-load)	Comma-separated database id(s) (up to 5 from the marker). database-load uses the first.
K6_BIG_PAGE_ID	—	The 10k-block page id (seeder marker; informational).
K6_STRICT	1 (hard) / 0 (report-only)	The hard-vs-report-only switch (see above).
K6_PROFILE	full (realtime via the runner)	smoke vs full profile (per-suite shape).
K6_SKIP_SEARCH	— (search included)	1 drops the api-load search scenario. The dev/PR lanes set this — see below.
K6_VERIFY	1 (the DoD gate)	run-realtime-load.sh: run the host-side zero-op-loss check after k6. 0 = skip (report-only tier).
K6_JWT_SECRET	— (required by the realtime runner + verify)	The server JWT_SECRET; the suites mint HS256 tokens for the perf users under it. Must equal the running API/realtime JWT_SECRET.
K6_USERS / K6_VUS / K6_RAMP_SECONDS / K6_HOLD_SECONDS	per-profile	realtime-load scale overrides.
K6_DBS / K6_RECORDS / K6_BLOCKS / K6_PAGES	1 / 100000 / 10000 / 200	seed-100k.mjs shape.
K6_P95_MS	1 (breach-demo)	The breach-demo's p95 budget (set impossibly low to force a breach).

Why the dev/PR lanes set K6_SKIP_SEARCH=1

The seed-100k.mjs fixture inserts its 100k records directly via Prisma — no indexing events, no search_documents mirror. The OpenSearch content index is only ever created by the worker, which the report-only lanes do not run. So in those lanes the search endpoint has no corpus and 100% of the search scenario's requests would fail — a corpus/fixture limitation, not a suite defect. The dev/PR lanes therefore skip that scenario (K6_SKIP_SEARCH=1) and measure the other three at scale. Search-query perf at the 100k-doc scale is the perf-search job's job (it seeds its own corpus via fixtures/search-100k.ts). The staging-hard lane seeds the corpus explicitly (perf:seed:search) and keeps search in (no K6_SKIP_SEARCH).

---

Running locally

pnpm docker:up          # postgres, redis, minio, opensearch
pnpm db:deploy && pnpm db:seed
pnpm dev                # api :4000, web :3000, realtime :4001, worker

The realtime gateway must run with REALTIME_TRUST_PROXY=1 (per-VU X-Forwarded-For connect-limiter identity) and its JWT_SECRET available as K6_JWT_SECRET.

The 100k fixture + the four suites

# 1. The API must be up. Seed the 100k fixture (idempotent; ~4-5 min).
pnpm --dir tests/performance seed:100k

# 2. Read the seeded ids into env (from the marker row).
eval "$(node tests/performance/extract-100k-env.mjs)"

# 3. The four suites — SMOKE (seconds, mechanics) first, then FULL.
K6_STRICT=0 pnpm --dir tests/performance api-load:smoke
K6_STRICT=0 pnpm --dir tests/performance database-load:smoke
K6_STRICT=0 K6_PROFILE=smoke K6_VERIFY=0 K6_JWT_SECRET=<secret> \
  pnpm --dir tests/performance realtime-load:smoke
K6_STRICT=0 pnpm --dir tests/performance editor-local

# FULL profiles (the real §26 shape; on a machine sized for them):
K6_STRICT=1 pnpm --dir tests/performance api-load
K6_STRICT=1 pnpm --dir tests/performance database-load
K6_STRICT=1 K6_VERIFY=1 K6_JWT_SECRET=<secret> \
  pnpm --dir tests/performance realtime-load        # 10k conns + zero-op-loss verify
K6_STRICT=1 K6_PROFILE=full pnpm --dir tests/performance editor-local

The archive-parse gate (no live stack needed):

pnpm --dir tests/performance check:load

The red-proof (threshold-breach demo)

# Exits 99 (a hard breach) — this is what the CI lane asserts:
k6 run tests/performance/threshold-breach-demo.js \
  --env K6_API_URL=http://localhost:4000 --env K6_STRICT=1 --env K6_P95_MS=1; echo "rc=$?"

# Exits 0 (report-only) under K6_STRICT=0.

---

Interpreting results

• Local dev runs are report-only. A laptop or CI runner with docker-compose services will not hit the p95 budgets; treat local numbers as trend data (regressions between commits), not pass/fail.
• Hard budgets are enforced in staging (the 4 vCPU self-hosted runner — the same instance class the spec targets), where the thresholds in each script cause the run to fail (k6 exit 99 on a threshold violation) and the results are uploaded as an artifact in CI.
• Thresholds are declared in each script's options.thresholds block — they are the single source of truth; do not edit the pass criteria here.
• The editor-local budgets (interaction < 50 ms, move < 16 ms) are client- side / local — measure them on the machine serving the web, not across a network.

---

k6 v0.57 quirks that shape these suites

These are verified behaviors of the pinned k6 (v0.57.0) that the scripts are written around. Re-probe before "fixing" any of them — several look like bugs but are the actual contract:

• ws.connect() returns null on success. Any property access on the return value throws TypeError: Value is not an object: null. The join-success oracle is the server hello frame (a joined flag), not the return value. (realtime-load.js uses this in all three execs.)
• Rate thresholds are fractions (0..1), not percents. The always-pass bound is rate<=1 — not rate<1 — because a true 100% failure rate is exactly 1.0, which rate<1 would (wrongly) turn red. All soft Rate thresholds use rate<=1.
• sleep and log are not globals. import { sleep, log } from "k6".
• crypto.hmac(alg, secret, data, "base64") — the 4th arg is the output encoding. The token signing uses a single base64 encode; double-encoding the HMAC breaks the JWT.
• No warn: soft-threshold prefix. The expression parser rejects any method after warn:, so a "soft, reported" p99 cannot be a threshold — it is surfaced via summaryTrendStats (the run summary + artifact always print p(99)) instead.
• A threshold on a metric with ZERO samples passes vacuously (an empty Trend = 0). This is load-bearing for K6_SKIP_SEARCH=1: dropping the search scenario leaves search_ms empty, and its p(95)<300 threshold passes vacuously — so no threshold change is needed when the scenario is skipped.
• ramping-arrival-rate takes startRate + stages (it rejects duration/phases/targetRate), and needs integer rates (a fractional target like 0.4 is rejected). Each spec scenario is one ramping-arrival-rate with three stages: [0→peak over RAMP] [peak held HOLD] [peak→0 over RAMP_DOWN].
• check() keys must be string literals — the goja engine rejects computed (non-literal) object keys inside check(). Per-scenario latency is carried by Trend metrics; the checks guard status + body only.
• Hold-loop pattern. ws.connect is non-blocking, so each realtime exec ends with while (Date.now() < deadline) sleep(1); the socket close ends the iteration.

---

Known apps/ defects that affect the realtime + editor lanes (out of scope for TASK-0135)

These are app bugs, not suite defects. TASK-0135 is test/infra-only and does not modify apps/. The suites are written to be correct once these are fixed; until then they behave as documented.

1. Realtime permission resolver 500s every ?room= doc-room join

apps/realtime/src/prisma-store.ts:331 passes the Prisma enum "MEMBER" (uppercase) into the @factory/permissions engine, which is keyed by wire-case role. levelRank(undefined) throws → a 500 on every doc-room join (evidence: a one-shot join returns 500 unknown permission level: undefined; the DB role is "MEMBER"; the error repeats across the realtime log). The one-line fix is roleFromPrisma(membership.role) (packages/permissions/src/levels.ts:106-112).

How the suite observes it (correctly): the rejected join hits the server's catch → socket.destroy() before handleUpgrade, so k6 sees a close only — no JSON frame, no hello. The joined oracle stays false → counted as join_failures, not protocol_errors (so the hard protocol_errors = 0 threshold is not polluted by this bug). The host-side zero-op-loss verify also fails today (the verify writer never lands its ops), which is why the dev/PR tier runs with K6_VERIFY=0 (report-only) and the real gate (K6_VERIFY=1) bites in staging — where a red is a legitimate blocker.

After the resolver is fixed: re-run realtime-load:smoke — expect join_failures ≈ 0, op_ack_ms / connect_ms populated, and the hard tier green.

2. Committed CSP / devtool defect makes the dev web unbootable under its own headers

TASK-0134's webSecurityHeaders (packages/shared/src/security-headers.ts:84) sends script-src 'self' 'unsafe-inline' with no 'unsafe-eval', and Next 14 dev builds emit their chunks via eval() (devtool eval-source-map) — so the dev web is rejected by its own CSP and hangs on "Loading workspace…" with zero API calls. A decisive e2e-shaped probe (no bypassCSP, live :3000) confirms: BODY: "Loading workspace…", 0 block rows, 0 api-calls, pageerror: EvalError … 'unsafe-eval'.

• editor-local.mjs bypasses CSP on the test context (bypassCSP: true, documented in its header) so it can still measure — but the proper fix (a dev-only devtool override / 'unsafe-eval' carve-out) is an app change, out of scope here.
• The e2e job's fixture does NOT bypass CSP, so CI e2e hits the same wall until the parent fixes the headers — a cross-task defect to escalate.

---

Adding a suite

1. New .js file in this directory; use the per-VU data() auth cache + the ws.connect (ignore return) / hello-oracle / hold-loop pattern from api-load.js / realtime-load.js.
2. Add a HARD_THRESHOLDS + SOFT_THRESHOLDS pair and select on K6_STRICT.
3. Add a pnpm script ("<name>": "k6 run ...") + a :smoke variant.
4. Add it to check:load (the archive-parse gate).
5. Update the suite table above.