Performance — AI Software Factory

How to run, read, and gate the performance suites (k6 + Playwright lag e2e). This doc is the developer-facing side: the architecture, query shapes, index layout, tuning decisions, and the measured numbers live in search-architecture.md §10 ("Performance"). The task spec (SLA values, profiles, acceptance criteria) is phase-08-search.md (TASK-0100). Related: testing.md (the suite pyramid), NFR-4 / NFR-S5 in non-functional-requirements.md.

1. The suites at a glance

Shared fixture: tests/performance/fixtures/search-100k.ts — 100,000 deterministic docs (42% pages / 26% records / 16% files / 10% comments / 5% databases / 0.4% users), 2 perf-* workspaces, 200 users. Idempotent seed/reset; the seeder verifies the landed count is exactly 100,000 before finishing.

2. Running the search suites locally

Prereqs: the dev stack (API :4000, worker :4003, OpenSearch :9200, Postgres, Redis — see development-guide.md), a k6 v0.57.0 binary on PATH, and the API started with a raised per-user rate limit (RATE_LIMIT_API=100000; the default 100 req/min/user would 429 the load).

cd tests/performance
pnpm perf:seed:search        # idempotent; seeds exactly 100,000 docs (~20 s OS bulk)
pnpm check:search            # k6 archive parse gate (syntax only — a first RUN is the real test)
pnpm search:100k:reduced     # quick: ~1 min, 2×-flake thresholds
pnpm search:100k             # full: ~5 min, DoD thresholds (p95<500 ms, p99<1 s, err<0.1%)
pnpm perf:reset:search       # tear the fixtures down (OS + DB rows)

k6 env knobs (all overridable per run): K6_API_URL, K6_JWT_SECRET (must equal the API's JWT_SECRET — the suite mints real HS256 access tokens for the fixture users), K6_SEARCH_WORKSPACES, K6_VUS / K6_DURATION_SECONDS / K6_RATE, K6_P95_MS / K6_P99_MS / K6_ERROR_RATE, K6_VERSION (fingerprint label only).

The lag e2e runs like any other e2e spec:

cd tests/e2e
pnpm test search-index-lag --project=chromium-desktop

3. Reading a report

Every k6 run writes (into the package dir, since k6 writes to its CWD):

• reports/search-full.json / reports/search-reduced.json — raw k6 --summary-export (per-metric values incl. the configured summaryTrendStats).
• search-summary.json — same data, always written by k6.
• reports/search-last.json — the suite's fingerprinted report (handleSummary): latencyMs (p50/p95/p99/avg), serverTookMs (the OpenSearch took the API reports — cross-check the client number), errorRate, docCount (re-fetched from the alias at summary time), fingerprint (k6 version label, OpenSearch version, the thresholds in effect), and the profile (VUs/duration/rate/query/workspaces).

docCount is the one field that must be checked before trusting the numbers: a run against an empty or partially seeded index "passes" trivially. The seeder's own count verification plus a spot-check of docCount in the report is the guard.

Current baseline (2026-09-29, 4 vCPU / 15 GB dev machine — smaller than the DoD reference environment, 8 vCPU / 16 GB):

Both profiles over docCount 100,000, OpenSearch 2.15.0. The full profile passes its DoD thresholds (p95 < 500 ms, p99 < 1 s) with a ~10× margin.

4. Regression gating (how a number becomes a failure)

1. CI, merge gate — the perf-search job (.github/workflows/ci.yml) runs the reduced profile on push to main (self-contained: its own postgres/redis/opensearch services, pinned k6 v0.57.0, API under tsx, seed, run, artifact). Its thresholds are the 2×-flake values (p95 < 1000 ms, p99 < 2000 ms, err < 0.1%): a noisy runner must not fail a merge, but a real regression (≥ 2× the target) still trips it.
2. CI, nightly — the same job's full profile (100 VUs / 5 min / 50 req/s) with the DoD thresholds (p95 < 500 ms, p99 < 1 s, err < 0.1%) on the scheduled cron + manual workflow_dispatch. A breach fails the night and leaves the report as an artifact.
3. Baseline diff (maintainer step) — the task spec's "> 20% p95 regression vs. baseline fails the nightly" comparison is not yet automated: the baseline is reports/search-last.json, which per the spec is committed on demand by a maintainer, not per-PR. Until the baseline is committed, the nightly gate is the absolute threshold above; the relative comparison is a maintainer judgment on the artifact. (Recorded as a follow-up, not a Phase-8 gap: the suite already writes both numbers side by side in the report.)
4. Lag e2e — the Playwright spec's 30 s deadlines are the hard SLA checks; they run in the normal e2e job.

5. Why the numbers are what they are (pointers, not a repeat)

• Client latency ≈ server took + one network hop; at 100k docs the query is a bool (must: multi-match; filter: the grant-snapshot term/terms clauses) over a 1 s-refreshed alias — see search-architecture.md §4–§5.
• refresh_interval stays at 1 s (no tuning — the 30 s lag SLA is met by a wide margin); no JVM heap or thread_pool changes were needed at 100k — search-architecture.md §10.4.
• The hybrid (BM25 + kNN + RRF) path is not exercised by this suite — the k6 suite drives the default full-text mode only (hybrid is a Phase-12 capability-flag feature; see §9 of the architecture doc).

6. Troubleshooting