Non-Functional Requirements — AI Software Factory
Measurable, verifiable quality attributes. Every target has an ID (
NFR-xx), a measurement method (how a test or dashboard proves it), and a verification owner. Performance targets derive fromPRODUCT-SPEC.md§26 and the ROADMAP exit criteria; security targets from security-architecture.md; deployment/DR targets from deployment.md. Test mapping (which suite proves which NFR) is in testing.md §8.
Verification vocabulary: perf suite = tests/performance (k6/Artillery,
run in CI nightly + on Phase 15 gate); load profile = named scenario in the
perf suite; chaos = integration-stage fault injection; SLO burn = Grafana
SLO dashboard evaluated over 28 d windows in staging/prod.
1. Performance targets (from PRODUCT-SPEC §26)
| ID | Metric | Target | Measurement method | Verified by |
|---|---|---|---|---|
| NFR-1 | Initial page load (authenticated app shell, page open) | < 2 s (p75, local dev / reference self-host box) | Playwright performance fixture: navigationStart → load + first meaningful paint of page title; 3 runs, median; run in CI on the e2e job's reference hardware |
e2e (perf fixture) + Lighthouse CI (perfmance ≥ 80) |
| NFR-2 | Editor interaction (type, convert block, toggle todo) | < 50 ms (p95, keystroke → rendered) | In-app performance hook: performance.mark on input event → requestAnimationFrame after commit; sampled 1 % of events, reported via OpenTelemetry metric editor.interaction.duration; perf suite also drives 1 000 keystrokes and asserts client-reported p95 |
unit (engine ops) + e2e + otel metric assertion in perf |
| NFR-3 | Local block movement (drag reorder, same page) | < 16 ms (p95, commit → next frame) | Same mark/measure around dnd-kit onDragEnd → React commit; additionally a headless e2e drag of a 500-block page asserting frame time (via requestAnimationFrame delta) |
e2e (phase-4 exit criterion) |
| NFR-4 | Search response (POST /search) |
< 500 ms (p95) | k6 HTTP check on a 1M-doc fixture index (100k pages + 500k blocks); p95 over 1 000 requests; OpenSearch query timing also logged server-side for cross-check | perf suite + integration (index-lag test) |
| NFR-5 | API p95 (all public endpoints, excluding uploads) | < 300 ms (p95) | k6 per-endpoint assertions from a recorded VU replay; server-side otel histogram http.server.request.duration as source of truth (client p95 as proxy); per-endpoint budgets published in the Grafana dashboard |
perf suite + staging SLO |
| NFR-6 | Realtime update latency (edit → remote render) | < 200 ms (p95, same deployment) | Two headless browsers in the same Playwright worker: browser A applies an edit, browser B marks on Yjs apply; p95 over 500 iterations; ws round-trip also measured (ws.ping) |
e2e (phase-7 exit criterion) |
| NFR-7 | 100k-block page renders (virtualized) | Tree loads & is interactive in < 2 s; memory < 500 MB for the page route | e2e: open seeded 100k-block page (fixture script), measure to first paint + run 10 scroll operations without > 100 ms jank; Chrome DevTools protocol memory snapshot | e2e + perf (node RSS in containerized run) |
| NFR-8 | 100k-record database renders | Table view opens with first rows in < 2 s; scroll at 30 fps; POST /databases/:id/query returns a page of 100 rows in < 300 ms |
e2e with 100k-record fixture (seed script in tests/integration/fixtures); query timing from API response headers |
e2e + perf |
| NFR-9 | File upload | 500 MB file completes; throughput ≥ 50 MB/s on 1 GbE; 99 % of uploads < 2× theoretical time | k6 multipart upload test (50 MB × 20 VUs) + one 500 MB e2e (phase-9 exit) | e2e + perf |
| NFR-10 | Concurrency: two editors, same block | No lost updates; both texts merge (CRDT) | e2e: 2 browsers edit the same paragraph from opposite ends simultaneously, 20 iterations; final text contains both edits | e2e (phase-7) |
| NFR-11 | Batch API: POST /databases/:id/records (100 records) |
< 1 s (p95) | k6 on 10k-record database; assertion on response time | perf |
| NFR-12 | Import: 10k-block markdown file | Completes < 60 s; progress events every ≤ 2 s | e2e timing of the import job | e2e (phase-10) |
Scaling headroom rule: targets are measured on the reference self-host box (4 vCPU / 8 GB RAM, local Postgres/Redis/OpenSearch/MinIO) for self-host parity; the Kubernetes cluster profile (8 vCPU / 16 GB per api pod × 3) must hold the same targets at 3× concurrent users.
2. Scalability
| ID | Requirement | Target | Measurement method |
|---|---|---|---|
| NFR-S1 | Record volume | A single database sustains 100 000 records; a workspace sustains 1 000 databases / 1 000 000 records | Seed fixture; query plan inspection (EXPLAIN ANALYZE archived in perf run); p95 view-load < 2 s at 100k (NFR-8); record inserts at 10 rps for 10 min without degradation |
| NFR-S2 | Concurrent editors | 10 000 concurrent WebSocket editors across 100 pages (100/page) | Artillery ws scenario: 10k connections, 10 % editing (2 updates/s each), 30 min; assertions: p95 update < 200 ms, memory per realtime pod < 2 GB, no connection drops > 0.1 % |
| NFR-S3 | API throughput | 1 000 rps sustained (read:write = 80:20 mix) | k6 ramp to 1k rps, 30 min; p95 < 300 ms, error rate < 0.1 %, CPU < 70 % headroom |
| NFR-S4 | Event bus throughput | Domain events: 5 000/s peak, no loss (at-least-once), consumer lag < 30 s | Load with 10k editors + automations; Redis stream XLEN growth + consumer-group lag metric bus.consumer.lag |
| NFR-S5 | Search index size | 10M indexed documents, query p95 < 500 ms, index job throughput ≥ 500 docs/s | OpenSearch cluster stats + perf run |
| NFR-S6 | Storage | 50 TB object storage per deployment; 1 000 files/s upload rate at the pipeline | Per-plan quotas (functional §12); pipeline queue depth metrics under load |
Capacity planning: every deployment ships with a capacity dashboard
(rows per table, connection pool utilization, queue depths, WS count) so
operators see the approach to these ceilings; horizontal scaling of
api (stateless), realtime (rooms shard by consistent hash of doc id),
worker (queue-backed) is the first lever — see
deployment.md §3.
3. Reliability & availability
| ID | Requirement | Target | Measurement / guarantee |
|---|---|---|---|
| NFR-R1 | API availability | 99.9 % monthly (≤ 43.8 min downtime) | SLO burn: 28 d rolling window on http.server.request success ratio (5xx excluded from "up" definition: an error ≠ outage, but sustained 5xx > 5 % counts); error budget alert at 50 % burn |
| NFR-R2 | Realtime degradation | Realtime server down ⇒ app remains fully usable: edits buffer (offline queue, NFR-O1), presence shows "realtime unavailable" chip, no crash loop; recovery auto-reconnects | chaos test: kill realtime pods mid-edit in integration e2e; assert no JS errors, edits persisted after 60 s |
| NFR-R3 | Search degradation | OpenSearch down ⇒ search returns 503 → UI empty state "Search is unavailable"; all other features unaffected | chaos: point SEARCH_URL at a dead port; run the 13 MVP flows |
| NFR-R4 | Queue degradation | Redis down ⇒ API returns 503 on enqueue-bound ops (file pipeline, automations, webhooks) with a clear error; reads/writes of core content still work (events logged to Postgres outbox fallback) | chaos: redis failover drill in staging |
| NFR-R5 | Data loss | Zero-loss for committed operations (durable writes); at-least-once for events; no silent drops | Postgres durability (synchronous_commit=on default), event outbox reconciliation job test |
| NFR-R6 | Pod/replica failure | Killing any single api/worker/realtime pod causes < 30 s user-visible impact (LB health checks + graceful drain) |
k8s chaos: kubectl delete pod during load; assertion on error spike < 30 s |
| NFR-R7 | Error budget policy | Error budget (0.1 %) exhausted ⇒ feature freeze for non-reliability work (release process gate) | Release checklist references SLO dashboard |
4. Security (targets — design in security-architecture.md)
| ID | Requirement | Target | Measurement |
|---|---|---|---|
| NFR-SE1 | OWASP ASVS | Level 2 conformance | ASVS L2 checklist (v4.0.3) executed in Phase 15 + dependency SCA (0 known critical/high unpatched, 30 d max patch SLA) + pen-test report filed |
| NFR-SE2 | Authorization bypass | 0 unauthorized reads/writes across the permission test matrix | Integration suite: full decision matrix (role × resource × level) run against the live API — 100 % pass required (CI gate) |
| NFR-SE3 | Injection | 0 SQL/NoSQL/XSS command-injection vectors | SQLi/XSS payloads in e2e security suite (parameterized queries via Prisma; output encoding); ZAP baseline scan (0 high) in CI |
| NFR-SE4 | Secret hygiene | 0 secrets in repo/history; all secrets from env/secret store | gitleaks + trufflehog in CI (blocking); secret-scanning PR check |
| NFR-SE5 | Auth endpoint abuse | Rate limits hold: 5 000 brute-force attempts → ≤ 5 account lockouts bypasses = 0 | Load test on /auth/login with credential stuffing simulation; assert 429s |
| NFR-SE6 | File pipeline | Malware sample (EICAR) blocked in 100 % of pipeline runs; 0 direct object key reads without permission | Integration: EICAR upload test (phase-9 exit); signed-URL expiry test |
| NFR-SE7 | Audit completeness | 100 % of mandatory events (security-architecture §12) emitted for the audit test script | Integration: run the audit scenario script, assert every event present in audit_logs |
| NFR-SE8 | Transport | 100 % of traffic TLS 1.2+ (1.3 preferred); HSTS max-age ≥ 1 year, includeSubDomains | curl/TLS scanner in CI against staging; header assertions in e2e |
5. Privacy & data retention
| ID | Requirement | Target |
|---|---|---|
| NFR-P1 | Tenant isolation | No query may return rows outside the caller's workspace (proven by the cross-tenant test suite: 50 random cross-workspace request pairs, 0 leaks — CI gate) |
| NFR-P2 | Data minimization | API responses contain only fields the caller needs (DTO projections); no PII in logs (PII scrubber on log pipeline; test: seed identifiable strings, assert absent from Loki) |
| NFR-P3 | Retention (platform defaults) | Trash: 30 days → purge. Password reset tokens: 1 h. Verification tokens: 24 h. Webhook delivery payloads: 90 days. Automation logs: 180 days. Audit logs: 400 days (org policy override, NFR-P4). File objects: with the file row (purge job) |
| NFR-P4 | Enterprise retention policies | retention_policies (PURGE/HOLD, per scope) enforced by nightly job; HOLD prevents purge (litigation-hold); test: policy set → job run → rows purged/kept per policy (phase-14 exit) |
| NFR-P5 | GDPR-style rights (self-host posture) | Export (personal data per user) and delete (user + cascade) operations available to workspace owners; deletion is soft (30 d) then hard; both audited |
| NFR-P6 | Cookies & tracking | No third-party cookies; no analytics by default; first-party functional cookies only (session, csrf); documented in the cookie notice (enterprise deploys may disable the notice) |
6. Backup & disaster recovery
| ID | Requirement | Target | Measurement |
|---|---|---|---|
| NFR-B1 | Postgres backup | WAL archiving continuous (PITR capable); base backup nightly | Restore drill: PITR to a point 10 min before a seeded "incident" marker; data verified (row-count checksum) — quarterly in prod, monthly in staging (runbook in deployment.md §8) |
| NFR-B2 | Object storage | S3/MinIO versioning + cross-bucket (or cross-region) replication | Restore drill: recover a deleted file to a prior version |
| NFR-B3 | OpenSearch | Snapshot repo (S3), snapshot hourly (delta) | Restore drill: reindex from snapshot, verify doc count + sample query |
| NFR-B4 | RPO | ≤ 5 minutes (all data stores) | Measured by t(now) - t(last consistent backup/WAL/Snapshot) during drills |
| NFR-B5 | RTO | ≤ 1 hour (full platform in a fresh cluster) | DR drill: destroy the cluster, execute the documented restore runbook, time to "13 MVP flows pass in the restored environment"; target < 60 min, hard ceiling 2 h |
| NFR-B6 | Backup integrity | Every backup verified (restore test) within 24 h of creation; unverified backups are flagged in the backup dashboard | Backup job emits backup.verified metric; alert if stale |
| NFR-B7 | Secrets & config | All config reconstructable from repo + secret store (IaC: terraform state encrypted); a fresh deployment from git + secret store works | "Clone-to-cluster" drill (part of the DR drill) |
7. Observability
7.1 Signals
| Signal | Standard | Details |
|---|---|---|
| Traces | OpenTelemetry (OTLP/HTTP) | One service name per app (api, worker, realtime, web); spans on: HTTP handlers, services (domain ops), repositories (DB queries), queue jobs, WS operations, external calls (S3, OpenSearch, AI providers); sampling: 100 % errors, 10 % others (configurable) |
| Metrics | Prometheus (pull) via otel bridge | Histograms: http.server.request.duration (by route, code), db.query.duration (by op), queue.process.duration (by queue), ws.message.duration, search.query.duration, ai.request.duration, editor.interaction.duration (client → RUM); gauges: queue.depth, ws.connections, bus.consumer.lag, redis.pool.checked_out, pg.pool.checked_out; counters: events.published, webhooks.delivered (by status), audit.events (by action), files.uploaded (by status) |
| Logs | Structured JSON (pino) → Loki | Fields: ts, level, msg, service, request_id, user_id?, workspace_id?, route?, duration_ms?, error?; no secrets, no PII (scrubber); correlation via request_id (HTTP) and doc_id (realtime) |
7.2 Required dashboards (Grafana)
- Overview (SLO burn, error rate, p95 latency, throughput, WS connections, queue depths).
- API (per-route latency/error, rate-limit hits, 4xx/5xx breakdown).
- Realtime (connections by workspace/room, message rate, p95 update latency, dropped messages, Redis update-log size).
- Queues (BullMQ per queue: waiting/active/completed/failed, processing time, DLQ depth).
- Search (query latency, index job throughput, consumer lag, OpenSearch cluster health).
- Storage (upload rate, pipeline stage durations, scan verdicts, storage used by workspace).
- AI (tokens in/out by workspace, cost estimate, provider latency/error, budget burn).
- Capacity (table row counts, DB connection pools, WS memory, disk).
7.3 Alert rules (initial set — full list in deployment.md §9)
| Alert | Condition | Severity |
|---|---|---|
| API error rate | > 1 % 5xx over 5 min | page (critical) |
| API latency | p95 > 500 ms over 10 min | warn |
| SLO burn | error budget 50 % consumed | warn → page at 90 % |
| Queue backlog | depth > 1 000 or oldest job > 5 min (search index), > 30 min (webhook) | warn |
| DLQ | any new dead-letter (automation/webhook) | warn |
| WS saturation | connections > 8 000 per realtime pod, or p95 update > 400 ms | warn |
| Consumer lag | events older than 30 s unprocessed | warn |
| Disk | > 80 % on any stateful volume | warn → critical 90 % |
| Backup | last successful backup/verify older than schedule × 2 | critical |
| Certificate | TLS expiry < 14 d | warn |
| Realtime unreachable | health check failing 3× | critical |
| Virus detections | > 0 in an hour (anomaly) | warn |
| AI budget | workspace budget 90 % consumed | warn (to workspace admin) |
8. Offline behavior (NFR-O series)
| ID | Requirement | Target / rule |
|---|---|---|
| NFR-O1 | Offline editing | Full block editing works with network down; every keystroke applies locally < 50 ms (same budget as online — Yjs local ops are synchronous) |
| NFR-O2 | Sync durability | Offline edits survive browser restart (IndexedDB-persisted Yjs doc + mutation queue); no edit loss on crash (queue written before ack) |
| NFR-O3 | Reconnect | Within 30 s of connectivity return: full sync complete for ≤ 50 pending edits; user sees "Syncing…" then "All changes saved" |
| NFR-O4 | Conflict handling | CRDT: concurrent text edits merge automatically (0 user action); non-CRDT fields (title, property values): last-writer-wins + toast; deleted-record conflict: conflict dialog (keep my edit as new record / discard) |
| NFR-O5 | Scope | v1 offline = editor + read cache + mutation queue. Offline database queries are not supported (cached views read-only); offline file uploads queue and start on reconnect (NFR-O6) |
| NFR-O6 | Queued uploads | Files dropped offline hold a local preview (blob), upload starts on reconnect; user-visible state "Waiting for connection" |
9. Maintainability
| ID | Requirement | Target | Measurement |
|---|---|---|---|
| NFR-M1 | Test coverage — engine packages | ≥ 90 % (line + branch) for packages/database-engine, packages/permissions, packages/editor (positions + codec), packages/shared (schemas) |
Vitest coverage gate in CI: build fails below threshold (coverageThreshold) |
| NFR-M2 | Test coverage — apps | ≥ 70 % line for apps/api (Jest), ≥ 60 % for apps/web components, 60 % apps/worker, 60 % apps/realtime |
Jest/Vitest coverage gate |
| NFR-M3 | Engine coverage — formula | 100 % of the 16 functions covered + fuzz suite (10 000 random valid/invalid expressions, 0 crashes, 0 hangs) | dedicated fuzz job (nightly) |
| NFR-M4 | Type safety | strict: true everywhere; 0 any outside explicit // type-escape: <reason> comments (lint rule counts them, ratchet down only) |
typecheck + eslint gate |
| NFR-M5 | Dependency hygiene | 0 unreviewed new deps (PR justification required — coding-standards §10); majors bumped with a changelog entry + test pass; SCA: 0 critical | PR template + Dependabot/renovate PRs reviewed as majors |
| NFR-M6 | Module boundaries | Import-boundary lint rules (coding-standards §3) pass with 0 exceptions file | eslint import/no-restricted-paths in CI |
| NFR-M7 | Build time | Monorepo full CI (lint → typecheck → unit → build) < 15 min cold | Turbo pipeline timing in CI; regression > 20 % flagged |
| NFR-M8 | Docs currency | Every merged feature PR updates: task file status, API docs (if API changed), and at least one of {product, ux, security, database} docs when behavior changed | PR checklist (required tick) + CI check that docs/ or CHANGELOG.md touched (informational) |
10. Browser & platform support
| Platform | Supported (v1) | Notes |
|---|---|---|
| Chrome / Edge | Latest 2 major (desktop) | Primary test target |
| Firefox | Latest 2 major (desktop) | |
| Safari | Latest 2 major (desktop + iPadOS) | |
| Mobile browsers | iOS Safari 16+, Android Chrome 115+ | Responsive web (no app shell features: no offline on iOS Safari without PWA install — PWA manifest ships, install is optional) |
| Node (server) | ≥ 20 LTS | CI matrix: 20 + 22 |
| Dropped | IE 11, Safari 14-, Firefox ESR | Explicitly unsupported; graceful "please update" screen via feature detection (not UA sniffing) |
E2E matrix (Playwright): chromium + firefox + webkit on ubuntu-latest, desktop viewport 1440×900 + mobile 390×844 (phase-gated per task).
11. Internationalization readiness
| ID | Requirement | Target |
|---|---|---|
| NFR-I1 | String extraction | 100 % of user-visible strings pass through the i18n wrapper (t("key")) — enforced by an eslint rule in apps/web (no raw string literals in JSX text); keys namespaced domain.area.item |
| NFR-I2 | Dates/numbers | All rendering via Intl.DateTimeFormat/Intl.NumberFormat with the user's locale + timezone (stored on users); database date properties always stored UTC, rendered in user TZ |
| NFR-I3 | Plurals & gender | ICU MessageFormat strings (plural categories); no hardcoded English in code paths (error codes stable, messages translatable) |
| NFR-I4 | Layout | No truncation of translated strings below 120 % length (component text overflow rules); RTL: not in v1 (logical CSS properties used so RTL is a token flip, documented) |
| NFR-I5 | Search | OpenSearch analyzers: en default + standard fallback; UI search is locale-aware but cross-language matching is not promised in v1 |
| NFR-I6 | Locale data | Locale picker in user settings (v1: en-US only functional; the pipeline exists, translation catalogs are post-v1 — PR-166 is readiness) |
12. Self-hostability (deployment posture)
| ID | Requirement | Target |
|---|---|---|
| NFR-H1 | One-command local | pnpm install && pnpm docker:up && pnpm db:migrate && pnpm db:seed && pnpm dev from a clean clone works (Phase 0 exit criterion, re-verified in CI on every merge) |
| NFR-H2 | Production from git | A fresh Linux server (8 GB) can run the full platform from docker compose (all apps + infra) or Kubernetes manifests, with config only via env vars / secret store |
| NFR-H3 | Resource floor | Minimum supported production sizing: api × 2 (1 vCPU/1 GB each), realtime × 2 (1 vCPU/2 GB), worker × 2 (1 vCPU/2 GB), web × 2 (1 vCPU/1 GB), Postgres 4 vCPU/16 GB/100 GB, Redis 2 GB, OpenSearch 1 node 4 vCPU/16 GB, MinIO 100 GB — documented in deployment.md §2 |
| NFR-H4 | Upgrades | Zero-downtime rolling deploy (k8s) or docker compose pull && up -d with migration step; backward-compatible API (v1 contract); upgrade runbook < 30 min (NFR-B5 sibling) |
| NFR-H5 | Telemetry opt-out | All telemetry (otel) disabled by default in self-host; enabling is a flag (OTEL_ENABLED), no data leaves the deployment |
13. NFR → verification map (summary)
| NFR group | Primary proof | Gate |
|---|---|---|
| NFR-1…12 (perf) | tests/performance + e2e perf fixtures |
Phase 15 exit + nightly |
| NFR-S1…S6 | Load profiles (Artillery/k6) | Phase 15 exit |
| NFR-R1…R7 | SLO dashboards + chaos integration tests | Ongoing (staging SLO burn) |
| NFR-SE1…SE8 | Security suites + ASVS audit + SCA/ZAP | Phase 15 + every PR |
| NFR-P1…P6 | Cross-tenant suite + retention integration tests | Every PR (cross-tenant) / Phase 14 |
| NFR-B1…B7 | DR drills (runbook) | Quarterly (prod) |
| NFR-O1…O6 | Offline e2e (Playwright network emulation) | Phase 7 |
| NFR-M1…M8 | CI coverage/lint/build gates | Every PR |
| NFR-I1…I6 | eslint i18n rule + locale integration tests | Every PR |
| NFR-H1…H5 | Phase 0 CI matrix + clone-to-cluster drill | Every merge / quarterly |