Phase 15 — Production Hardening

Purpose: Task specs for Phase 15 — the final phase before v1.0: the OWASP ASVS L2 security audit + hardening, and performance/observability/backup-DR + the v1.0 release with the final full-flow E2E.

Phase exit criteria (ROADMAP.md, Phase 15):

• All perf targets met in the tests/performance suite.
• Restore drill documented + executed.
• Dashboards + alerts live in staging.

Task range: TASK-0134 … TASK-0135 (2 tasks — this agent owns exactly TASK-0134..TASK-0135; TASK-0096..0120 are owned by other agents).

Section references: performance targets are PRODUCT-SPEC §26 and docs/product/non-functional-requirements.md; the metric list cited by the brief as "§62" does not exist in the 28-section PRODUCT-SPEC.md — the metric set is defined by docs/product/non-functional-requirements.md and this file's TASK-0135 metrics requirement (recorded conflict per AGENTS rule 29).

Task Table

ID	Title	Estimate
TASK-0134	Security audit + hardening (OWASP ASVS L2)	L
TASK-0135	Performance, observability, backup/DR + release v1.0	L

Staging Environment (TASK-0135 precondition)

The staging cluster is where the hard perf gates, dashboards, backup drill, and final E2E run. Specification (provisioned in infrastructure/kubernetes/ by this task's infra work; the cluster itself is an environment, not code):

Component	Staging spec
Nodes	3× (4 vCPU / 16 GB) worker + 1× control (managed); k8s 1.29+
factory-api	3 replicas, 1000m/512Mi requests (2000m/1Gi limits), HPA 3–10 on CPU 70%
factory-web	2 replicas, 500m/512Mi, behind the same ingress
factory-worker	2 replicas per queue group (ai/webhooks/maintenance/email pools), 1000m/1Gi
factory-realtime	2 replicas, 1000m/1Gi, max 5k ws conns per pod (the 10k load test spans both)
PostgreSQL 16	primary (8 vCPU/32 GB, 500 GB gp3) + 1 read replica (4 vCPU/16 GB, 250 GB); PgBouncer 2 replicas in front (transaction mode)
OpenSearch 2.x	3-node cluster, 10 GB heap, 200 GB; indices per workspace-tenant pattern
Redis 7	single node with AOF (dev-grade acceptable for staging DR — documented), 16 GB
MinIO	4-node erasure set, 500 GB, S3 versioning on files + backups
Observability stack	OTel collectors (daemonset + deployment), Prometheus (2 h local + S3 remote-write optional), Grafana, Alertmanager, Loki optional (pino → stdout is primary; Loki scrape of pod logs is the staging convenience path)
S3 (AWS)	factory-staging-backups (versioned, private, KMS), factory-staging-files, factory-staging-exports
CI	self-hosted runner (4 vCPU) for the local editor perf test; staging deploy token; ghcr org robot for image pushes

Environment config lives in infrastructure/kubernetes/values.staging.yaml (the single values file per environment, the convention for the release). The staging AI provider is AI_PROVIDER=mock (TASK-0126's scripted mock — the final-flow AI step is deterministic and cost-free in staging).

Phase Completion Checklist

• [ ] Dependency audit (npm audit high+ fail CI), gitleaks secrets scan (repo + artifacts), and license check all wired into the CI contract and green.
• [ ] Auth-path review complete: token handling, refresh reuse detection, session fixation, cookie flags — each with a passing test (report in docs/security/audit-report.md).
• [ ] Permission fuzzing: property-based (random user/role/grants/resource) asserting the permission engine matches a brute-force oracle, 10k iterations in CI, green.
• [ ] CORS/CSRF/CSP verified for every crawled route; file-upload abuse suite (polyglot, zip slip, oversized, slow-loris) green; formula-engine fuzz (10k expressions) with the 100k-op evaluation limit + 50 ms timeout guard implemented and verified.
• [ ] Rate-limit verification (auth, API, webhooks, forms) green; secret scan of Docker images/artifacts clean; every finding tracked to a fix + regression test in the audit report.
• [ ] docs/security/audit-report.md contains the signed ASVS L2 checklist (L2 controls mapped: pass / N/A with justification / remediated).
• [ ] Profiling of the top-5 hot paths done (page tree load, block batch save, db query, search, record open); Redis caching live (page tree with version-key invalidation, permission decisions TTL 60 s + event invalidation, search results short TTL, db column metadata); missing indexes added via migration from EXPLAIN analysis; Prisma query-shape fixes (narrow selects, N+1 elimination); PgBouncer + read-replica configuration in infrastructure/kubernetes/.
• [ ] tests/performance k6 suites pass the §26 criteria in CI (report-only in dev, hard in staging): API 1k rps p95 < 300 ms; realtime 10k conns / 200 msg/s p95 < 200 ms; database 100k records table p95 < 500 ms + 100 rps record create; editor E2E interaction p95 < 50 ms + block move < 16 ms (local).
• [ ] OpenTelemetry in all 4 apps (traces, metrics, pino logs with requestId correlation); Prometheus scrape + recording rules; Grafana dashboards (API, Realtime, Workers/Queues, Search, AI, Infra) as code in infrastructure/observability/; alert rules (error rate, p95, queue depth, WS drop, disk, cert) → webhook + email.
• [ ] Backup/DR: pg PITR + daily versioned S3 base backups, OpenSearch snapshot repo (RPO ≤ 5 min / RTO ≤ 1 h); restore runbook in docs/development/deployment.md; executed drill to a scratch namespace with 3 verified queries, results documented.
• [ ] v1.0 released: CHANGELOG finalized, tag v1.0.0, ghcr images, staging manifests + smoke E2E, docs completeness check (every module README, every endpoint in OpenAPI + docs/api), self-hosting admin guide (docker + k8s), support matrix page.
• [ ] Final full-flow E2E green in staging: register → workspace → pages → editor → databases → relations → search → files → forms → automation → AI → API key → webhook.

---

TASK-0134

Title

Security audit + hardening (OWASP ASVS L2)

Objective

Run a structured OWASP ASVS Level 2 audit across the platform — dependencies, secrets, auth paths, permissions (property-based fuzzing vs a brute-force oracle), transport/headers (CORS/CSRF/CSP), file-upload abuse, formula-engine fuzzing, and rate-limit verification — and harden every finding with a fix + regression test, closing with a signed ASVS L2 checklist in docs/security/audit-report.md.

Background

Phases 0–14 shipped the full product surface: auth with rotating refresh tokens (Phase 1), RBAC + resource permissions via packages/permissions (Phase 2/3), files with presigned uploads + virus scan (Phase 9), the safe formula AST evaluator in packages/database-engine (Phase 6), public API keys/OAuth/webhooks (Phase 13), SSO/SCIM (Phase 14). docs/security/security-architecture.md is the threat-model home; .env.example documents MASTER_ENCRYPTION_KEY and the auth secrets. CI already runs lint → typecheck → unit → integration → build → security scan → E2E (Phase 0) — this task makes the security scan real (npm audit, gitleaks, license) and adds the audit suites. The permission engine's brute-force oracle: the spec is small enough (roles Owner/Admin/Member/Guest × resource visibility × grants) that an exhaustive reference implementation in the test can enumerate every (subject, resource) pair and compute the expected decision by direct rule evaluation — the engine under test must match on all 10k random cases.

Requirements

• Dependency audit (CI, tests/security/deps.test.ts + CI job): pnpm audit --audit-level=high — any high/critical in the runtime dependency tree fails CI (dev-only reported, not failing); pnpm licenses check against the allowlist (docs/security/license-allowlist.md — MIT/Apache-2.0/BSD/ISC/0BSD/BlueOak-1.0.0 + documented exceptions); a lockfile drift check (pnpm install --frozen-lockfile already in CI — asserted explicit). New deps added by fixes in this task require a justification entry in the allowlist file (AGENTS rule 17).
• Secrets scan: gitleaks in CI over the full git history (not just the worktree) + over build artifacts (dist/, Docker image layers via a scan step that extracts and scans apps/*/dist and the final image manifest); baseline file for any pre-existing finding (each baseline entry = a tracked remediation task, see remediation tracking). .env.example and test fixtures with fk_test_…/whsec_test_… values are allowlisted by pattern, not by file (test-value syntax is reserved: *_test_*).
• Auth path review (each item → written finding + passing test): (1) Token handling — access JWT claims minimal (sub, workspaceId, authMethod, iat/exp 15 min), no sensitive claims; JWT alg pinned (HS256/RS256 as configured — none rejected; a forged alg:none token test fails auth); (2) Refresh reuse — Phase 1 rotation: reuse of an already-rotated refresh token revokes the family (test: rotate, reuse old → family dead, all sessions of the family 401); (3) Session fixation — session ID/cookie changes on privilege change (login, workspace switch with different role) (test: pre-login cookie ≠ post-login cookie); (4) Cookie flags — Secure, HttpOnly, SameSite=Lax (Strict evaluated; decision recorded: Lax for the app cookie, Strict for the CSRF token cookie if separate), Path scoped; test asserts flags on every Set-Cookie in the E2E auth flow; CSRF: double-submit token on all state-changing web routes (Phase 1/4 — verify coverage by a route-walk test: every POST/PATCH/DELETE web route except the CSRF-token issue + OAuth/SAML callbacks (which use state/nonce instead) requires the token; API-key/OAuth-token/incoming-webhook routes exempt by credential type, documented).
• Permission fuzzing (tests/security/permission-fuzz.test.ts): property-based (fast-check) generating random worlds: 1–8 users with random roles, 1–10 resources (pages with parents, databases, records, files) with random share/permission rows (the Permission + Share models), 1 random action (read/write/share/manage); the engine decision must equal the brute-force oracle (a second, deliberately naive, exhaustive-enum implementation in tests/security/oracle.ts — independent code path, e.g. a direct SQL-free walk of the same rules) for 10,000 iterations (seed fixed for CI reproducibility + a --fuzz-seed random mode in nightly CI). Any mismatch: the fuzzer shrinks to a minimal counterexample (asserted output on failure) and the case is added to the regression suite.
• CORS/CSRF/CSP verification: a crawler test (tests/security/headers.test.ts) boots the API + web app and crawls every route (router enumeration, not guessing): CORS — only the configured origin(s) allowed; no * with credentials; no sensitive headers exposed cross-origin (test with a disallowed origin: no Set-Cookie echo, no auth headers readable); CSRF — the coverage walk above; CSP — every HTML route returns a CSP with default-src 'self', no unsafe-eval, script-src restricted (the swagger-ui bundle needs a hashed/CDN decision — record: swagger-ui runs from same-origin static assets, CSP nonce for its inline scripts; if swagger-ui can't run under the CSP, serve it from a dedicated path with a scoped CSP — tested), frame-ancestors 'self' (clickjacking), plus X-Content-Type-Options: nosniff, Referrer-Policy: strict-origin-when-cross-origin, Permissions-Policy minimal.
• File upload abuse (tests/security/upload-abuse.test.ts): polyglot files (a PDF that is also valid JS / image with script — assert the content-type is sniffed (magic bytes, not the client's header) and the file is served with Content-Type from the stored MIME + Content-Disposition: attachment for non-renderable types, no X-Content-Type-Options downgrade); zip slip — a zip whose entries contain ../../etc/passwd (assert the pipeline rejects path-traversal entries — if the zip pipeline doesn't exist in v1, the test asserts the validator exists for file blocks that reference zip paths, and the finding is recorded as N/A-with-guard for v1); oversized — a 501 MB upload (plan cap 500) rejected at presign (413) and mid-stream abort works; slow-loris — a drip-feed upload (1 byte / 2 s) is terminated by the gateway/worker timeout (test with a shortened 5 s timeout) and the partial file is purged (no orphan in MinIO + File row status FAILED).
• Formula engine fuzz (tests/security/formula-fuzz.test.ts): 10k random expressions (generator: the engine's own grammar with random nesting depth 1–40, random property names incl. nonexistent, random function trees over the Phase 6 function set) — assertions: no crash, no unbounded time, no memory blowup: every eval completes or returns a typed error. Guard implementation (this task implements if missing): the AST evaluator in packages/database-engine must enforce (a) an evaluation step/operation limit of 100k ops (counter incremented per node visit; exceeding → FORMULA_TIMEOUT typed error) and (b) a wall-clock timeout of 50 ms (cooperative: checked in the eval loop between node steps — deterministic for the test; plus a hard AbortController-style cap documented). A pathological expression (e.g. deeply nested concat of self-referencing props — the generator includes a worst-case family) must hit the op limit, not hang. Re-run of the Phase 6 fuzz suite (100% function coverage) stays green.
• Rate-limit verification (tests/security/rate-limits.test.ts): for each bucket — auth endpoints (login/register/reset: per-IP + per-account), API (user 600/min, per-key rateLimit, IP 120/min), webhooks (delivery worker: no client-facing limit but the incoming webhook endpoint has a per-workspace limit — set 60/min if missing, test it), forms (public submission: per-IP + per-form, e.g. 20/min — set if missing, test it) — assert the 429 + Retry-After + X-RateLimit-* contract, the sliding-window boundary (burst at window edge), and that limits are per the documented values (docs/api/authentication.md §11 + the two new buckets documented there).
• Secret scan of images/artifacts: the CI security job scans the built Docker images (each apps/* image + the worker) with a secrets scanner over the extracted rootfs (gitleaks --source over the extracted layers) + asserts no .env-family file, no private key file, no node_modules with known-vulnerable package versions (re-assert of the dependency audit at image level).
• Remediation tracking: every finding (from all suites above + the manual review below) gets an entry in docs/security/audit-report.md: {id, asvs-ref, severity, status: fixed|mitigated|accepted, fix-task (commit/PR), regression-test (path)}; a CI check (a small script in the report's companion test tests/security/audit-report.test.ts) asserts the report exists, every finding has a status + regression-test path, and the test path exists and passes.
• Manual ASVS L2 review (written, with test evidence per control where practical): injection (SQLi — Prisma parameterized, a test asserts no raw queryRaw with string interpolation exists in apps/api (grep-based CI check, allowlist for the migration runner); XSS — the content sanitizer test matrix from Phase 4 re-run + server-rendered HTML escaping check; SSRF — the TASK-0130 delivery-URL guard + webhook/OAuth redirect_uri allowlist re-verified; crypto — MASTER_ENCRYPTION_KEY usage: all AES-GCM, random IV, key from env only; error handling — canonical envelope on all 5xx (no stack leaks: a forced 500 test asserts no stack/SQL in the body); logging — no PII/secrets in logs (a log-scrubber test: seed a request with a secret-looking header, assert it never appears in the captured stdout).

Acceptance Criteria

• pnpm audit --audit-level=high over the runtime tree returns zero high/critical (or each is an allowlisted exception with a justification + date); the CI job fails on a new high (demonstrated by a fixture test that adds a vulnerable dep in a scratch workspace and sees the job fail).
• gitleaks over full history + artifacts is clean (baseline entries all have linked remediation tasks); the image-scan step runs on all 5 images (api, web, worker, realtime, and the test image) and is clean.
• Auth review: the 4 finding groups each have ≥1 passing test (alg-pinning, refresh-reuse family revocation, fixation cookie change, cookie flags on every Set-Cookie in the E2E flow) + the CSRF route-walk test proves 100% coverage of state-changing web routes (exemptions documented).
• Permission fuzzing: 10k iterations green in CI (fixed seed) — the test fails (with a shrunk counterexample) if the engine and oracle disagree; a deliberately broken engine (fixture) is caught.
• Headers suite: every crawled route passes CORS (disallowed origin gets no credentials/sensitive headers), CSP (no unsafe-eval, frame-ancestors 'self'), nosniff + referrer-policy present; swagger-ui path verified under CSP.
• Upload abuse: polyglot served with stored-MIME + attachment disposition; zip-slip entries rejected (or N/A-with-guard recorded); 501 MB rejected 413 at presign; slow-loris (5 s test timeout) terminated + no orphan object (MinIO list assertion) + File.status=FAILED.
• Formula fuzz: 10k expressions, zero hangs/crashes; the pathological family hits the 100k-op limit or 50 ms timeout with FORMULA_TIMEOUT (asserted error type, not an unhandled rejection); Phase 6's function-coverage suite still 100% green.
• Rate limits: all buckets (auth, API user/key/IP, incoming webhook, public form) return the documented 429 contract at the documented values (missing limits for incoming-webhook/forms are implemented this task + documented).
• docs/security/audit-report.md: ASVS L2 checklist complete (every L2 control: pass / N-A-justified / remediated-with-link), the findings table has 100% of rows with status + existing regression-test path (the audit-report.test.ts CI check enforces this), and the report is signed (author + date + "verified against CI run <id>").

Backend Changes

• packages/database-engine/src/formula/evaluator.ts: op-limit counter (100k) + cooperative 50 ms timeout guard (if not already present — inspect first; Phase 6 may have shipped a limit under different values: then this task changes the values to 100k/50 ms and updates the constant to a named config EVAL_OP_LIMIT=100_000, EVAL_TIMEOUT_MS=50); new typed error FORMULA_TIMEOUT surfaced via the formula service (distinct from FORMULA_ERROR — syntax).
• Missing rate-limit buckets: incoming-webhook endpoint limiter (60/min/workspace) + public form submission limiter (20/min/IP+form) if absent; X-RateLimit-* headers on both.
• Cookie flag centralization (apps/api/src/common/cookies.ts + apps/web next.config): Secure/HttpOnly/SameSite/Path in one place (single source for the cookie-flags test).
• CSP: next.config + API HTML routes (consent screen, docs swagger) — nonce or scoped-path decision implemented; Content-Disposition/stored-MIME enforcement in the file download service (Phase 9 fix if the header comes from the client); zip-entry path validation in the file pipeline (or the documented guard); upload stream timeout + orphan purge in the file service (worker cleanup job for FAILED partials).
• Log scrubber middleware (apps/api/src/common/log-scrubber.ts): redacts Authorization, Cookie, X-Factory-Signature, and any header matching the secret patterns from structured logs; applied to all 4 apps.
• No other backend behavior changes — this task is hardening + tests; any behavior change is a documented finding fix.

Frontend Changes

• None functional. Only if the CSP forces it: remove/replace any inline style=/script usages in apps/web (the CSP test drives the list; expected small: a few inline styles → CSS modules, per the no-hardcoded-styles invariant).

Database Changes

• None (no schema changes — this is an audit/hardening task; if the upload orphan cleanup needs an index, e.g. @@index([status, createdAt]) on File, it lands as a migration here with the justification in the report).

API Changes

• Behavioral only: new 429 buckets (documented), stricter headers on all responses, FORMULA_TIMEOUT error code added to the canonical table (docs/api/authentication.md §9.1 — new code, extensible by design). No endpoint additions/removals.

Tests

• All suites in Requirements are the deliverable: tests/security/{deps,gitleaks-config,auth-paths,permission-fuzz,headers,upload-abuse,formula-fuzz,rate-limits,images,audit-report}.test.ts (or .test.mjs for the shell-driven image/scan steps) + tests/security/oracle.ts (the brute-force permission oracle) + fast-check generators; all wired into the CI contract (a new security CI stage after build, before E2E, per the Phase 0 pipeline).
• Regression: each finding fix has its named test (referenced from the report table — the audit-report.test.ts cross-check enforces path existence).

Security

• This task is the security work; its meta-rules: no fix may weaken an existing control (a control regression = a finding, not a fix); the brute-force oracle is test-only code (never imported by app code — an import-boundary lint rule added: tests/** must not be imported from apps/**/packages/**); fuzz seeds fixed for CI, random in nightly (both modes documented); the report is append-only for findings (status changes recorded with dates, deletions forbidden — the CI check forbids removing finding rows).

Documentation

• docs/security/audit-report.md (the deliverable: scope, methodology, ASVS L2 checklist, findings table, sign-off); docs/security/license-allowlist.md; docs/security/security-architecture.md updated with the new controls (op-limit values, cookie centralization, log scrubber, new rate buckets); docs/api/authentication.md §9.1 (FORMULA_TIMEOUT) + §11 (new buckets).

Dependencies

• Phases 0–14 complete (the full surface exists to audit), CI pipeline (Phase 0) with the new security stage, Docker build (Phase 0 infra), the formula engine (Phase 6), permission engine (Phase 2/3), file pipeline (Phase 9), API/webhook/form endpoints (Phase 13/10).

Definition of Done

• [x] All 10 test suites in place and green in CI (deps, secrets, auth paths, permission fuzz ×10k, headers/CORS/CSRF/CSP, upload abuse, formula fuzz ×10k with the implemented 100k-op/50 ms guard, rate limits incl. the two new buckets, image scans, report cross-check).
• [x] Every finding remediated (fix + regression test linked) or explicitly accepted with justification in the report.
• [x] docs/security/audit-report.md ASVS L2 checklist complete + signed; license allowlist + updated architecture doc.
• [x] CI contract runs the security stage; a new high vuln / leaked secret / engine-oracle mismatch each fail the build (demonstrated by fixture tests).
• [x] lint/typecheck/tests green; no app-code imports from tests/.

---

TASK-0135

Title

Performance + observability + backup/DR + release v1.0

Objective

Take the platform to production shape: profile and fix the top-5 hot paths (Redis caching, indexes, Prisma query shape, pooling/replica), make the tests/performance k6 suites pass the PRODUCT-SPEC §26 criteria in CI, ship OpenTelemetry + Prometheus + Grafana + alerting as code, build PITR-capable backup/DR with an executed restore drill, and cut the v1.0 release with the final full-flow E2E green in staging.

Background

The system is feature-complete (Phases 0–14): apps/web (Next.js 14), apps/api (NestJS modular monolith), apps/worker (BullMQ: automations, AI, webhooks, email, maintenance queues), apps/realtime (ws + Yjs), Prisma/PostgreSQL (models incl. Block, DatabaseRecord, AuditLog, SearchDocument), OpenSearch, MinIO, Redis. Performance targets: PRODUCT-SPEC §26 (initial load < 2 s, editor interaction < 50 ms, local block move < 16 ms, search < 500 ms p95, API p95 < 300 ms, realtime update < 200 ms p95, 100k-record databases) + the brief's load-test matrix (1k rps API, 10k WS conns, 100k-record table, editor E2E latency) — all to be enforced in CI via k6 (report-only in dev, hard in staging). infrastructure/kubernetes/ has per-app manifests (Phase 0); infrastructure/docker/ the dev compose (postgres, redis, minio, opensearch). docs/product/non-functional-requirements.md is the verification owner for these targets. The brief's "§62 metric list" does not exist (PRODUCT-SPEC has 28 sections) — the metric set is defined by that NFR doc + the metrics requirement below (conflict recorded in the file header).

Requirements

• Profiling + hot-path fixes: profile the top-5 paths with 0x (Node) + pg_stat_statements/EXPLAIN (ANALYZE, BUFFERS) on the representative queries: (1) page tree load (the sidebar + page children tree — the Phase 3 "< 2 s for 100k blocks" path), (2) block batch save (editor autosave flush), (3) database query (filter/sort/page over records), (4) search (OpenSearch round-trip + result shaping), (5) record open (record → page → blocks → properties assembly). For each: a before/after p95 in the k6 suite + the fix:
  • Redis caching (apps/api/src/common/cache.ts + per-module usage): page tree — key tree:{workspaceId}:{parentId}:{version} where version = the workspace's content version counter (a monotonically increasing Redis counter bumped on every block/page write in the workspace — version-key invalidation: reads key by version, writes bump the version; no stale reads across concurrent writes, tested with a concurrent-write race test); permission decisions — TTL 60 s + event invalidation (the permission-relevant domain events from packages/shared publish a cache-bust keyed by {subject, resource-ancestors} — a grant change busts the granter/grantee + the resource chain; TTL is the safety net, tested: bust event → next decision is fresh; no event → stale ≤ 60 s); search results — short TTL 5 s keyed by {workspaceId, query-hash, filters-hash, principalId} (only cache filtered results — permission-relevant); database column metadata (property list + view settings) — TTL 60 s + bust on property/view changes. Cache entries are structured (zod-validated on read — a corrupt entry is a cache miss, never a 500).
  • Indexes (from EXPLAIN analysis of the top-5 queries + the 100k-record suite): the candidates — blocks (pageId, position), database_records (databaseId, archivedAt), record_property (recordId), audit_logs (workspaceId, createdAt DESC), search_documents (workspaceId, type), file (workspaceId, createdAt) — add the ones EXPLAIN proves are missing (each index = one migration + a note in the report; no speculative indexes).
  • Prisma query shape: select narrow fields on the tree/record-open paths (no include of unused relations — a lint rule + the query-shape test asserts the record-open path issues ≤ 4 queries for a 50-block record page, down from the profiled baseline); N+1 elimination on block batch save (single createMany/updateMany per collection per flush — the profiled N+1 is fixed; test asserts query-count flat across batch size 10/100/1000 blocks).
  • Pooling + replica: infrastructure/kubernetes/ — PgBouncer (transaction mode) manifests + DATABASE_URL via pgbouncer for the API/worker pools (pool sizing: DATABASE_TIMEOUT, connectionLimit documented per app); read-replica: DATABASE_REPLICA_URL — Prisma multi-client setup (prisma/clients.ts): writes → primary, reads for search-indexing, tree loads, listing endpoints → replica (a routing helper db.read()/db.write(); the replica lags — the helper documents the staleness contract: replica reads are acceptable for tree/listing/search, never for permission decisions or optimistic-concurrency version checks (the If-Match path reads primary — tested: a write-then-read-your-write on the same request always hits primary)).
• Load-testing pass (tests/performance/, k6 — the suites):
  • tests/performance/api-load.js: 1k rps mixed read API (pages tree, record list, search) + 5% writes, 5 min → p95 < 300 ms (the §26 API target), error rate < 0.1%.
  • tests/performance/realtime-load.js: 10k concurrent ws connections across 100 rooms, 200 msg/s aggregate (block edits) → update p95 < 200 ms (client-observed ack latency), connection-establish p95 < 1 s, zero dropped messages on the 200 msg/s stream (Yjs op count assertion).
  • tests/performance/database-load.js: a 100k-record seeded database (the seed is a fixture job tests/performance/seed-100k.mjs) → table view load (first page + scroll pages) p95 < 500 ms + sustained 100 rps record create (the create path incl. page + record_property writes) at p95 < 500 ms for 2 min.
  • tests/performance/editor-local.mjs (local, headless browser — the §26 editor targets): editor interaction (type, block create) p95 < 50 ms; block move (drag simulation via the API-free editor driver) < 16 ms locally — run in CI on the standard runner, reported (dev) and hard (staging).
  • Enforcement: CI matrix — dev/main: report-only (k6 thresholds as soft warnings, results posted to the PR); staging: hard (threshold failures block the release pipeline — this is the §26 enforcement the brief requires).
• Observability (as code in infrastructure/observability/):
  • OpenTelemetry SDKs in all 4 apps (packages/observability/src/ — new shared package: OTel SDK init, pino-otel bridge, resource attributes): service names factory-web, factory-api, factory-worker, factory-realtime. Traces: spans per route (api/realtime ws-room ops), per Prisma query (via the Prisma instrumentation — query text redacted to the operation shape), per Redis command, per BullMQ job (job name + queue as span attributes), per ws room operation (join/leave/broadcast/presence); attributes on every span: workspaceId (when resolvable — absent, not "unknown", when genuinely unresolvable at auth), requestId, queue (worker jobs). Trace context propagates: API → worker job (job metadata carries the W3C traceparent — the dispatcher stores it, the processor resumes the span), API → OpenSearch/MinIO (HTTP propagation), web → api (fetch headers). Metrics (Prometheus format, the NFR doc's list — the brief's "§62" set): http_request_duration_seconds histogram + http_request_count (per route, method, status — route label is the pattern, not the id), db_query_duration_seconds histogram (per model/operation), redis_command_duration_seconds histogram, queue_depth gauge + queue_job_duration_seconds histogram (per queue), ws_active_connections gauge + ws_messages_total counter (rate = messages/s), search_query_duration_seconds histogram (BM25 vs kNN labeled), ai_tokens_total counter (in/out labeled per model) + ai_request_duration_seconds histogram, storage_ops_total + storage_op_duration_seconds (MinIO), error_total counter (per service, per error code). Logs: pino → stdout (JSON) with requestId on every line (the correlation id from X-Request-Id/generated), workspaceId when known, level policy (error=always, warn=always, info=sampled 100% in staging); no secrets (the TASK-0134 scrubber).
  • Prometheus: infrastructure/observability/prometheus/scrape.yaml (all 4 apps /metrics, worker job metrics, node exporter) + recording rules (p95 per route-group, error rate 5 m, queue-depth per queue, ws connections).
  • Grafana dashboards (JSON in infrastructure/observability/grafana/, provisioned by the stack): API Overview (rps, p50/p95/p99 per group, error rate, by status), Realtime (active connections, messages/s, room counts, update latency), Workers/Queues (depth per queue, job duration p95, DLQ count, retry rate), Search (query duration by type, index lag gauge), AI (tokens/min by provider/model, request latency, budget-approaching gauge from the usage rollup), Infra (CPU/mem per pod, PG connections vs pool, disk, Redis hit rate, OpenSearch cluster health). Panels cover every metric in the metric list above (a dashboard-completeness test: every emitted metric name appears in at least one panel or is explicitly listed as unpaneld in infrastructure/observability/README.md).
  • Alert rules (infrastructure/observability/prometheus/alerts.yaml): error_rate > 1% (5 m window), api_p95 > 500 ms (10 m), queue_depth > 10k (any queue, 5 m), ws_connections_drop > 30% (15 m vs 1 h baseline), disk_usage > 80%, cert_expiry < 14 d (the ingress cert checker) → alertmanager → webhook (the platform's own incoming-webhook automation — self-dogfooding: an automation that notifies the admin channel) + email (nodemailer processor, the existing email queue).
  • Staging deploy: the k8s manifests include the otel collectors + prometheus + grafana + alertmanager stacks (Helm values in infrastructure/kubernetes/); "live in staging" = the exit criterion: dashboards rendering real data + an alert firing in a drill (a chaos step: stop the worker → queue-depth alert within 5 m, documented).
• Backup/DR: PostgreSQL — PITR: WAL archiving to S3 (wal-g, infrastructure/kubernetes/postgres/ — the statefulset + archiver sidecar config, WAL_S3_BUCKET env) + daily base backups (wal-g backup-start, 05:00 UTC) to the same bucket, versioned (S3 versioning on); OpenSearch — snapshot repository to S3 (infrastructure/opensearch/ — repository config + daily snapshot cron via the worker or a k8s CronJob, 05:30 UTC), index list = ai-chunks + the search index; MinIO — bucket versioning on for files (existing objects) + a note that object data is restorable from the DB's file metadata + re-upload (documented v1 position: MinIO is not snapshot-backed in v1 — RPO for file objects is the upload-queue replay window, recorded decision); Redis — no persistence in DR (rebuildable cache; documented). RPO ≤ 5 min (WAL ship interval 60 s, verified by the drill: data written at T, restored PITR to T+1 min, the row exists), RTO ≤ 1 h (target; the drill measures actual). Restore runbook (docs/development/deployment.md § Restore): step-by-step PITR (pick timestamp → scratch namespace restore-drill → restore → verify) + OpenSearch snapshot restore + MinIO recovery + the rollback procedure (cut traffic back). Executed drill (this task runs it, not just documents): restore a PITR point (5 min before the drill start) + an OpenSearch snapshot into the scratch namespace, run the verification queries (3, defined in tests/security/... no — tests/integration/drill/verify-restore.mjs: (1) a page created 10 min before the PITR target is present with its blocks, (2) a record created 2 min before is present, (3) a search query returns the seeded document), record actual RTO/RPO + results in docs/development/deployment.md (the executed-drill section: date, timestamps, measurements, deviations).
• Release v1.0: CHANGELOG.md finalized (Unreleased → 1.0.0, per-module entries from the phase checklists); tag v1.0.0 (annotated, on main, after the full-flow E2E is green in staging — the tag is the last step); Docker images built + pushed to ghcr (ghcr.io/<org>/factory-{api,web,worker,realtime}:v1.0.0 + latest), digests recorded in the release notes; k8s manifests applied to staging (the release pipeline: infrastructure/kubernetes/ → apply → smoke E2E = the Phase 0 smoke suite: login + page create + search, < 2 min); runbooks linked (deployment, restore, SSO setup, on-call: docs/development/ — each linked from the release notes + the README "Operations" section); docs completeness check (a CI script tests/security/... no — scripts/docs-completeness.mjs wired into the release pipeline: (a) every package/app has a README.md (list asserted against the workspace globs), (b) every live public route is in docs/api/openapi.yaml AND referenced by at least one docs/api/*.md file (reuses the TASK-0127 contract test + a docs-coverage assertion), (c) every ROADMAP phase box is checked — fails until Phase 15's own boxes are ticked last); self-hosting admin guide (docs/development/deployment.md § Self-hosting + a new docs/development/admin-guide.md): one-command docker path (docker compose up from infrastructure/docker/ with the full stack incl. observability, the .env checklist) and the k8s path (Helm/manifests, the values file per environment), storage sizing table, backup configuration (the PITR env), upgrade path (migrations run by the api init container; the rollback = the previous tag); support matrix + supported versions page (docs/support-matrix.md: Node 20 LTS, PG 15/16, OpenSearch 2.x, Redis 7, browsers per the Phase 0 policy, supported-versions table with EOL dates; linked from README).
• E2E final (staging, the closing gate): the full MVP+ flow in one Playwright run (tests/e2e/final-flow.spec.ts, runs against the staging URL in CI): register → verify email (test mailcatcher in staging) → create workspace → create page → editor (type 5 blocks, drag one) → create database + 3 records → add a relation + a formula → search finds the page → upload a file (10 MB) → create a form + submit it publicly → create an automation (record-created → notify) and trigger it → AI (Ask AI answer streams with a citation — the staging AI provider is the mock, AI_PROVIDER=mock) → create an API key → curl-style create page via the API key (fetch from the test) → webhook delivery received + signature verified → log out. All green = release authorized.

Release pipeline specification

One CI pipeline (infra/pipeline/release.yml, the Phase 0 pipeline extended), stages in strict order — a stage cannot start on a red predecessor, and the tag step runs only at the end:

#	Stage	What it does	Gate
1	install+lint+typecheck	pnpm frozen install, ESLint, tsc strict	hard
2	unit	Vitest (packages) + Jest (api/worker)	hard
3	integration	tests/integration (Postgres/Redis/OpenSearch/MinIO via compose services)	hard
4	security	TASK-0134's 10 suites (deps, gitleaks, auth paths, permission fuzz, headers, upload abuse, formula fuzz, rate limits, image scans, report cross-check)	hard
5	build	all apps + packages built; Docker images built (not yet pushed)	hard
6	deploy-staging	values.staging.yaml → k8s apply (api, web, worker, realtime, observability, pgbouncer, replica)	hard (apply + readiness)
7	smoke	Phase 0 smoke suite against staging (login + page create + search), < 2 min	hard
8	performance	the 4 k6/local suites, hard thresholds (the dev run of the same suites in PRs is report-only)	hard
9	backup-health	last base backup < 26 h, last WAL ship < 10 min, OpenSearch snapshot < 26 h	hard
10	final-e2e	tests/e2e/final-flow.spec.ts (13 steps) in staging	hard
11	docs-completeness	scripts/docs-completeness.mjs (READMEs, OpenAPI+docs coverage, ROADMAP boxes)	hard
12	push+tag	push ghcr images (v1.0.0 + latest), create annotated tag v1.0.0 on main, publish release notes (digests, drill results, E2E run link, runbook links)	hard (tag is the last act)

Rules: stages 8/10 read staging after stage 6's rollout is Ready (a rollout wait step, 10 min budget); a failure at any stage pages the on-call channel (the Alertmanager webhook from the observability work — self-dogfood); the pipeline is re-runnable per stage (--from-stage), and the tag step is idempotent (re-run pushes the same images/digests, never moves the tag). The dev/PR pipeline runs stages 1–5 + a report-only performance stage + E2E against compose (not staging) — the staging cluster is release-pipeline-only, so release load never corrupts PR feedback timing.

k6 scenario matrices (the four suites, exact shape)

Suite	Scenarios (vusers)	Profile	Thresholds (staging-hard)
api-load.js	tree-load 400 (GET page tree, 200 distinct pages), record-list 300 (GET database pages, 5 distinct 100k-adjacent DBs), search 200 (POST /search, 100 distinct queries), record-create 100 (POST records, 5% of traffic)	5 min ramp 0→1k rps (constant-arrival-rate), hold 3 min, ramp down 1 min	overall p95 < 300 ms; per-scenario p95 < 300 ms; http_req_failed < 0.1%; p99 < 800 ms (soft, reported)
realtime-load.js	connect 10k over 5 min (100 rooms × 100 conns, jittered join), edit all 10k sending block-edit ops, 200 msg/s aggregate (rate-limited per-conn), presence 1k (join/leave churn 10/min)	15 min steady state	update p95 (client-observed op-ack) < 200 ms; connect p95 < 1 s; zero lost Yjs ops (final doc state hash equals the sequential-replay hash); disconnects < 0.5% (excl. scripted)
database-load.js	table-first 50 (table view first page on the seeded 100k DB, fresh cache), table-scroll 100 (pages 2..50), record-open 50 (record → page → blocks), record-create 100 (sustained 100 rps creates)	2 min table-*, 2 min record-create, 1 min record-open	table view p95 < 500 ms; record-open p95 < 500 ms; creates p95 < 500 ms + create rate ≥ 100 rps sustained (no > 5 s stalls); error rate < 0.1%
editor-local.mjs	headless Chromium on the local runner: type 500 key-insertions across 10 blocks, create-block 100 (Enter/slash), move-block 200 drag-simulations (the editor's programmatic move API — the real DnD path), render frame-time sampling during the above	single browser instance, 10 min	interaction p95 < 50 ms (input→commit); move < 16 ms (the local §26 target, p95 and p99 both asserted); no frame > 100 ms during move-block

The seed fixture seed-100k.mjs builds, per staging run: one 100k-record database (5 properties incl. 1 date, 1 select), one 10k-block page (for tree-load), 200 pages (for search/tree), committed to the DB directly (Prisma createMany in 1k batches, ~3 min wall clock) — it is a test-data job, run by the pipeline stage 8 precondition, and is idempotent (re-runs are no-ops by marker table test_seeds).

Alert rules (the six, exact expressions)

Alert	Expr (Prometheus)	For	Severity	Route
ApiErrorRateHigh	sum(rate(http_request_count{status=~"5.."}[5m])) / sum(rate(http_request_count[5m])) > 0.01	5m	page	webhook (automation) + email
ApiP95Slow	histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[10m])) by (le, route_group)) > 0.5	10m	page	webhook + email
QueueDepthHigh	queue_depth > 10000 (any queue)	5m	page	webhook + email
WsConnectionsDrop	ws_active_connections < 0.7 * avg_over_time(ws_active_connections[1h])	15m	page	webhook + email
DiskUsageHigh	node_filesystem_usage{mountpoint="/var/lib"} > 0.8	10m	warn	email (webhook on page-only)
CertExpiringSoon	probe_ssl_earliest_cert_expiry - time() < 14 * 86400 (blackbox probes on the ingress)	1h	page	webhook + email

Plus two operational (warn, email-only): BackupStale (stage-9 conditions as a continuous alert) and SearchIndexLagHigh (indexer lag > 30 s, the Phase 8 SLO). Alertmanager config: repeat_interval: 4h, the webhook receiver posts to the platform's own incoming-webhook automation ("Alert: <name> firing, <value>, <link>") — the dogfood — and the email receiver uses the nodemailer queue (Phase 7/11 processor).

Final E2E flow (the 13 steps, exact)

tests/e2e/final-flow.spec.ts, one project, sequential (state carries), against the staging URL, ~6 min budget:

#	Step	Assertion
1	Register (unique email per run, final-e2e-<runId>@test)	201 + verification email in staging mailcatcher (API poll)
2	Verify email (click link in the mailcatcher HTML)	session established, profile page
3	Create workspace "Release Co"	workspace shell + sidebar (Home/Inbox/Search/…)
4	Create page "Release Plan" under Home	page route renders, title in the doc
5	Editor: type 5 blocks, drag the 3rd below the 1st	order persists across reload; the move used the DnD path
6	Create database "Tasks" (3 props: title, status select, due date) + 3 records via UI	table renders 3 rows
7	Add relation Tasks→"People" db + a formula column (isOverdue)	rollup/relation links render; formula value correct on 1 row (asserted via API read)
8	Global search: query "Release Plan" finds the page; query a unique record title finds the record	result links resolve (click → correct page/record)
9	Upload a 10 MB file into the page (file block)	preview renders; file metadata via API (size, mime)
10	Create a public form (1 question) + submit it anonymously (context-isolated page)	record created in the mapped db; form.submitted fired (audit)
11	Create automation: trigger record.created(Tasks) → action notify (in-app) → trigger by creating a 4th record via API	automation run row SUCCEEDED; notification appears in Inbox
12	AI: open Ask AI on the page (staging AI_PROVIDER=mock), ask about the page	streamed answer (≥ 3 incremental DOM updates) + a citation chip linking to the page
13	API + webhook: create API key (UI reveal), fetch POST /api/v1/pages with the key, then the test receiver gets page.created ≤ 10 s with a valid X-Factory-Signature (recomputed in-test); then log out	201; signed delivery + HMAC verified; session gone (refresh 401)

Step 13's webhook receiver is the same in-process HTTP server pattern as TASK-0130's E2E; the run's artifacts (mailcatcher dump, receiver log, k6-less timing trace) are attached to the CI job for the release notes.

Acceptance Criteria

• The 5 hot paths each have a written before/after profile (docs/development/perf-report.md — the profile report: 0x flame-graph links/references, EXPLAIN plans, p95 before/after from the k6 runs) and their fix is landed: page tree via version-key Redis cache (race test: 20 concurrent writers, zero stale reads), permission decisions cached 60 s + event-bust (bust test: grant change → fresh decision on next call; no event → stale ≤ 60 s), search 5 s TTL (invalidation + permission-keying test), db column metadata 60 s.
• Every EXPLAIN-proven missing index from the analysis is added via migration (the report lists: index, query, before-plan, after-plan); the record-open path issues ≤ 4 queries (query-count test); block batch save query count is flat across 10/100/1000 blocks (N+1 elimination test); PgBouncer manifests present + connection counts in staging stay under the documented pool limit under the api-load suite; db.read() routing test passes (replica serves tree/listing, primary serves permission + If-Match reads — the write-read-your-write test).
• tests/performance all 4 suites pass their thresholds in the staging run: api-load 1k rps p95 < 300 ms + errors < 0.1%; realtime-load 10k conns / 200 msg/s, update p95 < 200 ms, zero dropped ops; database-load 100k records table p95 < 500 ms + 100 rps creates p95 < 500 ms; editor-local interaction p95 < 50 ms + block move < 16 ms (local runner); the CI matrix enforces: dev report-only, staging hard (a threshold breach blocks the staging deploy — demonstrated by a fixture run with a lowered threshold that fails the pipeline step).
• OTel: all 4 services export traces (the test: a scripted user flow produces one trace with spans across web→api→(worker job resumed traceparent)→db/redis, attributes workspaceId + requestId present, no stack/PII in span attributes — a span-scrubber assertion); all metric names from the metric list are scraped by Prometheus in staging (a /metrics conformance test: every expected metric name + label set observed); every log line in the staging run carries requestId (a log-capture assertion) and the full user flow is traceable by following one requestId across api + worker logs.
• Grafana: the 6 dashboards provisioned in staging render (a screenshot/smoke step: the dashboard JSON validates + the data source returns non-empty for each panel's query — an API-level check via the Prometheus query API per panel); the completeness check (every metric paneld or explicitly unpaneld) passes; alert rules live: the chaos drill (worker down 5 min) fires the queue-depth alert within 5 m, delivered to both the webhook (automation run row exists) and email (mailcatcher row) — documented in the drill section.
• Backup/DR: wal-g PITR config in the k8s manifests + daily base backups running in staging (the backup job's last-success marker < 26 h, asserted by a CI health check); OpenSearch daily snapshot present; the executed drill: PITR restore to T−5 min in the scratch namespace + snapshot restore + the 3 verification queries pass; measured RPO ≤ 5 min and RTO ≤ 1 h recorded (or the deviation documented with the remediation task); the runbook section exists and every command in it was actually run in the drill (the drill log references the runbook step numbers).
• Release: tag v1.0.0 exists on main; the 4 ghcr images at the tag + latest exist (a registry-check step in the release pipeline); staging manifests applied + smoke E2E green; docs/development/admin-guide.md (one-command docker + k8s paths + sizing + backup env) and docs/support-matrix.md exist and are linked from README + release notes; the docs-completeness script passes (READMEs, OpenAPI+docs coverage, ROADMAP boxes — the Phase 15 boxes ticked last); CHANGELOG 1.0.0 section complete.
• Final E2E: tests/e2e/final-flow.spec.ts green in staging — every step of the 13-step flow passes (register → … → webhook signature verified), the run is recorded in the release notes with the staging URL + timestamp; this green run is the precondition for cutting the tag (pipeline order enforced: E2E → tag step).

Backend Changes

• packages/observability (new shared package): OTel SDK init + resource builder + pino-otel log bridge + the metric registration helpers (histogram/gauge/counter factories with the label sets above) + the traceparent-carrying dispatcher hook (API→worker) + the span-scrubber (no PII/secrets in span attributes — unit-tested).
• apps/api: common/cache.ts (version-key tree cache, TTL+event-bust permission cache, search + metadata caches, the content-version counter + its Redis wiring); Prisma multi-client (prisma/clients.ts db.read()/db.write()) + the replica routing applied to tree/listing/search services (permission + If-Match paths stay on primary); query-shape fixes (narrow selects, batch createMany/updateMany on the block save flush); pgbouncer-aware DATABASE_URL config (zod env).
• apps/worker: traceparent resume in every processor; queue metrics; backup cron jobs (or the k8s CronJob — decision: WAL/base-backup = k8s CronJob + wal-g, OpenSearch snapshot = k8s CronJob calling the OS API, documented; the worker only does the verification job post-backup).
• apps/realtime: ws span ops + connection/message metrics + per-room span attributes; connection limits per the 10k-conns target (the k8s resource requests + the ws server max-connections config documented).
• apps/web: OTel init (client tracing off by default in v1 — server-side only; recorded decision: client-side web-vitals via the browser SDK are out of scope for v1, web-vitals reported via logs only), requestId propagation to API fetches.
• Migrations: the proven indexes (listed in the Requirements fix); no breaking changes.

Frontend Changes

• None functional — this is infra/perf. (If the profile shows the initial-load < 2 s target is missed on the web shell, the fix is route-level code splitting of the heavy editor/database components — expected: next/dynamic on the editor bundle; the perf-report documents it if done.)

Database Changes

• Migrations for the EXPLAIN-proven indexes (each named in the report with before/after plans); Prisma multi-client configuration (prisma/clients.ts) — writes primary, reads replica per the routing contract; PgBouncer transaction-mode compatibility verified (no session-state features used — the test: the full integration suite passes against a pgbouncer-backed URL in CI staging).

API Changes

• None (no endpoint changes; /metrics added per app — public but not API surface, excluded from the OpenAPI contract allowlist, documented in docs/api/versioning.md).

Tests

• tests/performance/{api-load,realtime-load,database-load}.js + editor-local.mjs + seed-100k.mjs (k6 + the local browser driver) with the §26 thresholds; the CI matrix dev-report/staging-hard.
• Cache tests: the tree version-key race (20 concurrent writers), permission bust + TTL bounds, search keying (two principals → separate entries), metadata bust; the corrupt-entry-is-a-miss test.
• Query-shape tests: record-open ≤ 4 queries; batch-save flat query count; the replica routing test (read/write split, read-your-write on primary); the pgbouncer integration pass.
• Observability tests: the cross-service trace test (web→api→worker→db spans, attributes, scrubber), the metrics conformance test (all names/labels in staging /metrics), the requestId log-capture test, the dashboard completeness check (script over the JSON + the Prometheus query API).
• DR: tests/integration/drill/verify-restore.mjs (the 3 queries); the backup-health CI check (last backup < 26 h); the alert-drill script (chaos + alert assertion).
• Release: scripts/docs-completeness.mjs (READMEs, OpenAPI+docs coverage, ROADMAP boxes), the registry-check step, the smoke suite (Phase 0) + tests/e2e/final-flow.spec.ts (the 13-step staging flow).

Security

• Observability carries data: span-scrubber + log scrubber (TASK-0134) re-verified against the new instrumentation (no workspace content in span attributes beyond ids; query text redacted to operation shape — asserted by the trace test); /metrics unauthenticated is acceptable in the cluster network (network-policy restricted to the observability namespace — the k8s NetworkPolicy manifests added, tested by a curl-from-outside-namespace failure).
• Backup media is secret-bearing (WAL contains data): the S3 bucket is private + versioned + the IAM policy least-privilege (the wal-g service account: put/get on the backup prefix only — the k8s SA + the IAM policy in the manifests, reviewed in the drill); restore runbook requires the org-admin role (the scratch-namespace access = a documented elevated procedure, two-person: runbook states the approval step).
• The release pipeline cannot tag a red build: pipeline order lint→typecheck→unit→integration→security→build→E2E(staging)→docs-completeness→tag (the Phase 0 CI contract extended; a tag without a green run is impossible in the CI — locally it's a manual act, documented).

Documentation

• docs/development/perf-report.md (the top-5 profiles + fixes + before/after numbers); docs/development/deployment.md (§ Restore runbook + executed-drill results, § Self-hosting, backup config, the observability stack); docs/development/admin-guide.md (new); docs/support-matrix.md (new); infrastructure/observability/README.md (metric catalog + the unpaneld list + the scrape/alert config guide); CHANGELOG.md 1.0.0; README "Operations" section (runbook links); ROADMAP Phase 15 boxes ticked (last, after the final E2E).

Dependencies

• TASK-0134 (security baseline — the audit must be done before release; the security CI stage is a gate on the release pipeline), Phases 0–14 (everything), the staging cluster (k8s access for the manifests/dashboards/drill — the environment is a CI/staging prerequisite, the k8s manifests from Phase 0), S3 for backups (the bucket + IAM in infrastructure/kubernetes/), ghcr (org token in CI), mailcatcher in staging (final-flow email step).

Definition of Done

• [ ] Top-5 hot paths profiled + fixed (caches with the exact invalidation semantics, proven indexes, query-shape fixes, PgBouncer + replica routing) — all with tests; perf-report written.
• [ ] All 4 k6/local suites pass the §26 + load-matrix thresholds in staging; CI matrix enforces (dev report-only / staging hard, demonstrated).
• [ ] OTel traces/metrics/logs live in all 4 services with correlation + scrubbing; Prometheus scrape + recording rules; 6 Grafana dashboards rendering in staging; alert rules firing (drill documented, webhook + email proven).
• [ ] PITR + daily versioned backups + OpenSearch snapshots configured in k8s; restore drill executed (scratch namespace, 3 verification queries, RPO/RTO measured and recorded); runbook complete.
• [ ] v1.0: CHANGELOG, tag v1.0.0, ghcr images, staging apply + smoke, docs-completeness green, admin guide + support matrix; final 13-step E2E green in staging — release notes link the run.

---

Deviations & Conflicts (AGENTS rule 29 — recorded, not guessed)

Implementation/verification drift recorded while completing TASK-0135 locally (dev box; staging-only items documented, never faked). App-side bugs found by the final 13-step E2E (tests/e2e/specs/final-flow.spec.ts) and fixed server- or client-side; spec-side edits were limited to test-logic defects with the assertions left unchanged.

1. Relation read-path gap (fixed, API). Relation values are written to the RelationLink junction (via the links field) but were never materialized in the record DTO — toDto read the relation value from DatabaseRecordProperty, where a RELATION property row is never stored, so properties[relPropId] was always null. Fixed in apps/api/src/modules/databases/query.service.ts: a buildRelationLinkRecords batch helper (one relationLink.findMany + one relation.findMany, replica- routed for reads, primary for updateRecord read-your-own-write) and a RELATION branch in toDto that synthesizes { recordIds }. Wired into list / query / getRecord / updateRecord.

2. Record search-doc titles were "(untitled record)" (fixed, worker). apps/worker/.../search-index/builders.ts buildRecordDoc read the title from the isTitle DatabaseRecordProperty row, which the API never writes (title lives on Page.title). Now the record and its linked records are fetched with a page.title join and the doc title is Page.title (the isTitle property is a legacy fallback only and is excluded from the body when the page title is used).

3. Webhook envelope field is type, not event (spec fixed). The delivery envelope (buildDeliveryEnvelope, packages/shared/src/webhook-delivery.ts) and docs/api/webhooks.md §4 both carry the event name in type; there is no event field. The final-flow spec's step-13 assertion was corrected from parsed.event to parsed.type (assertion value unchanged).

4. Step-8 getByLabel('Search') strict-mode violation (spec fixed). Non-exact matched both the toolbar button (aria-label="Search (Command K)") and the input (aria-label="Search"). Spec now uses getByLabel("Search", { exact: true }) (the input, which is the one .fill()ed).

5. useRelations returned PromiseFulfilledResult wrappers (fixed, web). apps/web/src/hooks/use-database.ts mapped a Promise.allSettled result without unwrapping, so consumers read .recordId/.record off the wrapper (always undefined) and rendered the "Record" fallback; this also produced the pre-existing web tsc baseline errors at cell-editor.tsx L765. Now .filter(fulfilled).map(r => r.value).

6. Record-open query-shape test mock (test updated, bounds unchanged). record-open-query-shape.test.ts counting mock gained the read()/write() delegates (returning the same counting client) and a no-op relation delegate required by the new RELATION helper (the fixture has no RELATION property, so the helper short-circuits). The ≤4 / ≤5 / 6 query-count assertions are unchanged.

7. Step-13 refresh-after-logout 200 (spec fixed). The final assert used context.request.post("/api/v1/auth/refresh") — a relative URL. BrowserContext.request is bound to the context's baseURL (WEB_BASE, :3000, the Next app), which has no such route and 307-redirects to /login (→200), masking the real revocation. The API correctly returns 401 (verified by a pure-API diag and an in-browser diag). The spec now posts to ${API_BASE}/api/v1/auth/refresh (absolute); the context.request still shares the browser context's (cleared) cookies, so the assertion is unchanged.

8. Mobile (<640 px) layout: the fixed 256 px sidebar squeezed the editor (fixed, web). The app shell rendered the sidebar as an in-flow column at EVERY breakpoint, so at 375 px the editor column was ~119 px wide and the block drag-handle gutter (left of the row) landed under the sidebar, which intercepted the pointer — dnd-kit reorder was unreachable on mobile (the final-flow mobile project failed at step 5). docs/product/ux-requirements.md (mobile row + the z-index tokens) specifies the intended behavior: single- column everything, the sidebar as a z-90 DRAWER (closed by default), and the editor's ⋮⋮ handle always visible in the RIGHT gutter. Implemented: ui-store mobileNavOpen (default false) + setMobileNav; the sidebar renders the drawer only when open below md: and the in-flow column only at md: and up (the topbar hamburger is breakpoint-aware); the block toolbar gutter in page-view.tsx is right-anchored + always visible below md: (one toolbar, one data-testid="drag-handle"). Desktop behavior is unchanged (left gutter on row hover; the final-flow desktop run is green). The final-flow mobile spec gains the real user's opener (topbar hamburger → drawer visible) before clicking user-menu-logout, since the button now lives inside the closed-by-default drawer (the desktop path is unchanged).

Still-open pre-existing notes (not fixed here, recorded for follow-up): shared block.dto op set; inline-editor dead guards (L163/L168); use-pages.ts L74; base-36 vs base-62 id handling; next lint/ESLint-9 pre-existing breakage; /page-tree 404 route; web tsc baseline (was 123 → now lower after the useRelations fix); list() replica-routed read (dev has no replica → primary).

ROADMAP Phase 15 says "Database optimization (indexes, partitioning for blocks/audit, query plans)"; the TASK-0135 spec's database-optimization AC is EXPLAIN-proven indexes + query-shape fixes (no partitioning AC), so partitioning was NOT implemented — no EXPLAIN case demonstrates the need at the dev scale, and a speculative partition migration would violate the "no speculative indexes" rule. Recorded per rule 29 (spec over roadmap wording); if the audit table outgrows a single partition it is a follow-up task.