Deployment Guide — AI Software Factory

How the platform is deployed from local dev to production, and how it is released, migrated, backed up, and restored. Targets: NFR-H1…H5 (self- hostability), NFR-B1…B7 (backup/DR: RPO ≤ 5 min, RTO ≤ 1 h), NFR-R1 (99.9 % API SLO), NFR-H3 (resource floor). Related docs: development-guide.md §2.1 (compose service table), non-functional-requirements.md §7 (metrics/dashboards/alerts), ../security/security-architecture.md (transport headers & secrets).

---

1. Deployment topology (production reference)

                        ┌────────────────────────────────────────────┐
   Internet ──HTTPS───► │ nginx (TLS term, WS upgrade, gzip, headers)│
                        └───────┬──────────────────────────┬─────────┘
                                 │ / (web, /api/*, /share) │ /ws (upgrade)
                        ┌────────▼─────────┐      ┌────────▼─────────┐
                        │ web (Next.js)    │      │ realtime (ws)    │
                        │ ×2, stateless    │      │ ×2, rooms sharded│
                        └────────┬─────────┘      └────────┬─────────┘
                                 │                          │
                        ┌────────▼──────────────────────────▼─────────┐
                        │ api (NestJS modular monolith) ×2…N, HPA     │
                        └──┬──────────┬──────────┬──────────┬─────────┘
                           │          │          │          │
                    ┌──────▼───┐ ┌────▼────┐ ┌───▼──────┐ ┌─▼─────────┐
                    │ Postgres │ │  Redis  │ │OpenSearch│ │ MinIO/S3  │
                    │ (RDS/Pod,│ │(Elasti- │ │(1..3     │ │(bucket:   │
                    │ PITR)    │ │ Cache)  │ │ nodes)   │ │ versioned)│
                    └──────────┘ └─────────┘ └──────────┘ └───────────┘
                           ▲          ▲          ▲
                    ┌──────┴──────────┴──────────┴──────────┐
                    │ worker (BullMQ) ×2…N: search, files,  │
                    │ webhooks, automation, email, ai, jobs │
                    └───────────────────────────────────────┘
   Observability: otel-collector → {prometheus, loki} + grafana (side, internal)

All app workloads are stateless (state lives in Postgres/Redis/S3/ OpenSearch), which is what makes rolling deploys and HPA safe. The realtime gateway is the one stateful-ish component: room state is held in the Yjs doc cache (Redis-backed update log is the durable part), so pods are replaceable — a reconnecting client resumes from Redis, not from the old pod.

2. Environments

Env	Purpose	Infra	Config source	Notes
dev	Local development	docker compose (single host)	.env (git-ignored)	Everything in-process/localhost; AI + SMTP optional; no TLS; OpenSearch security plugin off
staging	Pre-prod verification; SLO burn source; ZAP target; DR rehearsal	Kubernetes (small, NFR-H3 floor sizing × 1)	SealedSecrets/SOPS + cluster	Full parity with prod topology (2× of each app), seeded with synthetic data, nightly perf-standard
prod	Customer traffic	Kubernetes (≥ NFR-H3 floor) or docker compose on a dedicated server (self-host)	external secret store (Vault) + SOPS-encrypted values	TLS mandatory, OpenSearch security on, all alerts live, telemetry opt-in per NFR-H5

Environment promotion: develop → staging on every merge to main (auto); main tagged vX.Y.Z → prod (manual, release process §6). Staging and prod share no data store; staging's Postgres is a synthetic clone (anonymized) refreshed weekly.

3. Kubernetes walkthrough

Manifests: infrastructure/kubernetes/ (plain manifests + a Helm-free "manifests as code" layout; Kustomize overlays per env).

infrastructure/kubernetes/
├── base/
│   ├── namespace.yaml            # namespace: factory
│   ├── api/{deployment,service,hpa,serviceaccount}.yaml
│   ├── web/{deployment,service}.yaml
│   ├── worker/{deployment}.yaml
│   ├── realtime/{deployment,service,hpa}.yaml
│   ├── ingress.yaml              # (alternative to nginx; see §5)
│   ├── pdb-{api,realtime}.yaml
│   ├── configmap-app.yaml        # non-secret config (URLs, limits)
│   └── networkpolicy-default-deny.yaml
└── overlays/{staging,prod}/
    ├── kustomization.yaml        # patches: replicas, resources, image tags
    └── patches/                  # env-specific (staging: 1 replica each; prod: floor)

3.1 Workloads

api Deployment (reference — others follow the same shape):

apiVersion: apps/v1
kind: Deployment
metadata: { name: api, namespace: factory }
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
  selector: { matchLabels: { app: api } }
  template:
    metadata:
      labels: { app: api }
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9464"          # otel sidecar / app metrics
    spec:
      serviceAccountName: factory-api
      terminationGracePeriodSeconds: 60      # drain in-flight requests + close WS
      containers:
        - name: api
          image: registry.example.com/factory/api:v1.4.2
          ports: [{ containerPort: 4000 }]
          envFrom: [{ configMapRef: { name: app-config } }]
          env:                               # secrets from the external store
            - name: DATABASE_URL
              valueFrom: { secretKeyRef: { name: factory-secrets, key: database-url } }
            - name: REDIS_URL
              valueFrom: { secretKeyRef: { name: factory-secrets, key: redis-url } }
            - name: JWT_SECRET
              valueFrom: { secretKeyRef: { name: factory-secrets, key: jwt-secret } }
            # … (full list = .env.example; see development-guide §4.3)
          resources:
            requests: { cpu: 500m, memory: 1Gi }
            limits:   { cpu: "2",  memory: 2Gi }
          readinessProbe:                    # must pass before traffic
            httpGet: { path: /healthz/ready, port: 4000 }
            initialDelaySeconds: 5; periodSeconds: 5; failureThreshold: 3
          livenessProbe:
            httpGet: { path: /healthz/live, port: 4000 }
            initialDelaySeconds: 15; periodSeconds: 10; failureThreshold: 3
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            capabilities: { drop: [ALL] }
            seccompProfile: { type: RuntimeDefault }

Workload	Replicas (prod floor)	Notes
api	3	stateless; HPA 3→8 (CPU 65 % or rps target 350/pod)
web	2	Next.js standalone; HPA 2→4 (CPU 70 %); image = standalone server build
worker	2	one container per queue family (search,files / webhooks,automation / email,ai,jobs) via QUEUE_FAMILY env; HPA on queue-depth metric (custom: queue.depth > 500 → +1)
realtime	2	HPA 2→6 on ws.connections > 4000/pod; room→pod routing by consistent hash on doc id (client picks pod via the WS router endpoint)
otel-collector	2 (DaemonSet alternative)	sidecar-free, receives OTLP from apps
(Postgres/Redis/OpenSearch/MinIO)	—	managed services (RDS/ElastiCache/OpenSearch domain/S3) in cloud; statefulsets in self-host k8s

HPA:

kind: HorizontalPodAutoscaler
metadata: { name: api, namespace: factory }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: api }
  minReplicas: 3; maxReplicas: 8
  behavior: { scaleUp: { stabilizationWindowSeconds: 30 },
              scaleDown: { stabilizationWindowSeconds: 300 } }   # no flapping
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 65 } }
    - type: Pods
      pods: { metric: { name: http_rps }, target: { type: AverageValue, averageValue: "350" } }

PDBs (prevent total outage during node drains / cluster upgrades):

kind: PodDisruptionBudget
metadata: { name: api-pdb, namespace: factory }
spec:
  minAvailable: 2            # ≥ 2 of 3 api pods always schedulable
  selector: { matchLabels: { app: api } }
---
kind: PodDisruptionBudget
metadata: { name: realtime-pdb, namespace: factory }
spec:
  minAvailable: 1
  selector: { matchLabels: { app: realtime } }

Services: api (4000), web (3000), realtime (4001, sessionAffinity: ClientIP not used — rooms are routed by the client hash, so the service is plain round-robin), all ClusterIP. Health endpoints: /healthz/live (process up), /healthz/ready (deps up: PG ping < 200 ms, Redis PING, Redis stream group exists; 503 otherwise).

Secrets via external store: the cluster never contains raw secrets in git. Two supported patterns (pick per deployment, set in overlay):

1. Vault (default cloud): vault-agent init container injects factory-secrets from the KV path secret/data/factory/<env> at pod start (agent auto-unseals via the cloud IAM auth method). Rotation = update KV + rolling restart (secrets are read at boot).
2. SOPS + SealedSecrets (default self-host): sops -e-encrypted values committed to overlays/<env>/sealed/; sealed-secret controller decrypts with the cluster's SealedSecret key. Rotation = re-seal + apply.

Key rotation policy: JWT_SECRET/REFRESH_TOKEN_SECRET support a 2-key overlap window (JWT_SECRET + JWT_SECRET_PREV) so a rolling restart never invalidates outstanding access tokens; MASTER_ENCRYPTION_KEY rotation is envelope-encryption key rotation (new DEK, re-wrap KEK refs lazily — see security-architecture §11).

Connection pooling (PgBouncer, transaction mode) + read replica. Prisma 5.22 has no client-side pool (PrismaClientOptions = {datasources, datasourceUrl, errorFormat, log, transactionOptions} — connectionLimit/poolTimeout are Prisma 6), so pooling is owned by PgBouncer in front of Postgres (postgres/pgbouncer.yaml), in transaction mode (Prisma's required mode). Each app connects with its own role through its own bounded pool (a burst in one app cannot starve another):

App	Role	DATABASE_URL user	Pool (pool_size)	Sizing rationale
api	api	api	20	the 1k-rps load path — busiest writer/reader
worker	worker	worker	20	BullMQ steady state (FILES_WORKER_CONCURRENCY ≤ 8)
realtime	realtime	realtime	10	Yjs ops are Redis-first; PG is the flush/commit path
web	web	web	5	server-side rendering reads only
(db.read)	reader	reader	15	the read-replica pool (read-only role)

Max total ≈ 70 server connections, under the PG16 max_connections=100 with headroom. The per-app share is the same value the app's DATABASE_POOL_SIZE env documents (configmap) — it is the PgBouncer pool_size, not a Prisma option. The app roles are created by the primary's first-boot hook (postgres/configmap.yaml zz-first-boot-pitr.sh: CREATE ROLE … GRANT factory TO …; reader is read-only via pg_read_all_data). pgbouncer authenticates each role with a static userlist.txt of scram-sha-256 verifiers (the factory-pgbouncer-auth secret, generated from the factory-db password).

Read replica (postgres/replica.yaml): a hot-standby StatefulSet (pg_basebackup + streaming replication from factory-db). The API's DATABASE_REPLICA_URL points at the pooler's factory;reader entry; db.read() serves tree / listing / search from it while db.write(), permission decisions, and the If-Match/baseVersion version checks stay on the primary — a write-then-read-your-write on the same request always hits the primary (the staleness contract, perf-report.md §6; proven by tests/integration/replica-routing.integration.test.ts). A replica loss is non-critical: db.read() degrades to the primary (no DATABASE_REPLICA_URL → reader = primary), at the cost of the read-scaling benefit.

Staging verification (the spec's pool-limit check): during the api-load suite, current_pool per role must stay ≤ the pool_size above — psql "postgres://factory:factory@factory-pgbouncer.factory.svc:6432/pgbouncer" -c "SHOW POOLS;". The pooler's auth wiring (scram verifiers + app roles) is confirmed at first staging apply (no cluster in-repo to test it — recorded per the AGENTS rule).

3.2 Ingress & TLS

Two supported front ends — nginx sidecarless (§5, default self-host, and the reference for k8s via the ingress-nginx controller) or a cloud LB + cert-manager. Kubernetes reference (ingress-nginx):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: factory
  namespace: factory
  annotations:
    cert-manager.io/cluster-issuer: lets-encry          # or acme-dns-01
    nginx.ingress.kubernetes.io/proxy-body-size: "512m" # file uploads
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
    - hosts: [factory.example.com]
      secretName: factory-tls
  rules:
    - host: factory.example.com
      http:
        paths:
          - path: /ws
            pathType: Prefix
            backend: { service: { name: realtime, port: { number: 4001 } } }
          - path: /api
            pathType: Prefix
            backend: { service: { name: api, port: { number: 4000 } } }
          - path: /
            pathType: Prefix
            backend: { service: { name: web, port: { number: 3000 } } }

TLS: cert-manager issues/renews (Let's Encrypt HTTP-01 or DNS-01); certs auto-rotate at 30 d; the certificate alert (NFR §7.3) pages < 14 d. HSTS is set by nginx/ingress (Strict-Transport-Security: max-age=31536000; includeSubDomains); TLS 1.0/1.1 disabled (ssl-protocols TLSv1.2 TLSv1.3).

4. Docker Compose (local & single-server production)

infrastructure/docker/docker-compose.yml (dev: infra services; prod single-server: add docker-compose.prod.yml overlay that brings up web/api/worker/realtime containers with the built images + the same four infra services + clamav + mailpit→smtp relay).

Single-server prod service table (ports are host-exposed only via nginx, which is also a compose service on 80/443):

Service	Image	Ports (internal)	Restart	Depends on
nginx	nginx:1.27-alpine (config: infrastructure/nginx/nginx.conf)	80, 443	always	web, api, realtime
web	factory/web:<tag>	3000	always	api
api	factory/api:<tag>	4000	always	postgres, redis
worker	factory/worker:<tag>	—	always	postgres, redis, minio, opensearch
realtime	factory/realtime:<tag>	4001	always	redis
postgres	postgres:16-alpine	5432	always	— (volume pgdata, synchronous_commit=on)
redis	redis:7-alpine (AOF appendfsync everysec)	6379	always	—
opensearch	opensearchproject/opensearch:2.x (security on, admin creds from env)	9200	always	—
minio	minio/minio	9000	always	— (bucket versioning enabled via init)
clamav	clamav/clamav	3310	always	—

Compose prod rules: all containers user: 1000:1000 where the image allows; no privileged; inter-service traffic on the internal network only (infra ports not published); healthcheck on every service (same as dev table); log driver json-file with max-size: 20m, max-file: 5 (or otel file-log forwarding when telemetry enabled).

5. Nginx configuration walkthrough

Reference: infrastructure/nginx/nginx.conf (+ conf.d/factory.conf). This is the TLS/WSS/gzip/header authority for the single-server deployment (and the contract the ingress-nginx annotations must match).

# ── TLS ─────────────────────────────────────────────────────────────────
server {
    listen 80;
    server_name factory.example.com;
    return 301 https://$host$request_uri;          # everything → TLS
}
server {
    listen 443 ssl http2;
    http2 on;
    server_name factory.example.com;

    ssl_certificate     /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_ciphers         ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256;
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 1d;

    # ── Security headers (exact values — security-architecture §7) ────────
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-Frame-Options "DENY" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    add_header Permissions-Policy "camera=(), microphone=(), geolocation=()" always;
    # CSP is set by the API/Next responses (per-route nuances); nginx does NOT
    # double-set it:
    #   Content-Security-Policy: default-src 'self'; script-src 'self';
    #     style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:
    #     https://*.s3.amazonaws.com; connect-src 'self' ws: wss: https://*.s3.amazonaws.com;
    #     frame-src https://www.youtube.com https://player.vimeo.com;
    #     object-src 'none'; base-uri 'self'; frame-ancestors 'none'

    # ── gzip ─────────────────────────────────────────────────────────────
    gzip on;
    gzip_vary on;
    gzip_min_length 1024;
    gzip_types text/plain text/css application/json application/javascript
               application/xml image/svg+xml font/woff2;

    # ── WebSocket upgrade (realtime gateway) ─────────────────────────────
    location /ws {
        proxy_pass http://realtime:4001;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;          # required
        proxy_set_header Connection "upgrade";           # required
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_read_timeout 3600s;    # long-lived; server heartbeat (30 s)
        proxy_send_timeout 3600s;    # keeps idle-but-alive sockets open
    }

    # ── API ──────────────────────────────────────────────────────────────
    location /api/ {
        proxy_pass http://api:4000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        client_max_body_size 512m;   # multipart uploads (presigned S3 direct,
                                     # but small metadata + form files come here)
        proxy_read_timeout 120s;     # exports/imports (job endpoints poll fast)
    }

    # ── Web (Next.js) ────────────────────────────────────────────────────
    location / {
        proxy_pass http://web:3000;
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;          # Next.js HMR/dev +
        proxy_set_header Connection "upgrade";           # (harmless in prod)
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
    }

    # ── Static caching for Next assets ───────────────────────────────────
    location /_next/static/ {
        proxy_pass http://web:3000;
        add_header Cache-Control "public, max-age=31536000, immutable";
        access_log off;
    }
}

Walkthrough of the non-obvious choices:

• WS upgrade: without proxy_http_version 1.1 + the Upgrade/ Connection pair, the realtime gateway returns 400 to all clients. proxy_read_timeout 3600s prevents nginx killing healthy idle sockets (the app-level 30 s heartbeat is the real liveness signal).
• Header policy: HSTS always (sent on errors too); X-Frame-Options: DENY (no clickjacking surface — public shares are same-site pages, not embeds); CSP is application-emitted because share/public routes relax frame-src for embeds and the API sets it on JSON error pages — one nginx-level CSP can't express that (see security-architecture §7 for the exact policy string).
• gzip: JSON + JS + CSS only; images already compressed; gzip_vary for shared-cache correctness.
• Body size: 512 m matches the 500 MB file cap (large uploads go presigned-direct to S3; this ceiling covers form-file and import uploads).
• Rate limiting (nginx level, first line of defense; app level is authoritative): limit_req_zone $binary_remote_addr zone=api:10m rate=30r/s; applied to /api/ with burst=60 nodelay; auth paths get a stricter zone (auth:10m rate=1r/s burst=5) mirroring the app tiers.

6. Release process

 tag vX.Y.Z ──► CI: build images ──► push (registry) ──► staging deploy
      │            (api, web, worker,      │               │
      │             realtime, tagged       │        verify (§6.2)
      │             vX.Y.Z)                │
      └── release notes (CHANGELOG.md auto-updated from conventional commits)
                                            ▼
                              prod: migrate ──► deploy (rolling) ──► verify ──► announce

Procedure (runbook infrastructure/runbooks/release.md):

1. Tag: on main, git tag -a v1.4.2 -m "…" && git push origin v1.4.2. CI builds the four images (multi-arch amd64/arm64) and pushes them to the registry as v1.4.2 + sha-<short>; the tag build also re-runs integration + e2e (chromium) as a final gate.
2. Staging first (automatic on tag): deploy to staging; run the release verification checklist (§6.2) including perf-smoke; staging must be green for ≥ 30 min (soak window) before prod.
3. Prod migration: kubectl -n factory exec deploy/api -- sh -c "pnpm db:deploy" (or the migrate Job in the release pipeline) — migrations run before the new pods, and must be backward-compatible (old code must still run on the new schema; see §7).
4. Prod deploy: kubectl -n factory set image deploy/api api=…/api:v1.4.2 (same for web, worker, realtime — order: api → worker → realtime → web so consumers are ready before producers of new events). Rolling update (maxUnavailable: 0) — PDBs hold ≥ minAvailable throughout.
5. Verify (§6.2) + watch dashboards 30 min (error rate, queue depth, WS reconnect spike).
6. Announce (status page / changelog) only after the checklist passes.

Rollback (any step fails verification):

1. kubectl -n factory set image … <previous tag> (reverse order: web → realtime → worker → api). Rolling, < 5 min total.
2. Migrations are NOT auto-rolled-back (destructive); the backward- compatibility rule means the old code works on the new schema. If a migration itself failed mid-way: follow §7 rollback.
3. If the rollback is due to a data-level bug, apply the hotfix forward (patch release) rather than a second rollback.

6.2 Release verification checklist

• [ ] /healthz/ready green on all pods of every deployment
• [ ] GET /api/v1/openapi.json matches the committed spec (contract test)
• [ ] Smoke: login → open page → edit → search finds it (manual or scripted)
• [ ] 13 MVP e2e flows pass against the release images (chromium)
• [ ] Error rate < 0.1 % and p95 within budget for 30 min post-deploy
• [ ] No new entries in the DLQ (automation/webhook) for 30 min
• [ ] WS connection count recovered to baseline (no stuck reconnects)
• [ ] perf-smoke profile: advisory metrics within 2× budget
• [ ] Backup job ran successfully post-deploy (new schema backed up)

7. Database migration procedure & rollback

Rules (mirror prisma/migrations/README.md + NFR-H4):

1. Migrations are additive-first: expand → migrate → contract.
   • v1: add column/index/table (deploy, both old+new code fine).
   • v2: backfill data (idempotent job, batched 5k rows, resumable).
   • v3 (next release): drop the old column. A column lives ≥ 2 releases.
2. Every migration file is small (< 500 ms on a 1M-row table) or explicitly marked long-running with a pgbouncer-safe ALTER TABLE ... NOT VALID + VALIDATE pattern for constraints.
3. Never DROP data-bearing objects in a release without a task-file review record; backups exist but recovery should not be the plan.

Procedure (prod):

# 1. Pre-check: current state vs migrations table
kubectl -n factory exec deploy/api -- pnpm prisma migrate status
#    → "Database is up to date" (or list pending)

# 2. Take a fresh base backup (belt & braces; PITR exists regardless)
kubectl -n factory create job pg-backup-$(date +%s) --from=cronjob/pg-base-backup

# 3. Apply (single runner — the api pod, or a dedicated migrate Job;
#    the Job takes a postgres advisory lock so two migrations can't race)
kubectl -n factory exec deploy/api -- pnpm db:deploy
#    → each migration runs in its own transaction (Prisma default);
#      long DDL (index CONCURRENTLY) runs outside — Prisma emits those as
#      separate steps; monitor with:
psql -c "SELECT pid, now()-query_start, query FROM pg_stat_activity WHERE query ILIKE '%ALTER TABLE%';"

Rollback decision tree (a failed/misbehaving migration):

Situation	Action
Migration failed mid-transaction	Prisma transaction rolled back automatically; re-run after fixing. Nothing to do.
Migration succeeded but new code misbehaves	Roll back the deployment, keep the schema (backward-compatible by rule 1). The expand column sits unused.
Migration succeeded and data is wrong (bad backfill)	Write a forward fix migration (re-backfill correctly) — never hand-edit prod data; record in the task file; verify with a checksum query before/after.
ADD COLUMN must be undone before the next release window	Ship a DROP COLUMN migration in the next release (contract step of expand/migrate/contract).
CREATE INDEX CONCURRENTLY failed (leaves INVALID index)	DROP INDEX CONCURRENTLY idx; then re-run the migration.
Catastrophic (schema corruption)	Full PITR restore (§8) — RTO budget 1 h; coordinate a freeze window.

Post-migration always: prisma migrate status clean, prisma validate, the e2e smoke, and a backup.verified before declaring the release healthy.

8. Backup & restore runbook

Objectives: RPO ≤ 5 min, RTO ≤ 1 h (NFR-B4/B5). Components and mechanisms:

Component	Mechanism	Frequency	RPO	Storage
Postgres	PITR via wal-g (pinned chekkan/wal-g:v3.0.7, infrastructure/kubernetes/postgres/): WAL archiving — Postgres-native archive_command = '/opt/walg/wal-push-wrapper.sh %p' (per-segment ship, seconds under load; archive_timeout=60 force-flushes at idle) → S3 pg/wal_005/; base backup — CronJob pg-backup 05:00 UTC, wal-g backup-push in online streaming mode (no maintenance window, no local pg_basebackup) → S3 pg/basebackups_005/	WAL continuous, base daily	≤ 5 min (per-segment ship + 60 s idle flush)	S3 bucket factory-backups, versioning ON (§8.7)
Object storage (files)	S3 versioning ON on the upload buckets (factory-uploads/factory-files). Documented decision (v1): no bucket snapshot — RPO for objects = the upload-queue replay window (a lost object is re-uploaded from its source on replay; files rows are restored by PITR and reconciled). Cross-region replication (cloud) is the DR copy.	continuous (versioning)	upload-queue replay window (documented deviation from RPO 0)	S3/MinIO
OpenSearch	Daily snapshot — CronJob opensearch-snapshot 05:30 UTC calls the OS snapshot API (documented decision: the snapshot is a k8s CronJob calling the API, not a sidecar and not the worker; the worker only runs the post-backup verification job). S3 repo factory-s3, indices by alias (factory-content → factory-content-vN, factory-docs, ai-chunks), ignore_unavailable	daily	≤ 24 h (search is rebuildable — the reindex job is the recovery path, the snapshot the fast path)	S3 factory-backups/opensearch/snapshots/
Redis	No persistence in DR — documented decision: rebuildable cache/queue, not a system of record. Local AOF/PVC (crash consistency) stays on, but a lost Redis is recovered by queue replay from the outbox (NFR-R4) + cache warm-up — there is no Redis artifact to restore	n/a	rebuild (seconds–minutes)	n/a
Secrets/config	SOPS-encrypted in git (versioned inherently) + Vault KV (cloud)	every change	0	git / Vault
Terraform state	encrypted state (S3 + DynamoDB lock / self-host: local + git copy)	every apply	0	S3

The three backup-health markers the release pipeline gates on (stage 9, §8.7) are the real artifacts above: newest LastModified under pg/basebackups_005/ (< 26 h), newest LastModified under pg/wal_005/ (< 10 min), newest SUCCESS snapshot start_time (< 26 h).

8.1 Restore: PITR (Postgres) — the critical path

GOAL: recover the database to time T (e.g. 5–10 min before the incident — the Phase-15 drill standard is T−5 min). The drill log must cite the step numbers below as it executes them. All commands assume the backup identity's credentials are exported (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) and a pinned wal-g binary is available (chekkan/wal-g:v3.0.7 — the same image the cluster uses, so behavior is identical to what was archived).

Step 1 — Pick the target time T
   • T = incident_start − 5 min, ISO-8601 UTC (e.g. 2026-10-03T13:35:00Z).
   • Sanity: T must be AFTER the newest base backup and BEFORE the newest
     archived WAL. Check both (wal-g backup-list + the WAL folder listing).

Step 2 — Provision the scratch instance in namespace restore-drill
   • kubectl create namespace restore-drill
   • Same postgres:16-alpine image, empty PVC, NO data. It is isolated from
     the live namespace — the drill never touches the live service DNS.
   • Export the wal-g env (the v3 names — verified against the pinned image):
       export WALG_S3_PREFIX="s3://factory-backups/pg"
       export AWS_ENDPOINT=<S3 endpoint> AWS_REGION=<region>
       export AWS_S3_FORCE_PATH_STYLE=true
       export AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=…
   • Sanity: wal-g backup-list        # must show the base backups

Step 3 — Fetch the base backup
   wal-g backup-fetch /var/lib/postgresql/data LATEST
   • LATEST = newest full/chain-completing base (verify the choice in
     backup-list output first; pin an explicit name if the incident predates
     LATEST).

Step 4 — Configure recovery to T
   • In $PGDATA/postgresql.conf (the restored data dir), add:
       restore_command = '/opt/walg/wal-g wal-fetch %f %p'
       recovery_target_time = 'T'
       recovery_target_action = 'pause'
     (wal-fetch exits 74/EX_IOERR when the archived WAL ends — the designed
     "end of archive" signal; no wrapper needed for this drill.)

Step 5 — Start the instance; watch recovery
   • postgres -D $PGDATA  (or the usual entrypoint)
   • psql: SELECT pg_is_in_recovery();  →  t   (replaying)
   • It PAUSES at T (recovery_target_action=pause).
   • While paused — VERIFY before promoting (the 3 drill queries, §8.8):
       node tests/integration/drill/verify-restore.mjs
     plus the marker spot-check:
       SELECT * FROM audit_logs ORDER BY created_at DESC LIMIT 5;
     → every row with created_at < T present; no rows with created_at > T.

Step 6 — Promote (if verification passed)
   SELECT pg_promote();
   • Wait for pg_is_in_recovery() → f; instance is now writable at T.

Step 7 — Cutover
   • Live incident: repoint the app (k8s: swap the DATABASE_URL target /
     promote the scratch instance to the production Service selector, rolling
     restart api/worker/realtime).
   • Drill: point a scratch env's DATABASE_URL at the restored instance
     instead (namespace isolation is the whole point of the drill).

Step 8 — Post-restore hygiene
   • prisma migrate status — re-apply only migrations that were SAFE at T.
   • Re-enable archiving on the promoted instance (archive_mode stays on;
     the new timeline's WAL archives as normal — old-timeline WAL in the
     archive is left in place; `wal-g delete` per §8.6 retention policy).
   • Record RTO: (T picked) → (app serving) minutes, into the drill log.

Time budget (target < 60 min): Step 2 scratch instance ~5 min → Step 3 base fetch 10 min (tens of GB over 1 GbE) → Step 4–5 WAL replay to target 15–30 min (WAL volume dependent; the optional wal-g wal-verify integrity pass adds ~2 min) → Step 5 verify 10 min → Step 6–7 promote + cutover 5 min.

(Superseded: the pre-TASK-0135 procedure restored pg_basebackup tarballs from pg-base/ and replayed pg-wal/ via aws s3 cp in restore_command; the storage layout moved to wal-g's pg/basebackups_005/ + pg/wal_005/ and the commands to wal-g backup-fetch / wal-g wal-fetch — the procedure is the same shape, the tools are now what the cluster actually uses.)

8.2 PITR drill timing (how the ~10 min / ~2 min offsets are produced)

The drill's marker rows are written at REAL wall-clock time (a row only exists in the WAL archive if it was actually committed). The PITR target T is chosen AFTER seeding, so the ordering — not sleeps — defines the offsets:

t0  node tests/integration/drill/seed-drill.mjs          # page + record + OS doc
    (the state file /tmp/pitr-drill-state.json records seededAt)
t0+~10 min   (drill operator waits, or: SELECT pg_switch_wal(); and confirms
              the newest pg/wal_005/ object in S3 advanced past the page's
              commit — that segment is what must be archived for the restore)
t0+~12 min   re-run seed with PITR_RECORD_ONLY semantics for the record row,
              OR seed both rows at t0 and pick T = t0 + ~12 min — the page is
              then ~12 min before T, the record (if seeded at t0+~2 min)
              ~10 min before T. The acceptance wording "~10 min / ~2 min
              before T" is satisfied by seeding the page first, waiting,
              seeding the record, waiting ~2 min, then setting T.
T   t0 + ~14 min → execute §8.1 in namespace restore-drill with
    recovery_target_time = T
    → §8.8 verify (3 queries) against the restored stack

The PITR_TARGET_TIME env var recorded in the state file is what verify-restore.mjs uses to print the measured RPO window (T minus each marker's commit time — both must be positive, i.e. both rows predate T and both must survive the restore).

8.3 Restore: files (S3/MinIO)

# Restore a specific deleted/overwritten object to its previous version:
mc cp --version 2 local/factory-uploads/ws_abc/file_xyz  # list versions: mc ls --versions
# Restore the whole bucket to a point in time (self-host disaster):
mc mirror --overwrite --version 2025-01-15T13:00:00Z s3://factory-backups/uploads-replica/ local/factory-uploads/

The files table is in Postgres (restored by PITR); object keys are deterministic (<workspaceId>/<fileId>/<version>), so table + objects re-converge; a reconciliation job (files:reconcile) flags rows whose object is missing and re-queues a fetch from the replica.

8.4 Restore: OpenSearch (snapshot)

The snapshot repo factory-s3 is created by the daily CronJob (infrastructure/kubernetes/postgres/cronjob-opensearch-snapshot.yaml); on a FRESH (drill or DR) cluster recreate it first, then restore:

# 1. Recreate the repo on the fresh cluster (same S3 target as the CronJob):
PUT _snapshot/factory-s3
  {"type":"s3","settings":{"bucket":"factory-backups",
   "base_path":"opensearch/snapshots","endpoint":"<S3 endpoint>",
   "region":"<region>","access_key_id":"…","secret_access_key":"…"}}

# 2. Pick the newest SUCCESS snapshot before the incident:
GET _snapshot/factory-s3/_all          # pick <snap> (factory-YYYYmmddTHHMM)

# 3. RESTORE — original index names. VERIFIED against OS 2.15 (drill 2026-10):
#    the ALIAS is restored automatically with the index, so there is no
#    alias-swap step on a fresh cluster:
POST _snapshot/factory-s3/<snap>/_restore
  {"indices":"factory-content,factory-docs,ai-chunks","ignore_unavailable":true}
#    (OS 2.15 rejects the ES8-style index_rename_* params — a SCRATCH restore
#     means restoring to the original names on a SCRATCH cluster, i.e. the
#     restore-drill namespace, never on the live cluster mid-incident.)
#    On the LIVE cluster mid-incident, a snapshot restore is a CUTOVER:
#    restore the snapshot, confirm it, and the app keeps talking to the same
#    index names — there is nothing to re-point.

# 4. Refresh + count sanity:
POST factory-content/_refresh
GET  factory-content/_count

# 5. Reconcile: reindex events since the snapshot (≤ 24 h — the snapshot is
#    daily): pnpm search:reindex (or let the worker's content pipeline catch
#    up; search is rebuildable, the snapshot is the fast path).

8.5 Full DR drill (quarterly, prod)

Scripted (infrastructure/runbooks/dr-drill.md): 08:00 declare incident → destroy the cluster (or the single server) → 08:05 execute: fresh cluster from terraform (20 min) → PITR restore (≤ 40 min, §8.1) → files/objects check (10 min, §8.3) → OpenSearch snapshot restore (15 min, §8.4) → deploy current tag with restored config → run the 3 verification queries (§8.8) against the restored env → 10:00 declare RTO met (budget 1 h). A drill that isn't timed and documented doesn't count (NFR-B5). Drill report lands in docs/ops/dr-reports/ using the §8.8 template.

8.6 Backup verification & retention

Continuous verification (the mechanisms the cluster actually runs):

• Every base backup (CronJob pg-backup) ends with wal-g wal-verify (WAL-archive integrity + timeline check) and a pg_class row-count sanity — a "successful" base backup of an empty DB cannot pass silently.
• The release pipeline gate (§8.7) independently re-asserts marker freshness from the artifacts themselves.
• Object versioning is the corruption path: with versioning ON (§8.7) a corrupt base backup or WAL segment is recoverable by restoring the previous object version (mc cp --version <n> / S3 restore-object).
• Retention (manual, monthly — deliberately NOT automated in the CronJob, so a misconfigured automation cannot destroy the PITR chain): wal-g delete retain FULL 7 (keep 7 daily fulls; their WAL ranges stay covered while the retained base + the WAL after it are kept), and OpenSearch snapshots: delete older than 14 daily (DELETE _snapshot/factory-s3/<snap>), always keeping ≥ 2.

8.7 S3 bucket factory-backups: versioning + the health gate

Versioning (bucket-level; verified at deploy time — not faked in manifests):

# AWS S3:
aws s3api put-bucket-versioning --bucket factory-backups \
  --versioning-configuration Status=Enabled
# MinIO:
mc admin bucket set-versioning local/factory-backups on
# Verify (the backup identity is allowed s3:GetBucketVersioning — see
# postgres/walg-iam-policy.yaml):
aws s3api get-bucket-versioning --bucket factory-backups   # → Status: Enabled

The backup-health gate (release pipeline stage 9, .github/workflows/release.yml → scripts/backup-health.ts, pnpm backup:health): reads the REAL artifacts — newest LastModified under pg/basebackups_005/ (< 26 h), under pg/wal_005/ (< 10 min), and newest SUCCESS snapshot start_time from GET /_snapshot/factory-s3/_all (< 26 h). A REACHABLE-but-STALE marker fails in every mode; an UNREACHABLE endpoint is a loud report-only skip in dev (pnpm backup:health) and a hard failure in staging (BACKUP_HEALTH_STRICT=1 / DEPLOY_STAGING=true).

8.8 Executed drill — log template (STAGING-ONLY NOTE)

STAGING-ONLY: the executed-drill evidence for the Phase-15 acceptance criteria is produced in staging, not production. Fill this template after each drill and commit it to docs/ops/dr-reports/. Every command in the drill MUST reference a §8.1–§8.4 step number (the acceptance criterion is that the drill log cites the runbook steps it executed).

# PITR drill — <date> (staging)

- Operator: <name>            Scratch namespace: `restore-drill`
- Marker:      PITR-DRILL-<ts>   (seed-drill.mjs state file: <path>)
- PITR target T: <ISO-8601 UTC>   (the §8.2 convention: T = last-good − 5 min)
- Seed wall-clock: page seededAt=<ISO>, record seeded <ISO>
- §8.1 Step 1 (pick T): <chosen T + why>
- §8.1 Step 2 (scratch instance): <kubectl commands; instance up at <time>>
- §8.1 Step 3 (base fetch): `wal-g backup-fetch … LATEST` → backup `<name>`,
  finished <time> (took <n> min)
- §8.1 Step 4 (recovery config): recovery_target_time=<T> set
- §8.1 Step 5 (replay + verify): paused at <time>; verify-restore.mjs
  (paste the 3 checks + measurements JSON)
  - measured RPO window: page committed <n> min before T, record <n> min before T
- §8.1 Step 6 (promote): pg_promote at <time>
- §8.1 Step 7 (cutover): <env re-pointed at <time>>
- §8.1 Step 8 (hygiene): migrate status clean; archiving re-confirmed
- §8.3 files: object versioning confirmed; <any restored object>
- §8.4 OS snapshot restore: snapshot `<snap>` restored at <time>,
  doc count <n> (pre-incident <n>)
- RPO measured: <max(T − commit time of rows that survived), min>  (budget ≤ 5 min)
- RTO measured: <T picked → app serving, min>                      (budget ≤ 1 h)
- Deviations: <none | what + remediation task id>
- Verdict: PASS / FAIL — <one line>

(Superseded pre-TASK-0135 sections removed: the old §8.3 OpenSearch rename-restore, old §8.4 DR-drill, and old §8.5 continuous-verification descriptions were replaced by §8.4–§8.7 above — same intent, now matching the wal-g v3 layout, the alias-based snapshot design, and the artifact-based health gate.)

9. Monitoring stack deployment

Stack: OpenTelemetry Collector (OTLP ingest) → Prometheus (metrics)

• Loki (logs); Grafana (dashboards + SLOs + alerting via Prometheus rules; Loki alerts for log patterns). All in namespace factory-observability (or observability), internal-only (NetworkPolicy: only apps + the nginx management path).

9.1 Components & manifests (infrastructure/kubernetes/observability/)

Component	Deployment	Config
otel-collector	Deployment ×2, resources: 500m/1Gi	receivers: otlp (http 4318, grpc 4317), prometheus (scrape api/worker/realtime node exporters); processors: batch, memory_limiter, attributes (drop PII keys: email, name, ip); exporters: otlp/prometheus (→ prometheus receiver), loki (logs), debug
prometheus	StatefulSet ×1 (or managed), 24 h retention in-memory + remote-write to long store for SLO math	scrape configs for all factory namespace services (kubernetes-pod relabel by prometheus.io/* annotations) + blackbox for /healthz/ready
loki	StatefulSet ×1 (single-tenant), 30 d retention	scrape_configs: read the collector's loki export (structured JSON already — no grok needed); labels: job, namespace, pod (low cardinality; user_id never a label — NFR-P2)
grafana	Deployment ×1, admin via external secret, no public ingress (SSO via the platform's own OIDC or basic-auth behind VPN)	dashboards as code: infrastructure/grafana/dashboards/*.json (the 8 from NFR §7.2), provisioned via grafana-datasources CM + dashboard provider
alertmanager	Deployment ×1	routes: critical → paging (PagerDuty/opsgenie webhook), warn → chat (Slack webhook); inhibition: SLO burn 90 % inhibits API latency

App-side (all four apps): otel SDK initialized from OTEL_ENABLED (default false in self-host — NFR-H5; true in cloud/staging); instrumentation: auto (http, pg via prisma middleware, redis, ws) + manual spans per NFR §7.1 (services + repositories + queue jobs). Logs: pino → stdout (json) → collector (docker/k8s log driver) → Loki.

9.2 Dashboards (8, as code)

The 8 dashboards from NFR §7.2 (Overview/SLO, API, Realtime, Queues, Search, Storage, AI, Capacity) live in infrastructure/grafana/dashboards/ as version-controlled JSON — a dashboard change is a PR. The Overview panel row: SLO burn bar (28 d), API p95, error rate, rps, WS connections, queue depth max, consumer lag, backup-verified age.

9.3 Alert rules (full initial set, infrastructure/prometheus/rules/)

#	Rule	Expression (abbrev.)	For	Severity
1	API error rate	sum(rate(http_server_request_count{code=~"5.."}[5m])) / sum(rate(http_server_request_count[5m])) > 0.01	5 m	critical (page)
2	API latency	histogram_quantile(0.95, sum(rate(http_server_request_duration_bucket[10m])) by (le)) > 0.5	10 m	warn
3	SLO burn (budget)	error-budget-consumed{window="28d"} > 0.5 / > 0.9	1 h	warn / critical
4	Queue backlog	queue_depth{queue="search"} > 1000 or queue_oldest_job_age_seconds{queue="search"} > 300	5 m	warn
5	Webhook lag	queue_oldest_job_age_seconds{queue="webhooks"} > 1800	5 m	warn
6	Dead letters	increase(queue_dead_letter_total[1h]) > 0	1 h	warn
7	WS saturation	ws_connections{pod} > 8000 or ws_update_duration_p95 > 0.4	5 m	warn
8	Consumer lag (events)	events_consumer_lag_seconds > 30	5 m	warn
9	Disk	node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.2	5 m	warn → critical < 0.1
10	Backup stale	time() - backup_last_verified_timestamp_seconds > backup_schedule_seconds * 2	1 h	critical
11	Certificate expiry	probe_ssl_earliest_cert_expiry < 14*24*3600	1 h	warn
12	Realtime down	absent(up{job="realtime"}) or blackbox fail ×3	1 m	critical
13	Postgres replication/lag (managed: failover)	pg_replication_lag_seconds > 30	5 m	warn
14	Virus detections	increase(files_scanned_virus_total[1h]) > 0	1 h	warn
15	AI budget	ai_tokens_used{workspace} / ai_tokens_budget{workspace} > 0.9	1 h	warn (→ workspace admin channel)
16	Redis memory	used_memory / maxmemory > 0.85	5 m	warn
17	OpenSearch cluster	opensearch_cluster_status_red == 1	2 m	critical
18	Pod restarts	increase(kube_pod_container_status_restarts_total[1h]) > 3	1 h	warn

Routing: critical pages on-call (24×7 in cloud; in self-host, routes to the operator's webhook); warn to the ops channel; every alert's annotation links the runbook section (this file + docs/ops/runbooks/).

9.4 Observability rollout order

1. Phase 0: pino structured logging + /healthz endpoints + otel SDK wired behind OTEL_ENABLED=false (zero cost, flip later).
2. Phase 7 (realtime): WS metrics + dashboards 1, 3.
3. Phase 15: full stack in staging → dashboards 1–8 + all 18 rules → SLO burn starts (28 d warmup) → prod rollout with NFR-H5 opt-in.

10. SSO (SAML / OIDC) setup — TASK-0131

The SSO protocol surface (/saml/*, /sso/*; /scim/* later) is served by the api service, mounted outside /api/v1 — the IdP (or the user's browser mid-handshake) talks to it directly. Reference: docs/api/sso.md (endpoints + validation matrix) and docs/security/security-architecture.md §2.5 (threat model + lockout runbook).

10.1 Origin topology (the one thing that must be right)

The protocol URLs (SP entityID, ACS, metadata, OIDC redirect_uri, /sso/start) must name the origin that serves them:

Topology	SSO_ORIGIN	Why
Single origin (reference nginx / ingress fronts web + api under ONE host)	leave empty (default)	WEB_ORIGIN serves both — nothing extra needed
Two origins (web :3000 + api :4000 dev; or separate public hosts)	the API's public origin	the IdP POSTs to <SSO_ORIGIN>/saml/acs; without it the ACS 404s on the web origin

Landing URLs (/login?error=…, /w/default) always use WEB_ORIGIN regardless. SSO_ORIGIN is in .env.example (root + apps/api); it is required for the two-origin dev/E2E layout (the E2E suite assumes SSO_ORIGIN=http://localhost:4000).

Routing. The reference infrastructure/nginx/nginx.conf already carries the two locations the protocol surface needs (mirror them on ingress-nginx as path rules to the api backend):

location /saml/ { limit_req zone=api burst=60 nodelay; proxy_pass http://api_upstream; }
location /sso/  { limit_req zone=api burst=60 nodelay; proxy_pass http://api_upstream; }

Without them the paths fall through to location / (Next.js) and 404. Note the paths are NOT under /api/ — they are browser bindings, not the JSON v1 API.

10.2 SAML — configuring an org's provider

1. In your IdP (Okta / Entra ID / ADFS / …), create a SAML SP app with:
   • ACS URL = <SSO_ORIGIN||WEB_ORIGIN>/saml/acs
   • SP entity ID = <SSO_ORIGIN||WEB_ORIGIN>/saml/metadata/<org-slug> (the SP publishes its metadata at the same base path for metadata-exchange IdPs).
2. In the IdP, map the user's email to the SAML mail attribute (optionally displayName), and set the NameID format to emailAddress — or keep the IdP's NameID and adjust config.nameIdAttribute / config.emailAttribute on the org provider.
3. In the org admin (Integrations → SSO tabs), create the provider: paste the IdP's signing certificate (PEM), set ssoUrl (the IdP's SSO login URL), and — for a green test-connection — the IdP's metadata URL. The cert is stored AES-256-GCM encrypted; the API returns only certificateConfigured afterwards (reveal-once).
4. Test connection (fetches the IdP metadata; green panel = usable descriptor), then optionally enforce (see 10.4).

Certificate rotation (IdP key rollover): paste the NEW cert on the provider — the validator accepts the current or previous cert, so there is a grace window without a login blackout.

10.3 OIDC — configuring an org's provider

1. In your OIDC IdP, register a confidential client with:
   • Redirect URI = <SSO_ORIGIN||WEB_ORIGIN>/sso/oidc/<orgId>/callback
   • scopes openid email profile
   • a public JWKS endpoint (the default <issuer>/.well-known/jwks.json is used unless jwksUri is set) and /.well-known/openid-configuration discovery.
2. In the org admin, create the provider: issuer, client id, the authorization + token endpoints (from the discovery document), and paste the client secret (reveal-once, encrypted at rest).
3. Test connection (fetches discovery + JWKS), then enforce if desired.

The handshake is authorization-code + PKCE (S256) with single-use state/nonce (Redis, 10 min); the id_token is RS256-verified against the JWKS (iss/aud/exp/nbf/nonce); the IdP access_token is discarded, never stored.

10.4 Enforce mode + lockout runbook

• On: PATCH /api/v1/orgs/:id/sso/enforce { "enforce": true } (admin UI: the toggle). Every member of the org must now sign in through the IdP; password logins return 403 { reason: "sso_required", orgs: […] } and the login page offers "Continue with SSO".

• The SSO path is never blocked by enforcement. Recovery if the IdP is down or a cert was rotated away:

  1. The owner (or any member with IdP access) signs in via SSO once the IdP is reachable — /sso/start/<org-slug>;
  2. or an operator with a pre-enforcement session (API token or the still-valid refresh cookie) calls PATCH …/sso/enforce { "enforce": false }.

  There is deliberately no elevated service-account credential for this: the enforcement state is plain data (SsoProvider.enforce + the Organization.policy.ssoEnforced mirror), so no second trust path is needed. Removing the provider (DELETE …/sso) also clears enforce.

• Self-host smoke test (mock IdP): the repo ships a full mock SAML IdP and OIDC issuer used by the E2E suite (tests/e2e/mocks/saml-idp.ts, tests/e2e/mocks/oidc-issuer.ts). To exercise SSO locally without a real IdP, run the dev stack with SSO_ORIGIN=http://localhost:4000 and drive tests/e2e/specs/sso.spec.ts (pnpm --dir tests/e2e exec playwright test sso.spec) — it stands up a per-run self-signed IdP, creates an org + provider via the admin API, and walks the full browser round trip (SAML + OIDC, tamper matrix, enforce + recovery, admin UI).

10.5 Observability

SSO logins audit sso.login.saml / sso.login.oidc (ip + UA) and every rejection audits sso.login_failed { reason } — the reason vocabulary is the validator's matrix (bad_signature, cert_mismatch, replayed, audience_mismatch, expired, …; see the API doc). Dashboards can slice logins by method via the audit rows; a spike in sso.login_failed{reason=cert_mismatch} is the early signal of an unmanaged IdP key rollover.

10.6 SCIM 2.0 — connecting an IdP (Okta / Entra) — TASK-0132

SCIM is the provisioning complement to SSO: the IdP creates/updates/ deactivates users in Factory over HTTPS. Contract details (auth, shapes, filters, errors, deactivation semantics) are in docs/api/scim.md.

Origin note (same rule as §10.1). The protocol surface is served by the API outside /api/v1, so the IdP must point at SSO_ORIGIN (falling back to WEB_ORIGIN) — the same origin that serves /saml/* and /sso/*. In the two-origin topology, put location /scim/ next to location /saml/ in the API vhost. If you only run the single-origin setup (nginx fronts both apps at one origin), the default works out of the box.

Steps (org admin, via the Integrations → SCIM card):

1. Enable SCIM. Choose the user email domain if your IdP sends @-less usernames (Okta's "username" is typically the full email — leave the domain blank then). Pick the default role (Member or Guest — SCIM can never grant more) and whether new users auto-join the org's workspaces.
2. Copy the bearer token. It is shown once (stored only as a SHA-256 hash). Paste it into the IdP as the SCIM API credential.
3. Point the IdP at the endpoint <origin>/scim/v2 with the bearer token. Per-IdP config:
   • Okta — App → People (or the provisioning integration) → Provisioning: Type SCIM 2.0, Base URL <origin>/scim/v2, Auth Bearer (API token field = the scim_… token). Map userName←email (or username + the org's email domain), active←lifecycle status.
   • Microsoft Entra ID — Enterprise application → Provisioning → SCIM-based → Tenant URL <origin>/scim/v2, Auth Bearer with the scim_… token.
4. Test the connection from the SCIM card (paste the token back — the server never stores it; it just resolves it against the org). Green = the token is valid and reaches this deployment.
5. Verify a test user. Start provisioning a single user in the IdP, confirm the account exists here and can log in (the SCIM payload's password is honored, or set one from the IdP side), then provision the rest.

Day-2 operations

• Rotate the token (SCIM card → Rotate token) when it leaks or on a schedule — the old token stops working immediately (Redis index swap). Re-copy the new reveal-once token into the IdP.
• Disable SCIM stops all provisioning at once (token → 401); provisioned accounts are kept.
• Deactivation is not deletion. When the IdP deactivates a user (replace active:false or DELETE), the Factory account becomes SUSPENDED: login and in-flight sessions are blocked (403 account_suspended), memberships are preserved. Re-activation restores access. This is a recorded decision (see docs/api/scim.md §Deactivation).
• Audit — every SCIM operation writes an audit row (scim.user.created/updated/deactivated, scim.group.created/updated/ deleted) with actorType: INTEGRATION (actor = the org); passwords are never audited.
• Groups map to Teams in the org's primary workspace; deleting a group removes the team only (members keep their workspace membership).

10.7 Org policies & data retention — TASK-0133

The org admin console (/org/{orgId}/admin) exposes the org policy and retention policies. Contract details (fields, bounds, error reasons) are in docs/api/org-admin.md; the threat model is in docs/security/security-architecture.md §11.3.

Enabling 2FA / SSO enforcement (per org, console → Policies):

• Require 2FA — once on, login for org members without TOTP returns 202 twoFactorRequired (a 5-minute single-use TOTP step); the password alone no longer logs in. Enable it before you need it — members must be able to reach the login page to enroll.
• Enforce SSO — only meaningful once a SAML provider is configured (§10.2). Non-SSO logins for org members get 403 sso_required; users who are NOT members of the org are unaffected. Lockout risk: the org OWNER is gated too — if the IdP goes down, the owner cannot log in with a password. The recovery path is the SSO recovery flow (§10.4) or a direct DB update of the org policy — know which you can run before you enable it.

Retention recommendations (self-hosters):

• audit_logs is HOLD by default — the platform keeps audit rows for the retention default (400 d) and purges nothing while HOLD is on. If your compliance posture needs longer, set an explicit PURGE with a higher retainDays (HOLD is a floor, not a ceiling). Setting a PURGE retention is deliberate and is itself audited.
• Trash (PURGE 30 d) and comments (PURGE 90 d) are the safe defaults; lower retainDays if storage is tight, but remember the purge is batched, audited (retention.purged with per-batch counts) and cannot cross orgs (every query is org-scoped by construction).
• Files default to retainDays: 0 = no purge (uploads outlive the page they are attached to until the org opts in). Opt in only if your file storage is the bottleneck; the byte count per purge is in the run report and the audit row.
• The daily cron mode is gated by WORKER_RETENTION_EXECUTE (default off = dry-run only): set it on to let the nightly job actually purge. Admin "Run now → Execute" in the console purges regardless of the flag (the flag scopes the unattended job, not the deliberate admin action). Dry-run writes lastDryRun, execute writes lastRun on the policy row — both are visible on the console.
• Worker dependency: retention runs are async worker jobs (maintenance queue). If the worker is down, "Run now" returns 202 processing and the report never appears — check the worker health endpoint before troubleshooting a "stuck" run.