phase-13-api.mddocs/tasks/phase-13-api.md← tasks79,305 B← Back to the app

Phase 13 — Public API

Purpose: Task specs for Phase 13 — the public REST API v1: formalization of the API surface (auth, scopes, rate limits, idempotency, versioning, error codes), the OpenAPI contract + interactive docs, the API-keys management UI, OAuth2 (authorization code + PKCE), production webhooks (signing, retries, DLQ, event-catalog audit, incoming webhooks), and the in-app developer hub.

Phase exit criteria (ROADMAP.md, Phase 13):

OpenAPI contract matches live endpoints (CI check).

Webhook retry with backoff proven.

API-key auth E2E-tested.

Task range: TASK-0127 … TASK-0130 (4 tasks — this agent owns exactly TASK-0127..TASK-0130; TASK-0096..0120 are owned by other agents).

Task Table

ID	Title	Estimate
TASK-0127	Public REST API v1 formalization (auth, scopes, rate limits, idempotency, versioning, OpenAPI + docs)	L
TASK-0128	API keys management UI + integration E2E	M
TASK-0129	OAuth2 authorization code + PKCE	M
TASK-0130	Webhooks production (signing, retries, DLQ, event catalog, incoming webhooks) + developer hub	L

Spec conflicts & decisions recorded (per AGENTS rule 29)

Per-key rate-limit default: the task brief says default 100/min for API keys, but docs/api/authentication.md §11 (published convention, Phase 1 artifact) says "Per API key: 600 req/min, or the key's rateLimit if set". This phase follows the published docs (600/min default) because docs are the external contract and AGENTS.md says to preserve backward compatibility; the brief's 100/min is treated as a product-tuning decision for a later change. The key's rateLimit field always wins when set.
Webhook signing secret storage: the task brief says the raw signing secret is stored AES-GCM encrypted with MASTER_ENCRYPTION_KEY, while the current Prisma model Webhook.secretHash stores a hash. Both are needed and not contradictory: secretHash (SHA-256) is kept for verification of incoming webhooks only when the raw secret is unrecoverable — the production path stores the AES-GCM-encrypted raw secret (new encrypted field) so that signature verification can recompute HMACs, and secretHash is retained for the incoming-webhook replay/verification path. Decision: add Webhook.secretEncrypted Bytes (migration in TASK-0130); secretHash stays populated for all rows (legacy + incoming). Recorded here, not guessed.

Phase Completion Checklist

[ ] All /api/v1 endpoints authenticated by API key (Bearer fk_… → hash lookup → workspace + scopes) with per-key scope enforcement (READ key on POST/PATCH/DELETE → SCOPE_DENIED per the error table's FORBIDDEN reason api_key_scope_write — the machine code surfaced is FORBIDDEN with that reason; "SCOPE_DENIED" is the documented alias, see TASK-0127) and per-key rate limits (api_keys.rateLimit, default per docs §11) with X-RateLimit-* headers.
[ ] Idempotency: Idempotency-Key on POSTs memoized 24 h in Redis; replayed responses carry Idempotency-Replayed: true.
[ ] Versioning + deprecation policy documented and enforced (Deprecation / Sunset headers; /api/v1 stable); error codes match the docs/api/authentication.md table on every public route (contract-tested).
[ ] OpenAPI 3.1 openapi.yaml covers the full v1 surface; CI contract test asserts spec paths ⊆ live routes and live public routes ⊆ spec (bidirectional); served at /api/v1/docs (swagger-ui) with a YAML download link.
[ ] Developers settings UI: create (name, scopes, expiry) with raw key shown once in a copy dialog + warning; list (prefix, scopes, last used, expiry, active); revoke (immediate 401); audit events api_key.created / api_key.revoked.
[ ] API integration E2E green: key created → curl create page → GET → PATCH; READ-scope key POST rejected.
[ ] OAuth2: full authorization-code + PKCE flow (consent screen, single-use 10-min code, token + refresh, refresh rotation) against the registered-apps config in integrations.config.oauthApps; E2E green; refresh rotation proven.
[ ] Webhooks: HMAC-SHA256 X-Factory-Signature: t=<ts>,v1=<hmac> with AES-GCM-encrypted signing secret; retry with exponential backoff 8 attempts → DLQ + delivery log (webhook_deliveries) + resubmit-failed endpoint; event-catalog integration test proves every catalog event fires; incoming webhooks (POST /api/v1/webhooks/incoming/:id) with signature verification + replay protection.
[ ] Developer hub in-app (overview + auth examples, keys, webhooks + delivery inspector with test send, event explorer with schemas + examples, per-resource curl examples, link to /docs/api).
[ ] Webhook E2E green: new key → create page via curl → signed page.created webhook received ≤ 10 s → signature recomputed and verified in the test → webhook URL mutated to dead → retries visible in the delivery log.

TASK-0127

Title

Public REST API v1 formalization: auth, scopes, rate limits, idempotency, versioning, OpenAPI + interactive docs

Objective

Make the existing /api/v1 endpoints the formal public surface: API-key authentication with scope enforcement, per-key rate limiting, POST idempotency, a versioning/deprecation policy, canonical error codes on every public route, and a machine-checked OpenAPI contract served interactively.

Background

Phase 2+ built /api/v1 endpoints (pages, databases, records, search, files, comments, workspaces — see docs/api/pages.md, docs/api/databases.md, docs/api/blocks.md, docs/api/workspaces.md) consumed by the web app with JWTs. docs/api/authentication.md already defines the API-key format (fk_ prefix, READ/WRITE/ADMIN scopes, X-Factory-Workspace header, per-key rate limit), the canonical error envelope + code table (§9.1), rate-limit headers (§11), and idempotency semantics. The Prisma models ApiKey (keyHash, prefix, scopes, rateLimit, lastUsedAt, expiresAt, revokedAt) and Integration (holds oauthApps config, TASK-0129) exist. This task does not add resources — it formalizes, secures, and contracts the surface.

Requirements

API-key auth middleware (apps/api/src/modules/apikey/apikey.guard.ts + apikey-auth.middleware.ts): Authorization: Bearer fk_… → SHA-256 hash → api_keys lookup (single indexed query, Redis-cached positive hits 60 s with invalidation on revoke) → resolve workspace (key's workspaceId, or X-Factory-Workspace if the key is org-admin scope — v1: key is bound to its workspace; a mismatched X-Factory-Workspace → 401 UNAUTHORIZED) → attach actor = {type:"API_KEY", id, scopes, workspaceId}. Expired key → 401 {reason:"api_key_expired"}; revoked → 401 {reason:"api_key_revoked"} (audit api_key.auth_failure, throttled per key 1/h). Update lastUsedAt (debounced 60 s per key to avoid write amplification).
Scope enforcement (decorator @ApiKeyScopes("READ"|"WRITE"|"ADMIN")): read endpoints require READ; mutations require WRITE; member/permission/api-key/OAuth/app management require ADMIN. A READ key on POST/PATCH/DELETE → 403 FORBIDDEN with details:[{reason:"api_key_scope_write"}] (documented alias: SCOPE_DENIED — the public name for this reason in docs/api/authentication.md, same code). Endpoints that require a user identity (invitations as a user, PATCH /users/me) reject API keys with 403 {reason:"requires_user_credentials"} (already specified in docs §7).
Per-key rate limits: Redis sliding window per key; limit = api_keys.rateLimit if set, else 600/min (decision recorded in header above re: the brief's 100/min); on breach 429 RATE_LIMITED + Retry-After + X-RateLimit-Limit/Remaining/Reset on every public response (existing global middleware extended, not replaced).
Idempotency: Idempotency-Key header (UUID or idk_-prefixed, ≤128 chars) on public POST routes → Redis memo key idem:{workspaceId}:{keyId}:{hash(key+route+body)} storing status + body for 24 h; concurrent duplicate → the in-flight result is awaited (Redis lock, 10 s); replay → stored response + Idempotency-Replayed: true header. Missing header on non-POST → ignored; on POST without the header → normal behavior (opt-in, per docs).
Versioning + deprecation policy (docs/api/versioning.md): /api/v1 is stable; breaking changes only in a new major (/api/v2); deprecated routes carry Deprecation: true + Sunset: <RFC 1123 date> headers (minimum 6 months sunset) and appear in the OpenAPI as deprecated: true. Middleware deprecation.middleware.ts reads a per-route config map (initially empty — the mechanism + one documented example: POST /api/v1/workspaces/:id/pages/duplicate-legacy placeholder is not added; instead the config map format is tested with a synthetic route in tests).
Error-code contract: every public route responds in the canonical envelope (global exception filter audit): a route test walks the live router and asserts each route's declared errors ⊆ the §9.1 table (new codes added in other phases, e.g. AI_BUDGET_EXCEEDED, are in the table by then).
OpenAPI: docs/api/openapi.yaml (3.1) for the full v1 surface — pages, blocks, databases, records, relations, search, files, comments, workspaces, members, invitations, api-keys, oauth, webhooks, AI endpoints, forms, automations (everything publicly exposed). Decision: hand-maintained with CI enforcement (generation from NestJS decorators would require annotation discipline across 40+ controllers; hand-maintained + the bidirectional contract test keeps it honest). CI contract test (tests/integration/api-contract/openapi.contract.test.ts): (a) every spec path+method has a live route; (b) every live public route (router enumeration, excluding internal/SSR routes) has a spec entry; (c) spec parameters ⊆ route parameters. Served at GET /api/v1/docs (swagger-ui, static bundle, no network dependency) + GET /api/v1/docs/openapi.yaml (download link in the UI header).
Request logging: public responses carry X-Request-Id (already global) and server header is removed (no framework fingerprint).

Acceptance Criteria

Integration: Bearer fk_… for a valid key authenticates GET /api/v1/workspaces/:id/pages (200) and PATCH on a page (200) for a WRITE key; the request's audit trail records actorType: "API_KEY".
Integration: a READ-scoped key POSTing to create a page receives 403 FORBIDDEN with details[0].reason === "api_key_scope_write"; an ADMIN endpoint (api-keys list) with a WRITE key → same 403; a user-identity endpoint with any key → 403 {reason:"requires_user_credentials"}.
Integration: revoked key → immediate 401 {reason:"api_key_revoked"} (cache-busting asserted: revoke, then request within 60 s → 401, proving invalidation beats the 60 s cache).
Integration: a key with rateLimit: 10 receives 10×200 then 429 with Retry-After and correct X-RateLimit-* headers; a key without rateLimit hits 600 (assert via header X-RateLimit-Limit: 600).
Integration: same Idempotency-Key + body POSTed twice → second response identical + Idempotency-Replayed: true; concurrent (parallel) duplicates → exactly one side effect (assert one page created); different body with same key → two side effects (body is in the memo hash).
Contract: the OpenAPI CI test fails when (a) a live public route is missing from the spec (add a temporary route in the test fixture) or (b) the spec lists a nonexistent path; both directions asserted; /api/v1/docs returns 200 HTML (swagger-ui) and /api/v1/docs/openapi.yaml returns valid YAML (parsed in the test).
E2E: curl sequence in a real browser-free API session: create key via JWT admin API → use it: POST /api/v1/pages → GET the page → PATCH title → all 2xx (this is the "API-key auth E2E-tested" ROADMAP exit check, with the full scenario extended in TASK-0128).

Backend Changes

apps/api/src/modules/apikey/: apikey-auth.middleware.ts, apikey.guard.ts (scope decorator + guard), apikey.service.ts (hash lookup, caching, lastUsed debounce, revocation events), apikey.module.ts global registration for the api/v1 prefix.
apps/api/src/common/idempotency.ts: idempotency middleware + Redis lock/memo (also used by TASK-0126 generation; export as a reusable utility).
apps/api/src/common/deprecation.middleware.ts + route config map.
Swagger: apps/api/src/docs/openapi.yaml (canonical location; docs/api/openapi.yaml is a symlink/copy — decision: canonical in docs/api/, API serves from there), docs.controller.ts (swagger-ui static + YAML), public route enumeration helper (used by the contract test).
Rate-limit middleware extended for per-key buckets (existing user/IP buckets unchanged).

Frontend Changes

None in this task (the web app keeps using JWTs; the Developers UI is TASK-0128). Internal API clients unchanged — API-key middleware must not intercept JWT requests (both auth paths coexist; a request authenticates by exactly one credential type).

Database Changes

None (models exist). Redis keys: apikey:pos:{keyHash} (60 s), idem:{…} (24 h), rate-limit buckets — documented in docs/database/ (add docs/database/redis-keys.md section).

API Changes

No new resources. Behavioral changes to the public surface: API-key auth accepted on all v1 routes; X-RateLimit-* on every response; Idempotency-Key honored on POSTs; Deprecation/Sunset headers per config; new routes GET /api/v1/docs, GET /api/v1/docs/openapi.yaml.

Tests

Integration: the scenarios in Acceptance Criteria (auth, scope matrix across all three scopes × read/mutate/admin/user-identity endpoints, revocation immediacy, rate-limit headers, idempotency memo/lock/replay, deprecation header on a synthetic route).
Contract: tests/integration/api-contract/openapi.contract.test.ts (bidirectional) + error-envelope walk (every route's 4xx/5xx in the canonical table).
E2E: curl key-creation → create → get → patch (the exit-criteria scenario).
Unit: hash normalization (whitespace, Bearer case), idempotency key hashing, deprecation config parsing.

Security

Only keyHash is stored/compared (SHA-256; the raw fk_ key is never persisted — verified by a schema test asserting no plaintext column and that creation returns the raw key exactly once).
API-key requests act as the workspace (audit actorType: "API_KEY", per docs §7); user-only endpoints reject keys (403, tested).
Rate limiting prevents brute force on key guesses (key lookup is a single hash index probe; timing does not leak existence — constant-time comparison via hash lookup only).
Idempotency memo is scoped by credential (no cross-key replay).
/api/v1/docs is public by design (public API documentation) but the swagger UI makes no live calls (static spec only) — no data leak; documented in docs/security/security-architecture.md.

Documentation

docs/api/versioning.md (new): versioning + deprecation policy; docs/api/authentication.md §7/§11 updates (scope reason alias SCOPE_DENIED, idempotency pointer, docs endpoint); docs/api/openapi.yaml (full spec); docs/database/redis-keys.md; docs/security/security-architecture.md (API-key threat model).

Dependencies

Phases 0–11 (all v1 routes exist), Redis, ApiKey model + admin key-management endpoints (docs §7.1–7.3 implemented in Phase 2 workspace module — if any of the three key-management endpoints is missing, this task lands them first as a prerequisite, since TASK-0128's UI needs them).
TASK-0126's idempotency need is satisfied by the utility this task exports (no ordering requirement; if TASK-0126 lands first, it ships a local copy that this task replaces).

Definition of Done

[ ] API-key middleware + scope guard + per-key rate limits live on the whole v1 surface; auth failure paths audited.
[ ] Idempotency (memo + lock + replay header) and deprecation middleware implemented + tested.
[ ] docs/api/openapi.yaml complete for the v1 surface; bidirectional CI contract test green; swagger-ui + YAML download served.
[ ] Versioning doc written; error-envelope contract test green.
[ ] Integration + E2E + contract tests green; lint/typecheck green; docs updated.

TASK-0128

Title

API keys management UI + integration E2E

Objective

Ship the Developers section of workspace settings (create / list / revoke API keys) and prove the full key lifecycle E2E: key created → create page via curl → GET → PATCH, and READ-key POST rejection.

Background

TASK-0127 formalized API-key auth. The key-management endpoints (POST/GET/DELETE /workspaces/:workspaceId/api-keys per docs/api/authentication.md §7.1–7.3) exist (or are landed by TASK-0127's prerequisite). The workspace settings shell from Phase 2 has a "Developers" placeholder (the Integrations placeholder exists; Developers is added as a sibling tab). ApiKey fields: name, prefix, scopes (READ|WRITE|ADMIN), rateLimit?, lastUsedAt?, expiresAt?, revokedAt?, createdById. The raw key (fk_live_…) appears only in the create response.

Requirements

Create dialog (workspace settings → Developers → "Create API key"): fields name (required, ≤64 chars, workspace-unique per docs — duplicate name → 409), scopes (multi-checkbox, at least one; ADMIN implies WRITE implies READ — the UI shows the implication and stores the minimal selected set), expiry (optional date picker, minimum now+1h, maximum +1 year), rate limit (optional number 1–10000 req/min, placeholder "600 (default)"). On success: a copy dialog showing the raw key exactly once, monospace, with a copy button (clipboard API + fallback), an explicit warning "This key will not be shown again. Store it securely.", a "I have stored it" confirmation that dismisses the dialog permanently (the raw key is not re-fetchable — the dialog is the only place it exists in the UI state; closing the tab loses it).
Key list: table rows prefix (fk_live_ab12…), name, scopes badges, last used (relative time, "never"), expiry ("never" or date, red when < 7 d, grey when passed → "expired"), status (Active / Revoked / Expired); actions: Revoke (inline confirm: "Revoke this key? Requests will fail immediately with 401."), copy prefix; empty state ("No API keys yet — create one to connect integrations") with a create CTA.
Revoke: DELETE → row updates to Revoked state (optimistic, server confirms); the raw key is dead immediately (TASK-0127 cache invalidation).
Permissions: tab visible to workspace Owner/Admin only (server enforces 403 for others; the UI hides the tab for Members/Guests — server is authoritative).
Audit: api_key.created (name, scopes — never the raw key) and api_key.revoked (already required by docs §7; this task verifies the emitters exist and adds the missing one if absent).
Copy helpers: each row's expand shows a curl example for that key (prefix masked, Authorization: Bearer <your key> placeholder).

Acceptance Criteria

E2E (the exit-criteria scenario): Owner creates a key (name "CI", scopes READ+WRITE, no expiry) via the UI copy dialog → the test reads the raw key from the dialog DOM → curl POST /api/v1/workspaces/:id/pages with the key → 201 → GET /api/v1/pages/:id → 200 with the created title → PATCH title → 200 + verified via GET.
E2E: a second key with READ scope only → POST create page via curl → 403 with reason "api_key_scope_write"; the same key GETs successfully.
E2E: revoke the WRITE key from the UI → immediate curl GET with it → 401 {reason:"api_key_revoked"}; the row shows Revoked; audit log contains api_key.created (×2) and api_key.revoked (×1) — asserted via the audit API.
UI: the raw key appears in exactly one DOM location after creation (the copy dialog); dismissing it and reopening settings never shows it again (assert on re-render); expiry < 7 d renders red with a tooltip.
Integration: a Member-role user → GET /workspaces/:id/api-keys → 403; the Developers tab is absent from their settings DOM.
Unit: scope-implication logic (selecting ADMIN shows/derives WRITE+READ; stored set is minimal), curl-example generator (correct header, masked prefix).

Backend Changes

Verify/complete the three key-management endpoints against docs §7 (creation returns raw key once; list never returns raw key; delete idempotent + revokes immediately). Audit emitters api_key.created/api_key.revoked present.
lastUsedAt debounce from TASK-0127 surfaces in the list (no backend change beyond what TASK-0127 ships).

Frontend Changes

apps/web/app/(workspace)/[workspaceSlug]/settings/developers/page.tsx (or tab component DevelopersTab.tsx in the existing settings layout), ApiKeyCreateDialog.tsx (form + scope implications), ApiKeyRevealDialog.tsx (raw key + copy + warning + confirm), ApiKeyTable.tsx (rows, status badges, revoke confirm, curl expand), ScopesBadge.tsx; settings nav entry "Developers" (Owner/Admin only in nav, hidden otherwise).
Loading (skeleton table), empty state, error toasts (409 duplicate name surfaces the field error), a11y (dialog focus trap, copy button announced, table with scope descriptions accessible via aria-describedby).

Database Changes

None.

API Changes

None new (uses §7.1–7.3). If the endpoints required backend fixes in Backend Changes, they match the documented contract exactly.

Tests

E2E: the two lifecycle scenarios (full curl roundtrip; READ rejection) + revoke immediacy + audit assertions + tab-visibility test.
Integration: permission 403 for Member; duplicate-name 409; raw-key-not-in-list assertion (response schema test).
Unit: scope implications, curl generator, relative-date formatting for lastUsed/expiry.

E2E result (TASK-0128)

developers.spec.ts (4 tests: desktop owner lifecycle + curl roundtrip + revoke + audit; desktop member tab-absence; mobile owner lifecycle; mobile member tab-absence) at the default workers: 2 config, repeated until stable under the FINAL code (fixes A+B+C+D server/client + D2 test-side, per C-13.5/C-13.6): run 1 = 3/4 (the double hard-goto re-render step — D2 above; run 1 ran under A/B/C only, before D/D2). Runs 2, 3, 4 = 4/4 each (31–45 s each). DB signal confirms the root fix, not a test band-aid: post-fix window shows 0 auth.refresh_reuse_detected (all 39 are the pre-fix baseline) and 2 auth.refresh_stale_retry (the new benign path firing on the cross-tab rotation race), and 0 live refresh tokens stranded on a revoked session (the pre-fix orphan defect class is gone). The pre-fix solo runs were already clean (5/5); the flake required the parallel fresh-context bootstraps that the single-flight + stale-grace + cookie-clear fixes now absorb. Diagnostic probe specs (zz-state.spec.ts / zz-owner2.spec.ts) were deleted before commit.

Security

Raw key handling: only in the 201 response body and the reveal dialog; never in logs, never in the list, never in audit; the reveal dialog state is not persisted (no sessionStorage).
Revoke is immediate (cache invalidation from TASK-0127 tested here end-to-end).
The Developers surface requires Owner/Admin (403 tested); the tab hiding is UX only (AGENTS rule 19 — never trust the frontend).

Documentation

docs/api/authentication.md §7 (no changes expected — this task is its UI realization; deviations, if any, recorded in this file); docs/ui/ Developers tab spec.

Dependencies

TASK-0127 (auth + revocation invalidation + lastUsedAt), Phase 2 settings layout, Phase 3 (pages API for the E2E), audit module.

Definition of Done

[x] Developers tab: create (with raw-key-once reveal dialog + warning), list (all fields + statuses), revoke (immediate) — Owner/Admin only. (E2E owner lifecycle: create→reveal→POST/GET/PATCH via curl→READ 403 api_key_scope_write→revoke→api_key_revoked; Member 403 + tab hidden; audit api_key.created×2 + api_key.revoked×1 asserted via the audit API.)
[x] Both E2E lifecycle scenarios green (including curl create → GET → PATCH and READ-POST 403); audit events asserted. (4/4 × 3 consecutive runs at workers: 2, see E2E result line.)
[x] Integration + unit tests green; a11y + loading/empty/error states done; docs updated. (integration: api-keys 18/18 + auth 10/10 (9/10 — the 1 rate-limit test is an environment-bound pre-existing failure, reported in C-13.6); unit: 214/214 web incl. api-keys 15, auth-refresh 7, auth-store 4; docs/ui/developers.md.)

TASK-0129

Title

OAuth2 authorization code + PKCE for third-party integrations

Objective

Implement the OAuth2 authorization-code flow with mandatory PKCE for third-party apps: registered-app config in integrations.config.oauthApps, the consent screen, single-use 10-minute codes, the token endpoint with refresh rotation, and the revoke endpoint — per the contract in docs/api/authentication.md §8.

Background

Docs §8 specifies the flow, endpoints, and error codes; Integration (workspace-scoped, config Json) is where oauthApps live: an org/workspace registers an app as {clientId, clientSecret? (hashed; absent = public client, PKCE mandatory), name, logoUrl?, redirectUris[], scopes[], grantTypes:["authorization_code","refresh_token"], createdAt}. JWTs issued here carry client_id + scope (no fixed workspaceId); each request then carries X-Factory-Workspace validated against the user's memberships and the scope grant (docs §8.2). Audit events: oauth.code_issued, oauth.token_issued, oauth.token_revoked. Refresh rotation follows §1.2 (opaque rotating refresh tokens, reuse detection).

Requirements

Registered apps: management endpoints (workspace Owner/Admin, JWT or ADMIN-scope API key): POST/GET/PATCH/DELETE /api/v1/workspaces/:id/oauth/apps (create returns clientId + clientSecret once, same reveal-once pattern as API keys). clientId = cli_-prefixed CUID; secrets stored SHA-256-hashed. PKCE: public clients (no secret) must present code_challenge (S256 only); confidential clients may omit it.
GET /oauth/authorize (Next.js route in apps/api or apps/web — decision: served by the API as an HTML page so it works pre-login redirect and matches docs "render consent screen, login first if anonymous"): validates response_type=code, client_id, redirect_uri (exact match against the registered allow-list; mismatch → error=invalid_request 400 page), scope (⊆ app's offered scopes, else INVALID_SCOPE), state (echoed), code_challenge(+method). Anonymous → redirect to /login?next=<encoded authorize url> (login flow from Phase 1; on return, resume consent). Consent screen: app name + logo, requested scopes as checkboxes with human descriptions (read:pages "View pages", write:pages "Create and edit pages", read:databases, write:databases, read:files "View and download files"), "Allow" / "Deny". Approve → issue code oc_… (single-use, 10-min TTL, bound to {clientId, userId, workspaceId, scopes, codeChallenge, redirectUri} in Redis; stored only by reference — never the verifier) → 302 redirect_uri?code=…&state=…. Deny → 302 ?error=access_denied&state=….
POST /oauth/token (JSON, RFC 6749 + PKCE): grants authorization_code (validates client — Basic or body; confidential client requires matching secret; code single-use: second use → 400 INVALID_GRANT; expired → INVALID_GRANT; verifier recomputed to the challenge (S256); redirect_uri must match the authorize step) and refresh_token (rotation per §1.2: new refresh issued, old invalidated, reuse of an old one → revoke the whole token family + audit oauth.token_revoked {reason:"reuse_detected"}). Response {access_token (JWT, 900 s, claims: sub, client_id, scope, type:"oauth"), token_type:"Bearer", expires_in, refresh_token, scope}. Errors: INVALID_GRANT / INVALID_CLIENT / INVALID_SCOPE / UNAUTHORIZED per docs §8.2 table.
POST /oauth/revoke (RFC 7009): always 200, revokes access (JWT denylist by jti) / refresh (family invalidation) tokens.
GET /oauth/clients/:clientId: public metadata (name, logo, offered scopes) — used by consent screen and third parties.
Scope enforcement on API calls: OAuth JWTs are validated by the existing JWT middleware (new type:"oauth" claim branch); each request with X-Factory-Workspace is checked: user is a member of that workspace AND every scope in the token covers the route (route→scope map mirrors the TASK-0127 scope decorator, extended with the five OAuth scopes; a read:pages token calling a record-write route → 403 FORBIDDEN {reason:"api_key_scope_write"} alias applies — documented: OAuth scope denial reuses the same reason with details[0].scope naming the missing scope).
Audit: the three events with metadata (clientId, scopes, workspaceId; never tokens/codes).

Acceptance Criteria

E2E (full code flow, the exit scenario): register an app (UI-less: via API in the test) → GET /oauth/authorize with PKCE params as a logged-in test user → consent screen renders app name + 5 scope checkboxes → allow → 302 with code + state → POST /oauth/token with code_verifier → 200 with access + refresh tokens (assert client_id + scope claims) → call GET /api/v1/workspaces/:id/pages with the token + X-Factory-Workspace → 200.
Integration: code single-use — second token request with the same code → 400 INVALID_GRANT; code after 10 min (Redis TTL faked) → INVALID_GRANT; wrong verifier → INVALID_GRANT; redirect_uri not in allow-list at authorize → 400 invalid_request; scope not offered by the app → INVALID_SCOPE.
Integration: refresh rotation — refresh once → new refresh works, old refresh → INVALID_GRANT; reusing the old refresh after rotation → whole family revoked (a third token is unusable) + oauth.token_revoked {reason:"reuse_detected"} in the audit.
Integration: scope enforcement — read:pages-only token: GET pages 200; POST page → 403 with the scope reason + missing scope named; workspace the user is not a member of → 401/403 (never 200; existence hidden → 401 per credential-type rules — assert 401).
Integration: public client (no secret) without code_challenge → authorize 400; confidential client with wrong secret at token → INVALID_CLIENT.
UI: consent screen is keyboard-accessible, shows scope descriptions, Deny returns error=access_denied (E2E asserts the 302); anonymous user is redirected to login and resumes at consent after logging in (E2E).

Backend Changes

apps/api/src/modules/oauth/: oauth.controller.ts (authorize HTML, token, revoke, clients), oauth.service.ts (code issue/consume, token issue/refresh/rotate, family revocation), scopes.ts (route→scope map + scope descriptions), oauth.module.ts; Redis structures (code store 10 min, token families); Integration.config.oauthApps accessors + admin endpoints in the same module (or integration module — decision: admin CRUD lives in the existing integration module, flow endpoints in oauth).
JWT middleware: type:"oauth" branch + workspace/scope check integration (shared with TASK-0127's scope guard).
Login flow: next param support for consent resume (Phase 1 login already supports next if implemented; if not, minimal addition).

Frontend Changes

Consent screen page (apps/web/app/oauth/authorize/page.tsx): app identity, scope checkbox list (all pre-checked), Allow/Deny buttons, error state (renders error param); served content is the same page the API route can reference (decision: the API renders the same React page via Next — implemented as a Next route in apps/web at /oauth/authorize, with the API redirecting there when OAUTH_AUTHORIZE_PATH is configured; default: API-served HTML so self-hosters without the web app still work — recorded decision: both exist, API-served HTML is the default, web-app page is used when WEB_ORIGIN is set).
No settings UI for apps in this task (apps are managed via API per the brief; UI in Phase 14 admin console if requested).

Database Changes

None (apps in integrations.config Json — the oauthApps shape is zod-validated on write; no new table; codes/tokens in Redis per §1.2/§8 conventions).

API Changes

GET /oauth/authorize, POST /oauth/token, POST /oauth/revoke, GET /oauth/clients/:clientId (docs §8.2 — now live), plus POST/GET/PATCH/DELETE /api/v1/workspaces/:id/oauth/apps (app registration; documented in docs/api/authentication.md §8.3 addition).

Tests

E2E: full flow + deny + anonymous-login-resume (Playwright with a test HTTP client as the "third party").
Integration: the Acceptance scenarios (single-use, expiry, PKCE verify, redirect allow-list, scope enforcement matrix, refresh rotation + reuse detection, client auth).
Unit: S256 challenge verification (RFC 7636 test vectors), scope map completeness (every public route has a scope or is explicitly public — the same walk as TASK-0127's contract test), token claims shape.

Security

Codes and refresh tokens in Redis only, short TTL, bound to client + user + workspace + challenge (no token in DB at rest beyond the Phase 1 refresh-token tables for the family).
redirect_uri exact-string matching (no prefix/regex matching — open-redirect protection); allow-list per app.
PKCE S256 mandatory for public clients; verifier never stored (challenge only).
Refresh reuse → family revocation (token-theft containment), audited.
Consent screen: scope descriptions are static server strings (no client-supplied text rendered) — no XSS vector; state is echoed verbatim but only into the redirect URL (validated redirect_uri is the only sink).

Documentation

docs/api/authentication.md: §8 (confirm live matches spec — update any deltas + add §8.3 app registration); docs/security/security-architecture.md OAuth threat model; docs/ui/ consent screen.

Dependencies

TASK-0127 (scope guard + JWT middleware branches + error envelope), Phase 1 (login, refresh-token rotation, sessions), Integration model, Redis.

Definition of Done

[x] Full authorization-code + PKCE flow live (authorize consent, token, refresh rotation with reuse detection, revoke, client metadata). — apps/api/src/modules/oauth/ (service/controller/module/scopes/token-keys); integration 20/20 + E2E oauth.spec.ts 4/4 (2 scenarios × 2 projects).
[x] App registration endpoints + oauthApps config contract; PKCE-mandatory-for-public-clients enforced. — workspaces/:id/oauth/apps CRUD (owner/admin-gated; secret once, never listed; delete drains client codes + refresh families); public client without PKCE → 400 (integration authorize block).
[x] Scope enforcement on API calls for OAuth tokens (route→scope map + workspace membership check). — lib/authorize.ts assertCanForOAuth (scope cap vs route need, details[0].scope names the missing scope) + X-Factory-Workspace membership (non-member → 401 workspace_member_required); integration scope-cap matrix (read-only token: GET 200 / POST 403 api_key_scope_write).
[x] E2E full flow + integration matrix green; audit events emitted; docs updated; lint/typecheck green. — integration 20/20; unit 20/20; E2E 4/4; audits oauth.app_created/updated/deleted, oauth.code_issued, oauth.token_issued, oauth.token_revoked {reason} asserted in the suite; docs: docs/api/authentication.md §8 (8.1 catalog, 8.2 table + rotation/reuse note, 8.3 app registration) + this task's C-13.7 record; API tsc --noEmit 0 (ESLint 9.39.5 remains repo-broken without flat config — pre-existing, C-13.6-era).

TASK-0130

Title

Webhooks production: signing, retries, DLQ, event catalog, incoming webhooks + developer hub

Objective

Make webhooks production-grade: HMAC-SHA256-signed deliveries with an AES-GCM-encrypted signing secret, exponential-backoff retry (8 attempts) with DLQ + delivery log + resubmit, a proven-complete event catalog, signed incoming webhooks with replay protection, and the in-app developer hub (overview, keys, webhooks + delivery inspector, event explorer, curl examples).

Background

Webhook (url, secretHash, events[], active) and WebhookDelivery (status, attempts, statusCode, error, deliveredAt, nextRetryAt) exist; the worker webhooks queue + processor are scaffolded; packages/shared has the domain event names and the event bus. Docs: docs/api/webhooks.md (subscriptions CRUD contract), docs/api/authentication.md (error envelope, rate limits). MASTER_ENCRYPTION_KEY exists in .env.example (application-level secret encryption). Catalog events to be proven: page.created|updated|deleted|shared, database.created|updated, record.created|updated, comment.created, form.submitted, automation.executed, file.uploaded, api_key.created. (The brief also lists these as the audit-catalog; the webhooks catalog reuses the same event names.)

Requirements

Secrets: creating a webhook generates a 32-byte random signing secret whsec_…; the raw secret is AES-256-GCM encrypted (packages/shared/src/crypto.ts encryptSecret/decryptSecret with MASTER_ENCRYPTION_KEY) and stored in a new Webhook.secretEncrypted Bytes column; secretHash (SHA-256) is also stored (used for the incoming-webhook verification path and legacy rows). The raw secret is returned once in the create response (same reveal-once pattern as API keys). Migration backfills secretHash from existing rows' secrets where available; rows with only secretHash cannot re-sign — their deliveries are signed with the hash-derived key (documented legacy behavior; new webhooks always get a raw secret).
Dispatcher (worker webhooks processor, apps/worker/src/processors/webhooks.processor.ts): consumes the domain event bus → for each active subscription matching event (exact name; page.* wildcard supported in events[]), enqueue a delivery job {webhookId, event, payload, attempt:0}. Payload envelope: {id: "evt_…", type, created_at, workspace: {id, name, slug}, data: {…resource…}, data_delta? (for updated events: changed fields only)}. Delivery: HTTP POST, 10 s timeout, expect 2xx within 5 s; success → webhook_deliveries.status=DELIVERED, statusCode, deliveredAt. Failure (non-2xx, timeout, DNS) → FAILED with nextRetryAt = now + backoff(attempt); backoff: min(30s * 2^attempt, 15min) + jitter ±20 %; after 8 attempts → DEAD_LETTER (DLQ) + alert log + admin notification (throttled per webhook 24 h).
Signature: every delivery carries X-Factory-Signature: t=<unix ts>,v1=<hmac> where hmac = HMAC-SHA256(secret, "<t>.<rawBody>") (signed over timestamp + raw body); plus X-Factory-Event: <type> and User-Agent: Factory-Webhook/1.
Resubmit: POST /api/v1/workspaces/:id/webhooks/:webhookId/resubmit body {deliveryId? (one) | "all_dead" | window?} → re-enqueues (attempt resets, new delivery row referencing the original via metadata.originDeliveryId); admin-scope (workspace Owner/Admin).
Subscriptions CRUD per docs/api/webhooks.md: POST (url must be http(s), events ⊆ catalog — unknown event → 400 listing the catalog), GET list / GET :id (never returns the raw secret; returns secretPrefix whsec_abc1… + createdAt + last delivery summary), PATCH (url, events, active — changing url or events re-validates; secret rotation = separate endpoint POST :id/rotate-secret → new secret returned once), DELETE (soft: active=false + deletedAt convention — deliveries stop; row kept for the log).
Event catalog completeness audit: a catalog module (packages/shared/src/events/catalog.ts — the single source: {name, description, payloadSchema (zod), example}) + an integration test that, for every catalog event, performs the operation that should fire it (create page, update page, delete page, share page, create database, update database, create record, update record, create comment, submit form, run automation, upload file, create API key) and asserts the event was observed on the bus AND delivered to a test HTTP receiver with a valid signature (recomputed in the test). If an event's emitter is missing (e.g. page.shared), this task adds the emitter at the operation's service layer.
Incoming webhooks: POST /api/v1/webhooks/incoming/:id — a workspace-scoped "trigger URL" that fires automations with trigger type MANUAL/WEBHOOK (Phase 11 automation module). Security: the route authenticates by signature, not credentials: the workspace registers the incoming webhook with a secret (same AES-GCM storage pattern); request must carry X-Factory-Signature over the raw body with that secret; verification fails → 401 UNAUTHORIZED {reason:"signature_invalid"}; replay protection: t must be within ±5 min of server time (else 401 {reason:"timestamp_expired"}) and t + body-hash nonce is deduped in Redis for 10 min (else 401 {reason:"replay_detected"}). On success: enqueue the automation trigger (returns 202 {data:{runId}}); the automation runs with its normal permissions (creator = webhook's createdBy user context — recorded in audit as actorType: "INTEGRATION").
Developer hub (in-app, apps/web/app/(workspace)/[workspaceSlug]/developers/): tabs — Overview (base URL https://<host>/api/v1, quickstart with curl for pages + auth examples for API key and OAuth), API Keys (embeds TASK-0128's Developers tab component), Webhooks (list + create/rotate/delete, delivery inspector: per-webhook delivery table with status/attempts/statusCode/error/nextRetryAt + per-delivery drawer showing payload JSON + signature header + a "Test send" button that delivers a webhook.test event with a fixed payload), Event Explorer (every catalog event with description, payload schema rendered, example JSON copy button), and per-resource "View curl examples" (from docs/api/*.md curated snippets). The hub links to /docs/api (the public docs — TASK-0127's swagger at /api/v1/docs + the markdown docs site route /docs/api if a docs page exists; if not, link /api/v1/docs directly).

Acceptance Criteria

E2E (the exit scenario): create a webhook (UI or API) → create a key → curl POST /api/v1/pages → the test HTTP receiver gets page.created within 10 s with X-Factory-Signature; the test recomputes HMAC-SHA256(secret, "<t>.<body>") and asserts it equals v1 (signature verified in-test); then PATCH the webhook URL to http://127.0.0.1:1/dead → create another page → the delivery log (via API) shows the delivery going PENDING → FAILED (attempts incrementing with nextRetryAt gaps matching backoff) — retries visible in the log (≥3 attempts observed before the test ends).
Integration: 8-failure run → DEAD_LETTER status + DLQ entry + admin notification; resubmit all_dead → new delivery rows with originDeliveryId link and attempts reset.
Integration: signature correctness — for 3 random payloads, the test recomputes the HMAC from the stored (decrypted) secret and matches the header format t=…; v1=… (exact header shape).
Integration: catalog completeness — every catalog event fires on its operation and is delivered (the test table is generated from catalog.ts; a new event added to the catalog with no emitter fails the build).
Integration: incoming webhook — valid signature → 202 + automation run created (assert run row); bad signature → 401 signature_invalid; timestamp 6 min old → 401 timestamp_expired; same t+body twice → second is 401 replay_detected.
UI: delivery inspector shows a failed delivery with payload + signature; "Test send" delivers webhook.test and it appears in the log; event explorer renders all catalog events with copyable examples.
Security: raw signing secret appears only in the create response + rotate response (schema test on list/get: no secret* raw field); webhook URL validation rejects non-http(s), file://, and private-IP SSRF targets for the delivery URL (decision: block RFC1918/link-local/loopback targets in non-dev environments — documented in docs/security/security-architecture.md; dev env override flag for local testing).

Backend Changes

Migration: Webhook.secretEncrypted Bytes? (+ backfill logic in the migration's JS for dev seeds; production rows created pre-migration keep hash-only legacy behavior).
packages/shared/src/crypto.ts: AES-256-GCM encryptSecret/decryptSecret (if not present) + unit tests (known-vector roundtrip, tamper detection).
packages/shared/src/events/catalog.ts: the catalog (names, zod payload schemas, examples) — single source for: dispatcher matching, API validation (400 unknown event), the completeness test, and the event explorer.
Worker: full webhooks.processor.ts (enqueue, sign, deliver, backoff, DLQ, resubmit jobs webhooks.deliver, webhooks.resubmit), DLQ = BullMQ repeatable/failed-queue + webhook_deliveries status; admin notification via the notification processor.
API: webhooks.controller.ts CRUD + rotate-secret + resubmit + deliveries list/drawers; incoming-webhooks.controller.ts (signature + replay middleware); SSRF guard for delivery URLs (apps/api/src/common/ssrf.ts used by the worker before sending).
Event emitters: audit every catalog event's operation for an existing emitter; add missing ones (expected candidates: page.shared, file.uploaded, api_key.created — verified, not assumed).

Frontend Changes

Developer hub layout + 5 tabs as in Requirements; WebhookDeliveryInspector.tsx (table + payload/signature drawer + test send), EventExplorer.tsx (schema renderer from zod → JSON schema → display; copy button), WebhookCreateDialog.tsx (reveal-once secret), curl-example panel (static curated snippets, key placeholder); mobile: tabs become a stacked accordion; a11y (tables with captions, drawer dialog semantics, live region for test-send results).

Database Changes

Prisma migration: Webhook.secretEncrypted Bytes?; index webhook_deliveries [webhookId, createdAt desc] (inspector pagination) if not present.

API Changes

Webhooks CRUD + rotate-secret + resubmit + GET :id/deliveries + POST /api/v1/webhooks/incoming/:id — all in docs/api/webhooks.md (updated to the final contract: signature format, envelope, backoff table, DLQ, resubmit, incoming-webhook auth). OpenAPI spec (TASK-0127) updated with these paths (contract test keeps them in sync).

Tests

Integration: the 7 scenarios in Acceptance Criteria + backoff gap assertions (mock clock), secret-rotation invalidation of old signatures, legacy hash-only webhook still signs, SSRF guard unit tests (loopback/private/valid public).
E2E: the full exit scenario (create → signed delivery ≤10 s → in-test HMAC verification → dead URL → visible retries).
Unit: signature format parser, catalog zod schemas (every example validates its own schema — a catalog self-test), backoff computation, replay nonce logic.

Security

Secrets: AES-GCM at rest with MASTER_ENCRYPTION_KEY (never logged; decrypt only at delivery/verification time in memory); reveal-once UI pattern; audit webhook.created/webhook.secret_rotated/webhook.revoked (raw secret never in audit).
SSRF: delivery URLs validated (scheme + hostname not private/loopback/link-local unless WEBHOOK_ALLOW_PRIVATE=true in dev); DNS resolved at send time and re-checked (DNS-rebinding mitigation: resolve once, connect to the pinned IP, verify SNI).
Replay protection on incoming webhooks (±5 min + nonce) prevents captured-delivery replay.
Payload minimization: data_delta for updates; no secrets, no other workspaces' data, no internal IDs beyond public CUIDs in payloads.

Documentation

docs/api/webhooks.md final (the canonical webhook contract — signature, envelope, events catalog reference, retries, DLQ, resubmit, incoming webhooks, security); docs/security/security-architecture.md (webhook threat model: SSRF, replay, secret storage); docs/ui/ developer hub spec; OpenAPI updated.

Dependencies

TASK-0127 (error envelope, rate limits, OpenAPI + contract test — the webhook routes join the contract), TASK-0128 (API keys UI embedded in the hub), Phase 11 (automation triggers for incoming webhooks, worker queue infra, notification processor), Phase 8/2/3/5/9/10 services (the operations that emit catalog events), packages/shared crypto + events.

Definition of Done

[x] Signed deliveries with encrypted secret storage, backoff retries (8) → DLQ + notifications, resubmit endpoint — proven by tests. — integration webhooks.integration.test.ts 15/15 (signed HMAC delivery + SSRF guard + backoff + DLQ + resubmit + delivery log); worker unit 347/347 (retry loop, markDeadLetter + 24 h-throttled admin notification); shared 272/272 (webhookBackoffMs canonical 30 s/15 min ±20 %, classifyDeliveryResult, signature vectors).
[x] Event catalog is the single source; completeness test enforces every event fires + delivers (new events fail the build until emitted). — WEBHOOK_EVENT_CATALOG (packages/shared/src/events/catalog.ts) drives dispatcher matching, API subscription validation (unknown → 400 listing catalog), the EventExplorer UI (via the browser-safe ./events/catalog subpath, C-13.8f) and the integration completeness test (all 13 events fire + deliver on their operation; a catalog entry with no emitter fails the suite).
[x] Incoming webhooks live with signature verification + timestamp/nonce replay protection, firing MANUAL/WEBHOOK automations. — POST /webhooks/incoming/:webhookId: HMAC verify → 401 WEBHOOK_SIGNATURE_INVALID, ±300 s window → 401 WEBHOOK_TIMESTAMP_STALE, Redis (t, body-hash) nonce (TTL 600 s, SET NX) → 401 WEBHOOK_REPLAY_DETECTED; 202 {data:{runId}} enqueues the linked automation (audited, INTEGRATION actor); covered by the integration suite + unit vectors.
[x] Developer hub complete (5 tabs) with delivery inspector + test send + event explorer + curl examples. — Overview · API keys · Webhooks · Events · Curl (developers/page.tsx); WebhooksTab (create/rotate with secret-once reveal, pause/delete), DeliveryInspector (10 s-poll log, payload + signature-hint drawer, Test send, per-row Retry 409-guarded, Resubmit dead), EventExplorer (catalog-driven), CurlExamples + OverviewTab (runtime base URL, placeholders only).
[x] E2E exit scenario green (signed delivery ≤10 s, in-test HMAC verification, visible retries on dead URL); docs/api/webhooks.md final; lint/typecheck green. — tests/e2e/specs/webhooks.spec.ts 2/2 (desktop + mobile): UI key+webhook creation (secrets shown once), bare-ctx POST /pages → signed delivery ≤10 s with in-test HMAC re-computation (timingSafeEqual), dead URL → FAILED row with attempts≥3 + visible backoff gaps (2 s → 4 s → 8 s) + nextRetryAt; delivery drawer shows payload + X-Factory-Signature hint. Docs final: docs/api/webhooks.md (rewritten to the implemented contract — wire payloads, bare response shapes, 30 s·2^(n−1) backoff, incoming §8, hub §9), docs/api/openapi.yaml (5 webhook ops + 9 OAuth paths; contract test 6/6 both directions), docs/security/security-architecture.md §10.4, docs/ui/developers.md. Typecheck: shared 0 / api 0 / worker 0 / web 124 (= baseline, 0 new). ESLint remains repo-broken on 9.39.5 without flat config (pre-existing, C-13.6-era — no flat config in repo).

Recorded conflicts (AGENTS rule 29)

C-13.1 — TASK-0127 per-key rate-limit default. The phase brief text said "100 req/min"; docs/api/authentication.md §11 (the published contract, precedent over the brief) and phase decision #1 say 600 req/min. Resolution: 600, as a dedicated RATE_LIMIT_API_KEY env (default 600), per-key bucket rl:api:{apiKeyId}; keys with their own rateLimit always win. Recorded, not guessed (docs/api/authentication.md §11 governs).

C-13.2 — TASK-0128 create-form bounds. The task text says name ≤64 and rate limit 1–10 000; the landed server DTO (the TASK-0127 prerequisite, api-keys.controller.ts) enforces name ≤100 and rate limit 1–100 000; the published docs/api/authentication.md §7 is silent on both. Resolution: the UI follows the SERVER (authoritative): maxLength=100, rate-limit input 1–10 000 (the task's practical window — a strict subset the server also accepts; the "600 (default)" placeholder stays). The UI never rejects what the server accepts.

C-13.3 — TASK-0128 "asserted via the audit API". No audit READ endpoint existed when this task was written. Resolution: added the minimal GET /api/v1/workspaces/:id/audit (OWNER/ADMIN, newest-first, bounded, optional action filter) — Phase 14's audit viewer builds on it.

C-13.4 — TASK-0128 mobile E2E project: isMobile vs the layout viewport. The DoD's "Mobile behavior" row required the Developers tab to be usable at < 640 px, and the pre-existing chromium-mobile Playwright project spread devices["iPhone 13"] (isMobile: true, iPhone UA, hasTouch) pinned to chromium. With isMobile, Chromium splits the layout viewport from a 375px visual viewport and processes the page's width=device-width meta differently; Playwright 1.49's CDP actionability hit-test then desynced on those pages — a dialog/table button's center intermittently resolved onto the backdrop or another node while the page's own elementFromPoint at the same coordinate returned the button (the reported interceptor varied run-to-run: backdrop / footer / fieldset / input), hanging every locator.click() and even locator.focus() (its pre-check runs the same hit-test). The app itself is correct: the viewport meta is served at byte 92 of the HTML, and with the create dialog open no element exceeds 375px and scrollWidth stays 375 (probed) — the app's viewport export (apps/web/src/app/layout.tsx, width=device-width, initial-scale=1) made the layout actually 375px on mobile. Resolution: the chromium-mobile project now uses a plain chromium context with viewport: 375×812, deviceScaleFactor: 3 and NO isMobile — identical CSS layout (breakpoints key off CSS px, identical at 375 either way), stable hit-test, verified 4/4 stable over repeated full runs. Trade-off (recorded, not guessed): the iPhone user agent and hasTouch are no longer emulated; the app does no UA sniffing and no touch-only interactions, so nothing under test changes. The test-side safeClick() helper in developers.spec.ts (plain click → coordinate-free el.click() fallback, with the caller still asserting the full post-click state) remains as defense-in-depth against any future emulation regression.

C-13.5 — TASK-0128 E2E: the ready-wait reload loop livelocked the auth bootstrap (CORRECTED diagnosis — initially recorded as a worker-count issue). The Developers spec flaked at its first-paint ready check and later at getByRole("row", { name: /CI/ }) — on BOTH desktop and mobile. The first hypothesis (recorded here for the record) blamed worker count vs the unbuilt dev stack: the instrumented poll showed the create→invalidate→refetch cycle completes in ~6.5s when idle, and worker experiments (CLI --workers=2 3/3 green, config workers: 2 2/3 failed, --workers=1 3/3 green) were inconsistent, so worker load was never confirmed as the cause. The decisive experiments: (a) a state-dump run captured the stuck page at the (app) layout's "Loading workspace…" auth gate (mainLen=0, no localStorage) with the network log showing POST /auth/refresh 200 + GET /auth/me 200 already completed; (b) a manual probe FROM the page's own cookie jar (plain fetch, bypassing the app) returned refresh:200, hasToken:true, /auth/me 200 user:demo@factory.local — the server session was provably valid, so the app's React state simply never committed the user; (c) a single uninterrupted load (no reloads) with a 30s budget cleared in 2.1–2.9s, 3/3. Root cause: the openDevelopers() helper's "ready or reload" loop reloaded every ~2–3s, but the fresh-context auth bootstrap (cookie → POST /auth/refresh → GET /auth/me → authStore.setUser) needs ~3s after the page's JS hydrates — each reload aborted the in-flight refresh→me chain before React processed the 200 response, so no load ever reached the setUser commit (the 200s were seen at the network layer; the component was already gone). The app is correct (single loads clear 3/3); the helper was the defect. Resolution: openDevelopers() now does one goto + toBeVisible({ timeout: 20_000 }) with NO reloads (20s absorbs a cold dev-server compile; a genuine stuck gate still fails). workers: 2 in all environments and the 20s budget on the CI-row assertion are RETAINED: 2 workers is a reasonable cap for the shared dev stack (CI parity) and the row budget covers the cold-refetch case that the poll proved can exceed 10s under concurrent compile — but neither was the root cause of the flake. Verified post-fix: see TASK-0128 E2E result line.

C-13.6 — TASK-0128 E2E flake #2: the parallel double-rotation race killed the session family (PRODUCT FIX, not a test-side band-aid). Symptom: under the default parallel config (workers: 2, 8 contexts each fresh-logging-in the demo user) the Developers spec flaked at a rate of ~1/3 owner-test runs; the stuck page parked at /w/default behind the (app) auth gate ("Loading workspace…", mainLen=0) with a redirect loop between /login and /w/default. Solo (single-context) runs were 5/5 clean, so the flake required concurrency — the probe that cracked it was a single-context double load (FAM probe): both loads cleared, but the demo user's DB then held 32 live refresh tokens, including rows with revokedAt=f, replacedBy=f (orphaned live tokens) sitting beside the chained revokedAt=t → replacedBy=t rows. The full root chain, confirmed at every level:

Client (2 refreshes per page load): reactStrictMode: true (apps/web/next.config line 3) double-mounts the (app) layout, so the useAuth bootstrap effect ran twice per load, each instance with a fresh hydrated ref → 2 concurrent POST /auth/refresh (COUNT probe 5/5 solo: load-refreshes=2 me=1). The bootstrap called authApi.refresh() → mutatingApiFetch("/auth/refresh") — which bypassed the module-level single-flight refreshAccessToken()/refreshInFlight (api.ts) that only the 401-retry path used. The comment at that call site ("reuses the same primitive") was false — the divergence was invisible.
Server (non-atomic rotation → orphaned live tokens): auth.service.ts refresh() did refreshToken.create(new) then refreshToken.update(old → {revokedAt, replacedBy}) as two separate awaits. Two concurrent refreshes of the same row both passed the row.revokedAt check, both created new tokens, and the unconditional update was last-writer-wins on replacedBy — leaving the loser's new row live and unreachable (the orphaned rows above). Every E2E context leaked a token per load; 8 contexts × fresh logins × loads = 32 live rows.
Server (stale-vs-reuse conflation → family kill): a presentation of any revoked row — including the benign "my concurrent twin already rotated" case — was treated as a reuse attack: the whole user token set + all ACTIVE sessions revoked, audit auth.refresh_reuse_detected, 401 refresh_token_reused. Worse, the kill path did not clear the httpOnly cookie, and the middleware (apps/web/src/middleware.ts line 28) gates on cookie existence only — so a dead-but-present cookie bounced /login → /w/default, the (app) gate bootstrapped with the revoked cookie → 401 → re-login attempt → …: the exact redirect loop + parked gate observed in the flake.

Rule-29 conflicts recorded (code vs architecture doc — the code was fixed to match the doc):

docs/architecture/security-architecture.md §1.2 refresh-rotation step 3 MANDATES the distinction the code lacked: row.revokedAt set with a live successor ("expected rotation chain, replacedBy") → stale retry of an already-rotated token — client re-logins → plain 401, no family kill; only a dead row with a dead successor → family kill + audit. The code killed on ANY revoked row. Fixed (see below).
Doc drift #2 (NOT fixed here, recorded): the doc names the cookie factory_rt; the code uses REFRESH_COOKIE = "factory_refresh" (apps/api/src/constants.ts). Functionally irrelevant; the doc predates the rename.
Doc drift #3 (NOT fixed here, recorded): backend-architecture.md:371 maps refresh-reuse detection to CONFLICT/409; the code returns 401 UNAUTHORIZED (correct per §1.2's "401" and RFC 6749 §5.2 error semantics).

Security trade-off (recorded, not guessed): the stale-retry grace means a stolen token can be replayed a bounded number of times (each replay 401s without collateral) as long as the chain advances legitimately — i.e. the victim's own client keeps rotating. Detection still fires on the first presentation where the chain has moved on WITHOUT the attacker (dead successor), at which point the whole family dies. This is precisely the doc's threat model (§T2 "rotation with reuse detection + family kill") and strictly reduces false-positive session loss (a family kill of a multi-device user who merely had two tabs race is worse than the attack it prevented, since the victim re-logs in).

Fix (4 parts, all in this task's diff):

A — server (apps/api/src/modules/auth/auth.service.ts refresh()): (a) stale-retry branch — revoked row + live successor (replacedBy set, successor row unrevoked) → audit auth.refresh_stale_retry, plain 401 reason refresh_token_stale, NO kill, NO cookie change (the client's live token is untouched); (b) rotation made atomic in one interactive $transaction with a conditional updateMany(… AND revokedAt IS NULL) — a concurrent winner's revocation makes the update match 0 rows, the whole tx (including the new token row) rolls back, and the loser returns the same stale 401 (audit raceLost: true); (c) the reuse-kill path now clearRefreshCookie(res) (breaks the middleware existence-gate loop) and its audit carries successorPresent (clone vs post-revocation replay). replacedBy-set is the clean discriminator because every other revocation path (logout, per-device revocation, admin kill) revokes WITHOUT it.
B — client (apps/web/src/lib/api.ts): authApi.refresh() now funnels through the SAME single-flight refreshAccessToken() primitive as the 401-retry path (the false comment is replaced by an accurate one); the primitive now rejects with a typed ApiError (parsing the server reason) instead of swallowing to null, so callers can branch on refresh_token_stale (isStaleRefreshError()).
C — client (apps/web/src/hooks/use-auth.ts + auth-store.ts): the bootstrap retries exactly once on refresh_token_stale — per-tab single-flight cannot dedupe across tabs (two tabs' cookie jars diverge at rotation time), and the browser jar already holds the rotated successor when the other tab's rotation response was observed. authStore.clear() (logout) now also resets initializing: false so a logout fired before the bootstrap settles can't leave the (app) gate spinning.
D — server, livelock hardening (found by the first post-fix E2E run): every TERMINAL cookie-dead path of refresh() — unknown token (refresh_token_invalid), expired token (refresh_token_expired), and the stale branches (refresh_token_stale, both the successor branch and the race-lost branch) — now also clearRefreshCookie(res). Reason: the middleware gates on cookie existence only, so a dead-but-present cookie keeps bouncing /login → /w/default behind the (app) gate, which can never authenticate (a stuck state with no escape — the user's only route out is the login page the middleware refuses). Clearing the cookie sends the user to a REAL login page instead. (The reuse-kill path already cleared the cookie in fix A(c).) The stale-retry integration test now asserts the Set-Cookie clear on the stale 401.
D2 — test-side (not a product band-aid): the first post-fix E2E run STILL failed (1/4), at the "re-render" step's two back-to-back hard gotos (members → developers, no settle between). Each hard goto is a fresh JS context whose one-time bootstrap posts POST /auth/refresh, so the second goto's bootstrap races the first's in-flight rotation across destroyed/recreated documents — a genuine cross-context race the app must tolerate. But it is NOT real user behavior: users navigate settings tabs via the SPA router (ONE bootstrap per tab, shared in-memory token — no re-rotation), never a hard reload per tab. The test now settles the first load (waits for the Members h1, which only renders after the bootstrap cleared the gate) before the second, capping the stress at exactly ONE in-flight bootstrap — the worst case the production single-flight + stale-grace fix must cover, and the one real two-tab users hit.

Tests: tests/integration/auth.integration.test.ts — the old "refresh reuse" test (one rotation + replay ⇒ family killed) is rewritten into the doc's two cases: stale retry (successor live ⇒ 401 refresh_token_stale, successor still 200, audit stale-not-reuse) and reuse attack (two rotations then replay the first ⇒ 401 refresh_token_reused, latest token dead, all tokens + session REVOKED, reuse audit fires); plus a concurrency test firing two in-flight rotations of one token ⇒ exactly [200, 401-stale], exactly ONE live token in the contested session (no orphaned live row), chain intact (first.replacedBy = winner). Web unit tests (apps/web/src/lib/__tests__/auth-refresh.test.ts): single-flight dedup — 2 concurrent callers → 1 fetch, shared outcome, in-flight slot resets after settle, stale/reused errors propagate typed (isStaleRefreshError table); (auth-store.test.ts): clear() ends initialization, setUser lifecycle. Reported, not fixed (environment-bound): the "rate limit: sustained login failures → 429" integration test fails in this environment: the shared test app raises RATE_LIMIT_AUTH (helper ?? 100000, repo .env 10000), so 20 logins in the 60s sliding window can never trip a 429 — the test as written only passes with an unset limit (zod default 10). The helper's own comment says throttling is unit-tested separately, but no such unit test exists (apps/api/src/common/__tests__/ has none) — the behavior is currently unverified anywhere. NOT a regression from this fix (the diff touches no login/rate-limit path); needs a dedicated fix in its own task.

E2E stability re-verification: developers.spec.ts at the default workers: 2 config, repeated until stable: run 1 (fixes A/B/C loaded, API tsx watch reloaded) = 3/4 — the 4th test failed at the double hard-goto re-render step (D2 above). Run 1 proved the PRIMARY flake (fresh-context bootstrap double-rotation → family kill → stuck gate) is GONE (that test now passes; the failure moved to the unrelated stress step). After fix D + D2, runs 2–3+ must be 4/4 (see the TASK-0128 E2E result line). The diagnostic probe specs zz-state.spec.ts / zz-owner2.spec.ts were deleted before commit. C-13.7 — TASK-0129 rule-29 conflicts recorded (task spec vs doc / design decisions — the TASK text wins where it itself mandates behavior; doc-only details follow the doc):

C-13.7a — scope format: the task text names scopes as read:pages / write:pages (action:resource) in one example line, while the locked design
- the published doc §8.1 example use pages:read / databases:write (resource:action). Resolution: resource:action (matches the doc, scopes.ts, and every enforcement site); the task's example format is treated as a typo for the same scopes.
C-13.7b — scope count: the task text and one example list FIVE scopes (omitting files:write); the design decision + doc §8.1 example grant databases:write and the symmetric family is pages|databases|files × read|write = six scopes. Resolution: the full six-scope catalog (OAUTH_SCOPES in apps/api/src/modules/oauth/scopes.ts); apps may offer a subset (the E2E app offers all six; nothing requires exactly five).
C-13.7c — RFC 6749/7009 error codes in the canonical envelope: the task's error table lists bare RFC strings (invalid_grant, invalid_client, invalid_scope); the repo's canonical error model (docs/api/ authentication.md §9, established Phase 1 + TASK-0127) is a single envelope {error:{code,message,details?,requestId}} with UPPER_SNAKE machine codes. Resolution: the RFC error NAMES mapped onto the canonical envelope — INVALID_GRANT / INVALID_CLIENT / INVALID_SCOPE added to ErrorCode (apps/api/src/common/errors.ts), HTTP 400 per RFC 6749 §5.2. The token endpoint therefore returns e.g. {"error":{"code":"INVALID_GRANT",…}} (not the RFC's top-level "error":"invalid_grant" key) — a deliberate envelope-consistency choice, recorded here rather than guessed.
C-13.7d — no FULL_ACCESS scope; app management needs a user session: the design has no OAuth scope reaching FULL_ACCESS (oauthScopeCap tops out at EDIT), so assertRole(["OWNER","ADMIN"]) routes (incl. app registration) reject OAuth tokens with 403 {reason:"oauth_role_denied"} — an app can never manage its own registration; only a user session (JWT/cookie) or an ADMIN-scoped API key can. Consistent with the doc §8.2; recorded because the task text does not explicitly say apps cannot self-manage.
C-13.7e — dead-token replay = family kill (task wins over §1.2 nuance): the task's acceptance line says verbatim "reuse of an old one → revoke the whole token family + audit oauth.token_revoked {reason:"reuse_detected"}". The architecture doc §1.2 (login-session rotation, see C-13.6) carves out a benign stale retry (dead token whose successor is still live → plain 401, no kill). Resolution: the TASK text wins — ANY presentation of a rotated/dead OAuth refresh token revokes the whole family + audits reuse_detected, even with a live successor (the stale-retry exemption and its oauth:family-rt:<fid> bookkeeping were REMOVED as dead code during implementation). Rationale: the OAuth contract is a third-party token — the stricter, simpler containment is the task-mandated behavior, and the §1.2 nuance was written for the first-party login session, not third-party refresh tokens. The §8.2 docs note records the deviation.

TASK-0129 implementation notes (verified):

Storage: all OAuth credentials live in Redis (per the task's Security section — "codes and refresh tokens in Redis only"): oauth:code:<code> (SETEX 600 s, single-use via GETDEL), oauth:rt:<sha256> (the rotating refresh record), oauth:family:<fid> (=revoked, the family kill switch), oauth:family-meta:<sha256> ({familyId, clientId, userId} — how a dead token is classified), oauth:jti:<jti> (access-token denylist, TTL = remaining token life capped at the access TTL), and per-client sets oauth:client-codes:<clientId> / oauth:client-rtset:<clientId> (drained on app deletion). No DB columns beyond the existing Integration.config.oauthApp row; no migration.
Consent CSRF: the consent page carries a short-lived signed consent_token (JWT, kind:"oauth_consent", bound to the re-resolved user + the exact authorize params); the approve form POST must echo it, and the server re-validates EVERY authorize parameter + re-resolves the user from the cookie/Bearer — the form's identity fields are never trusted.
Login resume: an anonymous browser hitting /oauth/authorize gets 302 <WEB_ORIGIN>/login?next=<absolute /oauth/authorize…> (both hops absolute — a relative /login from an API 302 would 404 on the API origin); the web login page's resolveNext() accepts relative paths or API-origin URLs only (open-redirect-safe), else /w/default.
Revocation: POST /oauth/revoke is always 200 (RFC 7009); a refresh token is hard-deleted AND its family killed (reason:"revoked", audited when a request context is present); an access token is jti-denylisted.
Tests: unit apps/api/src/modules/oauth/__tests__/oauth.scopes.test.ts (20 — incl. the RFC 7636 Appendix B S256 vector); integration tests/integration/oauth.integration.test.ts (20 — app CRUD, authorize incl. anonymous login-resume + PKCE-mandatory-public + invalid_scope, consent HTML, token endpoint claims + single-use + wrong-verifier + bad-secret, scope cap matrix, non-member 401, refresh rotation + reuse family-kill (login session untouched) + audit, revoke both hints, client metadata 200/404, audit events); E2E tests/e2e/specs/oauth.spec.ts (2 scenarios × 2 projects — the exit scenario via a real browser's cookie path: consent screen → Allow → 302 code+state (Location captured from the form-POST response; the third-party callback is a Playwright route sink) → token exchange → GET /pages + POST /pages with the OAuth token + X-Factory-Workspace; plus the anonymous → absolute web-/login?next= path).

C-13.8 — TASK-0130 rule-29 conflicts recorded (published doc vs task text vs implementation):

C-13.8a — signing-secret storage: "hashed" vs encrypted-at-rest. The task text and the pre-existing published docs/api/webhooks.md say the signing secret is "stored hashed" (one-way). Retries, test sends, and resubmits must re-sign with the original secret, so a hash alone cannot serve them — the worker needs the raw value. Resolution: dual store — new rows keep secretEncrypted (AES-256-GCM, @factory/shared crypto envelope, keyed by MASTER_ENCRYPTION_KEY) as the source of truth, plus secretHash (SHA-256) for integrity checks and for legacy hash-only rows (the resolveSigningSecret fallback tries decrypt-first, then hash-verify, so pre-existing test fixtures and old rows keep working). docs/api/webhooks.md (webhook object note) + docs/security/security-architecture.md §10.4 record the corrected model. The raw secret still appears exactly once (create/rotate) and is never read back; secretPrefix is leak-free.
C-13.8b — the published §4.3 retry table was never the schedule. The published table (1 min / 5 min / 30 min / 2 h / 6 h / 12 h / 24 h) and the "one row per attempt" reading of the delivery log do not match the task-mandated min(30s·2^(n−1), 15min) ±20 % backoff with 8 attempts. Resolution: the task wins — the worker owns the retry schedule (worker-only; the API only enqueues), ONE WebhookDelivery row per logical delivery (attempts increments in place, nextRetryAt carries the schedule — no intermediate per-attempt rows), attempt ≥8 → DEAD_LETTER (the DLQ) + in-app notification to admins (24 h throttle per delivery). docs/api/webhooks.md §4.3/§5/§6 rewritten to the implemented contract; base/cap are env-configurable (WEBHOOK_BACKOFF_BASE_MS / WEBHOOK_BACKOFF_MAX_MS, defaults 30 s / 15 min) for tests.
C-13.8c — response envelope: the published { data: {…} } wrappers were wrong. The pre-existing doc showed create/rotate/test/retry/resubmit wrapped in data, with a responseBody on test and recentDeliveries on the webhook object. The API convention (AGENTS rule 29: the architecture's no-global-wrapper rule, matching every other module) is BARE objects: create/rotate → {…webhook, secret, warning}; test → {deliveryId, status, statusCode, attempts, latencyMs} (no responseBody); retry → {deliveryId, status, attempts}; resubmit → {resubmitted, deliveryIds, originDeliveryIds}; list/deliveries keep {data: […], nextCursor}. The webhook object has no recentDeliveries (the delivery log is a separate endpoint). Resolution: implementation wins (it matches the architecture convention and the committed E2E assertions); the doc is corrected.
C-13.8d — REJECTED/DEAD_LETTER as first-class log statuses. The published log docs knew only PENDING|DELIVERED|FAILED; the implemented classifier adds REJECTED (4xx, terminal, no auto-retry — the doc's own §4.1 split required it to be observable) and DEAD_LETTER (the 8-attempt terminal state). The deliveries filter accepts all five. Doc + OpenAPI updated accordingly.
C-13.8e — dev-only .env override for the committed E2E exit scenario. The committed tests/e2e/specs/webhooks.spec.ts exit scenario runs against the live dev stack: it delivers to a 127.0.0.1 receiver (needs WEBHOOK_ALLOW_PRIVATE=true) and asserts 2 s / 4 s retry gaps (needs WEBHOOK_BACKOFF_BASE_MS=2000 / MAX_MS=16000). These vars live in the gitignored .env (dev only) — production defaults are unchanged (private-IP guard ON, 30 s / 15 min backoff), and the production behavior is proven by the integration suite (SSRF-guard tests + backoff unit tests with the canonical 30 s base). Recorded so a future .env cleanup does not silently break the E2E.
C-13.8f — the shared barrel is not browser-safe. @factory/shared's index.ts re-exports events.ts/crypto.ts/webhook-delivery.ts, all of which import node:crypto — any web import of the barrel made webpack fail the Developers page build with UnhandledSchemeError (page 500). The catalog module (src/events/catalog.ts) is zod-only and browser-safe. Resolution: a dedicated ./events/catalog subpath export (the package already ships source TS) and the web imports the subpath, never the barrel. This is what the EventExplorer/WebhooksTab comments document.