Phase 8 — Search (Task Specs)

Purpose

Phase 8 delivers workspace-wide search: an event-driven indexer that mirrors visible content into OpenSearch, a permission-filtered full-text search API, and a global search UI. It also lays the hybrid (keyword + semantic) foundation — vector mapping, chunker wiring, and a stubbed embedding pipeline — so Phase 12 (AI/RAG) can plug in without re-indexing the platform.

Phase Exit Criteria (from ROADMAP.md § Phase 8)

• [ ] Indexer (domain events → OpenSearch; permission metadata per document)
• [ ] Full-text + prefix + fuzzy search
• [ ] Filters (type, workspace, team, updated date, creator), recent searches
• [ ] Search UI (global search page, Cmd/Ctrl+K integration, suggestions)
• [ ] Exit: search response < 500 ms (p95); index lag < 30 s; permission-filtered results proven by integration test.

Tasks

ID	Title	Estimate
TASK-0096	Search indexing pipeline (events → OpenSearch + mirror + reconciliation)	L
TASK-0097	Full-text search API (POST /api/v1/search)	L
TASK-0098	Global search UI (/search)	M
TASK-0099	Hybrid AI search foundation (vector mapping + chunker + embedding stub)	M
TASK-0100	Search performance verification (index lag, query p95, E2E)	M

Phase 8 Notes — recorded decisions & conflicts (AGENTS.md rule 29)

Decisions are recorded as slices land; the authoritative record is docs/architecture/search-architecture.md § 0 (Implementation Notes).

• Layout: single index, alias factory-content, tenant isolation by the mandatory workspaceId filter; camelCase stored doc; dense_vector (OS 2.x) with env-tunable dimension (default 1536). The Phase-0 per-workspace factory-search-<wid> scaffold is retired.

• Single search core: worker indexer and API module both consume @factory/search (SearchClient, query builders, mapping). The Phase-0 hand-rolled clients (apps/api/.../search/opensearch.ts, the 5-index worker scheme, files.processor.ts direct doc write) are removed in S2/S3.

• ⚠️ Conflict (arch doc vs live guard): the page-read guard (effectiveLevel) does not enforce Page.isPrivate — a latent Phase-3 authorization gap. Search implements the intended model (private pages are workspace-invisible, visible only via explicit grant/public share), so search isolates private pages even though the live page-read endpoint does not. Filed as a Phase-3 security follow-up; not fixed in Phase 8.

• ⚠️ Typecheck baseline: at the Phase-8 start commit, tsc --noEmit is already red in Phase-2–7 code this phase does not touch (worker 47, api 146 pre-existing errors). Phase-8 scope follows the documented Phase-7 convention: the search code the phase owns is typecheck-clean (packages/search fully clean — 0 errors / 72 tests; rewritten worker/api search files 0); the pre-existing non-search baseline is unchanged, not faked, not expanded.

• ⚠️ UI contract conflict (resolved in TASK-0098): this spec says a /search page; the committed tests/e2e/specs/search.spec.ts pins a topbar-dialog UI (testids search-button / search-dialog / search-result-{id} / search-filter-* / search-recent* / search-suggestion). Both surfaces ship (page + ⌘K dialog sharing one result model); the e2e testids are the pinned contract.

• Reconciliation: contentHash sweep (arch doc §1.3/§7.3), not the count heuristic of TASK-0096 §7 (superseded — subsumes it). search_documents gains a contentHash String? column (migration).

• API shape: POST /search under /api/v1 (live controller path); docs/api/search.md is rewritten to the live shape in S3 (its Phase-0 draft documented a query field + 1–200 limit that never matched the endpoint). No Postgres-ILIKE fallback — OpenSearch down ⇒ 503 SEARCH_UNAVAILABLE.

• TASK-0099 S4 landed (hybrid foundation): packages/search types.ts / mapping.ts / queries.ts (chunk exclusion) / hybrid.ts (weighted RRF) / chunking.ts + tests (75 → 91 tests, 0 typecheck errors); worker chunk emission (expandChunkDocs, page + record only, #c<n> chunk docs, parent-driven deleteChunksOf, no mirror rows); embed:chunks stub job on queue ai (ai-embedding.ts, skip when AI_EMBEDDING_PROVIDER unset, placeholder AiEmbedding rows when set); scripts/opensearch-reindex.ts + root pnpm search:reindex (500-doc _bulk backfill from the mirror, atomic /_aliases swap, --i-know-this-swaps-the-alias guard); .env.example + arch doc §9. Reconciliations vs this spec (authoritative record: arch doc §9):

  • chunker.ts → landed as chunking.ts, char-based 800/120 (spec: 800 tokens / 15% overlap) — no tokenizer dep; swappable behind ChunkOptions.
  • bodyVector → kept the landed vector field (worker + tests + running index already use vector).
  • query-builder.ts / buildHybridQuery → not created; kept queries.ts buildHybridBody/buildHybridKnnOnlyBody + rrf.ts, added hybrid.ts buildHybridFusion (weighted).
  • chunk docId <type>:<refId>:chunk:<chunkId> → landed <type>:<refId>#c<n> (chunkId = c<n>).
  • AiEmbedding (workspaceId, docType, refId, chunkId, model, vector) → landed (workspaceId, chunkRef, chunkText, vectorId, model, dimension, metadata), chunkRef = the chunk content doc id, unique [workspaceId, chunkRef] — no migration (model pre-exists).
  • chunkModel mapping field → landed vectorModel (already present).
  • chunking page + record only in Phase 8 (comment/file/database/blocks → Phase 12).
  • NEW (load-bearing): excluding chunks via the public types filter is insufficient — the API omits the types clause when the caller sends none. Mandatory bool.must_not: {term:{type:"chunk"}} in every keyword body (full-text / prefix / fuzzy / hybrid BM25 leg); kNN bodies are exempt (chunks are the kNN target).
  • worker indexDocs writes IndexDocInput 1:1 to the index (it does not call the package buildIndexDocs / its #<n> chunker — that scaffolding stays unused); chunk expansion is the worker's expandChunkDocs.
  • chunk docs have no mirror rows; deletion is parent-driven (SearchClient.deleteChunksOf, _delete_by_query?conflicts=proceed).
  • embed:chunks file processors/ai-embedding.ts (flat, spec named ai/embedding-pipeline.ts); dispatched before the AI_API_KEY gate.
  • reindex script backfills from the search_documents mirror (whole docs); source-table backfill + chunk re-embed are Phase-12 loose ends.
  • Phase-12 loose ends: real provider call in embed:chunks; hybrid query path behind a capability flag on POST /search; real AiEmbedding vectors; token-based chunking; chunking for the remaining doc types.

Phase Completion Checklist

• [x] TASK-0096 complete (2026-09-28, 581b8ad5 + the TASK-0100 S0–S5 fixes that made it run): all domain events produce OpenSearch documents with a permission snapshot stamped at index time (grants — the creator- implicit-viewer model, S5 §8); deletes on trash/purge (doc deletion + chunk expansion); the search_documents mirror maintained; the 15-min reconciliation job drift-detects and repairs. Proven live: the S5 lag e2e (create→search < 30 s, delete→unsearchable < 30 s, 1k-event burst drained to backlog 0) exercises the full event→index→query path.
• [x] TASK-0097 complete (2026-09-28, 70060e58): POST /api/v1/search supports multi_match (BM25) + prefix + fuzzy, type/workspace/team/date filters, cursor pagination, highlights; the integration test (tests/integration/search.integration.test.ts) proves cross-workspace isolation (another workspace's doc never appears; a non-member workspaceId → 404 no-leak) + validation. (S6: the route now returns 200, matching docs/api/search.md + the integration test.)
• [x] TASK-0098 complete (2026-09-28, 4f61275a): the /search page + ⌘K dialog with debounce (225 ms), grouped results, deep links, keyboard nav, recent searches, suggestions, and the loading/empty/error states.
• [x] TASK-0099 complete (2026-09-28, fa1a083f): the vector (knn_vector) field in the mapping (OS 2.15 form, S5); the chunker wired; the embed:chunks stub job in the ai queue (real RAG = Phase 12). (The spec's "dense_vector" wording is the 1.x OS name for the same field — recorded in the architecture doc §9.)
• [x] TASK-0100 complete (2026-09-29): index lag < 30 s measured in E2E (create-side mean 3.7 s / max 4.3 s over 10 consecutive runs, delete-side mean 2.1 s — DoD mean < 15 s / never > 30 s); query p95 49.9 ms / p99 92.8 ms @ 100,000 docs via k6 full profile (DoD < 500 ms / < 1 s — a ~10× margin, on a 4 vCPU / 15 GB dev machine smaller than the DoD reference env); create-page → searchable → click E2E green (the 10-run suite asserts the exact page id in the URL + title match; the 1k-event burst drains to backlog 0 in 39 ms). CI wired (the perf-search lane — reduced on merge-to-main, full nightly); perf docs merged with the env fingerprint + tuning notes (arch §10 + docs/development/performance.md).
• [ ] Full CI green (lint → typecheck → unit → integration → build → E2E) — TASK-0100 S9 (2026-09-29): the perf-search lane (reduced on merge-to-main, full nightly) is committed + YAML-lint-clean + its exact commands verified live; the whole contract is CI-wired. The actual green run is CI-gated — pending the first green CI run (mirrors the Phase-7 close-out convention; never run locally, never faked).
• [x] docs/api/search.md merged and matches the live response shape — present (11.5 KB); documents POST /search → 200 (the S6 @HttpCode(200) fix) + the {data,…} result envelope + the q-is-the-field-name contract choice, matching the live API + the integration test.

Status / Resume Point (2026-09-29 — TASK-0100 CLOSED; PHASE 8 CLOSED; NEXT: Phase 9 — Files):

TASK-0100 (the Phase-8 exit verification) is complete across S0–S9 (all committed). The Phase-8 exit criteria are met: index lag < 30 s (measured in the 10-run E2E — create-side mean 3.7 s / max 4.3 s, delete-side 2.1 s); query p95 < 500 ms @ 100k (full k6 profile p95 49.9 ms / p99 92.8 ms — a ~10× margin, run on a machine smaller than the DoD reference env); permission-filtered results proven by the integration test (tenant isolation

• non-member workspaceId → 404 no-leak) and live (the e2e test 3: a second user finds 0 private-page rows, the creator finds theirs). The "Full CI green" checklist box stays open (CI-gated — pending the first green CI run, mirroring the Phase-7 close-out convention).

Finding recorded (TASK-0096 ↔ TASK-0100 interaction): the 100k perf fixture is seeded directly into OpenSearch (no search_documents mirror rows), but the worker's reconcile sweep (TASK-0096, 15-min backstop, plus the per-page reindex path) rebuilds a workspace's docs from the mirror — so any reindex/reconcile of the perf-search-* workspaces deletes the fixtures as "orphans" (observed live: the 10-run e2e loop triggered worker reconciles that wiped the 100k docs). The CI perf-search lane is unaffected: it boots the API only (no worker → no reconcile sweep) and seeds immediately before the run. Local runs must re-seed after any worker reindex, or the seeder should write mirror rows (a follow-up, not a Phase-8 gap — the CI lane, which is the DoD's gating environment, is deterministic).

Handoff to Phase 9 (Files): the file indexer path (f.uploadedBy → creator-implicit-viewer, S5 §8) is the model the Phase-9 file search docs should follow; the file fixture type (15,726 docs) is already exercised by the 100k perf suite.

---

TASK-0096

Title

Search indexing pipeline: domain events → OpenSearch indexer, search_documents mirror, reconciliation job

Objective

Make every searchable object (page, block, database, record, comment, file, user) appear in OpenSearch within 30 s of its domain event, with a permission snapshot per document, and keep the PostgreSQL search_documents mirror and the OpenSearch index consistent via a 15-minute reconciliation job.

Background

packages/search provides the OpenSearch client (packages/search/src/client.ts), a query builder (src/query-builder.ts), RRF fusion (src/rrf.ts), and a content chunker (src/chunker.ts). The worker already has scaffolded processors under apps/worker/src/processors/ including search-index and maintenance. Domain events are published to Redis streams with names from @factory/shared/events (packages/shared/src/events/index.ts): page.created|updated|deleted|shared, database.created|updated, record.created|updated, comment.created, form.submitted, automation.executed, file.uploaded, api_key.created, user.login|logout|register, permission.changed, workspace.created. The Prisma schema already defines SearchDocument (prisma/schema.prisma, table search_documents: id, workspaceId, type, refId, title, body, permissions Json, metadata Json, indexedAt, unique [type, refId]). OpenSearch is provisioned by the Phase-0 Docker environment.

Requirements

1. Implement apps/worker/src/processors/search-index/processor.ts consuming the Redis stream consumer group from the shared event bus, handling the events: page.created|updated|deleted|shared, database.created|updated, record.created|updated, comment.created, file.uploaded, permission.changed, user.register|login (user docs are created on register, refreshed on login for lastSeen), workspace.created.
2. Document shape (one OpenSearch doc per type + refId): { id: "<type>:<refId>", workspaceId, type: "page"|"block"|"database"|"record"|"comment"|"file"|"user", title, body, permissions: { granteeType[], granteeId[], levels[], scope, isPrivate }, metadata: { teamId?, parentId?, pageId?, databaseId?, blockId?, createdBy?, updatedBy?, createdAt, updatedAt, icon?, size?, mimeType? } }.
3. Body composition rules: page body = concatenated text of all descendant blocks (read via the block repository, depth-first, capped at 1000 blocks per page); record body = record title + rendered property values (all DatabaseProperty types; relations rendered as linked-record titles); comment body = comment text; file body = file name + mime type; database body = database title + property names; user body = display name + email.
4. Permission snapshot: computed by packages/permissions (inherit workspace role → team grants → explicit Permission rows, granteeType user/team/role, levels NONE/VIEW/COMMENT/EDIT/FULL_ACCESS) and stored on the doc as a compact snapshot: { scope: "private"|"workspace"|"team"|"public", teamId?, grantees: [{type, id, level}] }. permission.changed events must re-index every affected doc (resolved via Permission rows + descendant pages for page grants).
5. Delete semantics: page.deleted (soft/trash) removes the doc; a page re-indexed after restore re-adds it. Record archive (soft) removes; purge removes from mirror too. block-level deletes are handled by re-indexing the parent page (blocks are indexed as part of page body, plus a separate block doc per block containing rich text > 0 chars for block-level hits).
6. Every index operation writes/updates the search_documents mirror row (upsert on [type, refId], indexedAt = now). The mirror is the reconciliation source of truth.
7. Reconciliation: apps/worker/src/processors/maintenance/ repeatable BullMQ job search:reconcile (every 15 min, repeat: { pattern: "*/15 * * * *" }): for each workspace, count search_documents rows per type vs. OpenSearch _count per type; on drift > 0, re-index the affected type in batches of 500 (log per-batch progress to search:reconcile:<workspaceId> metric + structured log).
8. Idempotency: events are at-least-once; index upserts keyed by <type>:<refId> must be safe to replay. Poison events (missing source rows, e.g. purged before indexing) are logged, counted, and acked after 5 dead-letter attempts to search-index:dlq.

Acceptance Criteria

• Given a created page with 3 blocks, a page.created event produces an OpenSearch doc whose body contains all 3 block texts and a mirror row with matching indexedAt (integration test, tests/integration/search-index.spec.ts).
• A record.updated event rewrites the record doc's body to include the new property values within 5 s (test with Redis stream publish + poll).
• page.deleted removes the doc; restoring the page re-adds it (integration test, both directions).
• permission.changed on a page grants a team VIEW and the team's member sees the doc in a permission-filtered query; the grantee's member list in the doc permissions snapshot contains the team grant (unit + integration).
• Reconciliation: corrupt the index by deleting 10 docs directly in OpenSearch, trigger the job, and all 10 are restored with the mirror count and index count equal (integration test with job invoked directly).
• Replay the same event 3×; the doc is indexed exactly once (same doc version, no duplicates — GET _count stable).
• A page with > 1000 descendant blocks is indexed (body truncated at 1000 blocks, warning logged) without failing the job.

Backend Changes

• apps/worker/src/processors/search-index/: processor.ts, mappers/ (one mapper per doc type: page.mapper.ts, record.mapper.ts, database.mapper.ts, comment.mapper.ts, file.mapper.ts, user.mapper.ts), body.ts (block concatenation + property rendering), permissions.ts (snapshot builder calling packages/permissions).
• apps/worker/src/processors/maintenance/reconcile-search.ts + job registration in apps/worker/src/queues.ts (BullMQ repeatable).
• packages/search/src/indexer.ts (new): upsertDoc, deleteDoc, countByType, bulkUpsert, bulkDelete wrappers over packages/search/src/client.ts; index name template search-ws-{workspaceId} (one index per workspace) or single search-{env} index with workspaceId field — decision: single index factory-{env} with workspaceId field for simpler cross-workspace reconciliation, recorded in docs/architecture/search.md.
• Event handler registry entries mapping the 12 event names to mappers in apps/worker/src/events/handlers/search-index.ts.

Frontend Changes

None (indexer is worker-side; UI is TASK-0098).

Database Changes

• No schema changes: SearchDocument (search_documents) already exists. Add migration-only index if missing: verify @@unique([type, refId]) and @@index([workspaceId, type]) (present in schema).
• OpenSearch index factory-{env} created on worker boot (idempotent createIndexIfMissing using mapping from packages/search/src/mapping.ts).

API Changes

None (internal worker pipeline; public API is TASK-0097).

Tests

• Unit: body composition (block concatenation order/cap, property value rendering per all 22 PropertyTypes), permission snapshot builder (inheritance chains, team + role + user grants), mapper output shape.
• Integration (tests/integration/search-index.spec.ts): event → doc for all 7 types; delete + restore; permission.changed re-index; replay idempotency; reconciliation drift repair (10 docs deleted, restored in one run).
• Fixture: seed a 2-workspace dataset used by TASK-0097.

Security

• The permission snapshot must never widen visibility: docs for private pages get scope: "private" and only explicit grants; the query layer (TASK-0097) is the enforcement point — the snapshot is data, not policy.
• No secrets in docs; user docs must not include emails of users who have disabled profile visibility (check User settings at map time).
• Stream consumer uses the shared Redis ACL (no additional grants).

Documentation

• docs/architecture/search.md (new): index layout, doc shape, permission snapshot semantics, reconciliation design, replay/DLQ policy, and the single-index-vs-per-workspace decision.
• docs/development/search.md: how to flush the index locally (pnpm search:reset) and inspect docs.

Dependencies

• Phase 3 (pages, trash/restore), Phase 5 (databases/records), Phase 7 (comments), Phase 9 file model (files indexed; file pipeline may post-date — mapper must tolerate missing thumbnails).
• packages/search client + mapping; packages/permissions; @factory/shared/events.
• @factory/config OpenSearch + Redis env vars (Phase 0).

Definition of Done

• [ ] All 12 event handlers implemented and registered; no event left unhandled.
• [ ] Doc shape + permission snapshot match the spec in Requirements §2–4.
• [ ] Mirror row written on every index/delete operation.
• [ ] Reconciliation job registered (15 min), drift repair proven by test.
• [ ] DLQ + retry policy (5 attempts) implemented and logged.
• [ ] Unit + integration tests green; pnpm typecheck + pnpm lint green.
• [ ] docs/architecture/search.md written; decision on index layout recorded.
• [ ] No secret material or PII beyond spec in indexed docs.

---

TASK-0097

Title

Full-text search API: POST /api/v1/search with permission filtering, filters, cursor pagination, highlights

Objective

Provide the search endpoint used by the UI and (in Phase 13) the public API: BM25 multi_match + prefix + fuzzy matching over the 7 doc types, with type/workspace/team/updated-date/creator filters, query-time permission filtering via the doc permission snapshot, cursor pagination, and highlight snippets — response shape per docs/api/search.md.

Background

The indexer (TASK-0096) publishes docs with workspaceId, type, title, body, permissions snapshot, metadata (incl. teamId, createdBy, updatedAt, pageId, databaseId). packages/search/src/query-builder.ts scaffolds the OpenSearch DSL builder; src/rrf.ts is reserved for Phase 12 hybrid fusion (not used by this endpoint yet). The NestJS search module scaffold lives in apps/api/src/modules/search/. Auth middleware and the @factory/shared error model (404 vs 403 semantics per docs/api/authentication.md) are Phase 1. LIMITS.MAX_SEARCH_LIMIT = 50 (packages/shared/src/constants.ts) caps page size.

Requirements

1. Endpoint: POST /api/v1/search (NestJS controller apps/api/src/modules/search/search.controller.ts, service search.service.ts, zod DTOs in dto/search.dto.ts). Auth required (session JWT or — Phase 13 — API key).
2. Request body: { query: string (1..200 chars, trimmed), types?: ("page"|"block"|"database"|"record"|"comment"|"file"|"user")[], workspaceId?: string, teamId?: string, updatedAfter?: ISO-8601, updatedBefore?: ISO-8601, createdBy?: userId, scope?: "workspace"|"team", cursor?: string, limit?: 1..50 (default 20) }. Validate with zod; reject unknown types values.
3. Query construction (packages/search/src/query-builder.ts): bool query — must: multi_match on [title^3, body] (best_fields) combined with a prefix sub-query (minimum should 1) and a fuzzy sub-query (fuzziness: AUTO, prefix_length: 1) on title (boost 2); filter: term workspaceId (default: all workspaces the caller is a member of — never a single implicit one), term metadata.teamId when teamId set, range metadata.updatedAt for the date window, term metadata.createdBy, and the permission filter (see §5).
4. Cursor pagination: response nextCursor = base64 of { score, id } (search_after on [score, _id]); limit ≤ LIMITS.MAX_SEARCH_LIMIT; track_total_hits capped at 10 000.
5. Permission filtering at query time: build a bool should filter from the caller's identities — caller's user id, the user's team ids per requested workspace scope, and workspace role — matched against doc fields permissions.granteeId, permissions.granteeType, permissions.teamId, permissions.scope: scope:"public" → always visible; scope:"workspace" + caller is a member → visible; scope:"team" + doc permissions.teamId ∈ caller teams → visible; scope:"private" + grantee entry (user/team/role) matching caller with level ≥ VIEW → visible. minimum_should_match: 1 on that should group. No doc-level application of permissions in the UI (server is the source of truth).
6. Response shape (documented in docs/api/search.md): { results: [ { id, type, workspaceId, title, snippet (HTML with <mark> from OpenSearch highlight on body, 160 chars), icon?, metadata: { pageId?, blockId?, databaseId?, recordId?, teamId?, updatedAt, createdBy, size?, mimeType? }, permissions: { level: caller's effective level } } ], total: number, nextCursor: string | null, tookMs: number }. Snippet for block hits links back via metadata.blockId (+ page id).
7. tookMs is measured server-side (query time only). Structured log per request: workspaceId set, types, tookMs, total, query hash (no raw query text in logs beyond 64 chars).
8. Errors: 401 unauthenticated; 422 invalid body (zod); 503 with error code SEARCH_UNAVAILABLE when OpenSearch is down (do not leak backend details).

Acceptance Criteria

• Unit (Vitest, packages/search/src/query-builder.spec.ts): builder emits the expected DSL for each combination (types filter, team, date range, createdBy, scope); permission group matches only the caller's identities.
• Integration (tests/integration/search-api.spec.ts): user A in workspace 1 and user B in workspace 2; query by B never returns workspace-1 docs even when the query text matches exactly (cross-workspace isolation, proven with a deliberately unique phrase).
• Integration: private page (scope "private", only team X granted) is returned for team-X members and absent for other workspace members and other workspaces (private-page isolation).
• Integration: types: ["record"] returns only records; updatedAfter excludes older docs; createdBy filters correctly; limit: 51 rejected with 422 listing the constraint.
• Cursor round-trip: page 1 + page 2 of a > 50-result query are disjoint, ordered, and cover all top hits; invalid/base64-garbage cursor → 422.
• Highlight: a doc containing "quantum flux capacitor" returns a snippet containing <mark>quantum</mark> (assert HTML, stripped for UI).
• p95 sanity in integration env: 100 sequential queries over 1k docs each < 500 ms (soft assertion; hard target is TASK-0100 at 100k docs).

Backend Changes

• apps/api/src/modules/search/: search.controller.ts, search.service.ts, dto/search.dto.ts (zod), search.module.ts (inject OpenSearchClient from @factory/search).
• packages/search/src/query-builder.ts: implement buildSearchQuery(input, caller: { userId, teams: Record<workspaceId, string[]>, roles: Record<workspaceId, WorkspaceRole> }) + buildPermissionsFilter(caller).
• apps/api/src/modules/search/cursor.ts: encode/decode search_after cursors (constant-time base64 decode, 64-char cap).
• Caller identity resolver: from apps/api/src/modules/auth (Phase 1) — getCallerWorkspaceScope(req) returning member workspaces, team ids, roles.
• docs/api/search.md (new): full request/response contract, filter table, error codes, examples (curl + TS), and the permission-filter semantics.

Frontend Changes

None (endpoint only; UI is TASK-0098).

Database Changes

None (reads OpenSearch; no Prisma writes).

API Changes

• POST /api/v1/search as specified (new). Auth: session JWT (Phase 1) — API-key support added in Phase 13 without changing the contract.
• OpenAPI fragment contributed to the Phase-13 spec generator (operation search); document under docs/api/search.md now.

Tests

• Unit: query builder DSL snapshots (5 cases), cursor encode/decode round-trip
  • garbage rejection, zod DTO validation matrix (7 cases).
• Integration: the 7 scenarios in Acceptance Criteria (isolation, filters, cursor, highlight, 503 path with OpenSearch stopped via test-only client).
• Load smoke (Artillery, local only, not CI-blocking): 20 rps × 60 s.

Security

• Permission filter is mandatory: the service must throw (fail closed) if caller identity cannot be resolved — never fall back to an unfiltered query.
• Never log full query text beyond 64 chars; cursor does not encode workspace-scoped secrets.
• OpenSearch credentials come from @factory/config env; no client-side OpenSearch access (the API is the only gateway).
• 404-vs-403: search results simply omit invisible docs (no existence leak); a workspaceId the caller is not a member of yields 404 NOT_FOUND workspace (consistent with pages API).

Documentation

• docs/api/search.md (new, contract of record for Phase 13).
• Update README.md feature list (Search) with the endpoint.

Dependencies

• TASK-0096 (indexer + mapping + fixtures).
• Phase 1 (auth: JWT middleware, error model), Phase 2 (workspace membership
  • teams for the caller scope), packages/shared limits.
• @factory/search client wiring (Phase 0 env).

Definition of Done

• [ ] Endpoint live with zod validation, auth, and all 6 filters.
• [ ] Query-time permission filtering implemented and proven by the 2 isolation integration tests.
• [ ] Cursor pagination stable (disjoint, ordered pages).
• [ ] Highlights returned as bounded HTML with <mark>.
• [ ] tookMs measured and logged; 503 path tested.
• [ ] docs/api/search.md merged and matches live responses (CI diff in Phase 13).
• [ ] Unit + integration green; typecheck + lint green.

---

TASK-0098

Title

Global search UI: /search page with grouped results, deep links, keyboard navigation, recents, suggestions

Objective

Ship the global search experience: a debounced search page that groups results by type with icons and highlighted snippets, deep-links each hit to the right place (block → page + scroll to block, record → database + row), supports full keyboard navigation, remembers the last 20 searches per user in Redis, and shows suggestions while typing.

Background

The API exists per TASK-0097 (POST /api/v1/search), plus a lightweight suggestions endpoint implemented in the same module (POST /api/v1/search/suggestions: prefix completions over title with completion/prefix query, max 8, same permission filter). Recent searches are stored server-side: GET/PUT /api/v1/search/recent backed by a Redis list search:recent:{userId} (cap 20, newest first, 30-day TTL per entry). The web app is Next.js 14 under apps/web/; the sidebar (Phase 2) has a Search entry; the Cmd/Ctrl+K binding (Phase 4/7 editor keymap) routes to /search and focuses the input. Route structure follows existing app-router conventions in apps/web/src/app/.

Requirements

1. Page: apps/web/src/app/(workspace)/search/page.tsx (client component) + apps/web/src/components/search/ (SearchInput.tsx, ResultGroup.tsx, ResultItem.tsx, RecentSearches.tsx, SuggestionList.tsx, search-state.ts reducer). Debounce input 250 ms; cancel in-flight requests (AbortController); ignore responses out of order.
2. Results grouped by type in fixed order: Pages, Databases & Records, Files, Comments, Users; each group collapsible with a count badge; icons per type (reuse @factory/ui icon set: page, database, file, comment, user). Each item: title (highlight <mark> rendered as <mark> styling), snippet (max 2 lines, ellipsis), metadata line (parent page / database name / updated relative time / author avatar).
3. Deep links: page → /p/{pageId}; block hit → /p/{pageId}#block-{blockId} (editor scrolls + flashes the block, Phase-4 scroll anchor); database → /p/{pageId}?view={table}; record → /p/{pageId}?view={table}&row={recordId} (row flash); file → file preview route /files/{fileId} (TASK-0103); comment → /p/{pageId}#comment-{commentId}; user → member profile popover. Clicking a result records it in recent searches.
4. Keyboard navigation: ↑/↓ move active result across groups (group order = render order), Enter opens active result, Esc clears/focuses input, / focuses the input when on the page, type-ahead filters visible groups. Active state announced via aria-activedescendant + visible focus ring; every clickable result reachable and operable via keyboard (WAI-ARIA listbox pattern).
5. Recent searches: on empty input, show the last 20 (from GET /api/v1/search/recent) with a clock icon + per-item clear (×) + "Clear all"; clicking runs that query; recording happens server-side on result click and on Enter.
6. Suggestions: while typing (after 2 chars, 250 ms), a dropdown above results with up to 8 title completions (from /search/suggestions), each with type icon; arrow keys navigate the dropdown first, then results; Tab/Enter fills the query.
7. States: loading (skeleton rows, input spinner, "Searching…"), empty ("No results for '<q>'" + tips: check spelling, try fewer words, filter by type), error (inline banner with retry button; do not clear input), rate-limited (429 → "Too many searches, slow down" toast).
8. Type filter chips (All / Pages / Databases / Files / Comments / Users) toggling the types param; selected chip persisted in the URL (?type=page) for shareable links; workspace/team/date filters are out of scope for Phase 8 UI (API supports them; UI in Phase 13 search settings) — note recorded in the task file.
9. Mobile: full-width input, groups stacked, snippet 2 lines, touch targets ≥ 44 px; no horizontal scroll; keyboard nav degrades gracefully.

Acceptance Criteria

• E2E (tests/e2e/search.spec.ts, scenario "grouped results + deep link"): seed 3 pages + 1 record + 1 file with unique phrases; type the phrase → results grouped correctly with icons and <mark> highlights; click the block-level hit → lands on the parent page with the target block scrolled into view and flashed (assert scroll position + flash class).
• E2E: record hit opens the database table view with the target row flashed (assert ?row= param + flash).
• E2E keyboard: type query, press ↓×3 + Enter → 4th result opened; Esc clears; / focuses input from any page area.
• Recents: after 2 distinct searches, reload /search with empty input → both entries listed newest-first; × removes one (server list length 1); "Clear all" empties the list (integration test on the Redis-backed endpoint
  • E2E assertion).
• Suggestions: typing "qu" shows completions for "Quantum…"; Enter fills the query and runs the search.
• Error state: mock the API to 500 → banner + retry button appears, input preserved, retry re-fires the request (Playwright route mock).
• Debounce: 3 rapid keystrokes within 500 ms fire exactly 1 request (assert request count via Playwright route handler).
• Lighthouse/mobile: page renders at 390 px viewport without horizontal overflow (Playwright screenshot diff + overflow assertion).

Backend Changes

• apps/api/src/modules/search/suggestions.controller.ts + service: POST /api/v1/search/suggestions { query (2..100), limit? ≤ 8 } → { suggestions: [{ id, type, title }] } (prefix query on title, same permission filter as TASK-0097).
• apps/api/src/modules/search/recent.controller.ts: GET /api/v1/search/recent → { queries: string[20] }; PUT /api/v1/search/recent { query } → upsert (dedupe, cap 20); Redis list search:recent:{userId} with 30-day TTL; per-user rate limit 30 writes/min.
• Both endpoints documented as addenda in docs/api/search.md.

Frontend Changes

• New route apps/web/src/app/(workspace)/search/page.tsx and apps/web/src/components/search/* (7 files per Requirements §1).
• apps/web/src/lib/api/search.ts: typed client (fetch, AbortController, cursor + suggestions + recent).
• Sidebar (Phase 2 component) Search entry → /search; Cmd/Ctrl+K global keybinding (registered in the app keymap, Phase-4 module) → navigate + focus.
• Editor scroll-anchor helper scrollToBlock(pageId, blockId, { flash: true }) in apps/web/src/lib/editor/anchors.ts (reused by deep links).

Database Changes

None (Redis only; no Prisma changes).

API Changes

• POST /api/v1/search/suggestions, GET/PUT /api/v1/search/recent (new, session-authenticated). Contract appended to docs/api/search.md.

Tests

• Unit: search-state.ts reducer (debounce/cancel/order, group ordering, keyboard index math across collapsible groups), suggestion list model.
• Integration: recent endpoint (cap 20, dedupe, TTL, rate limit 30/min), suggestions permission filtering (private doc not suggested).
• E2E: all 8 scenarios in Acceptance Criteria.

Security

• Suggestions and recents are server-side permission-filtered (same filter builder as TASK-0097) — a user's recents can never contain a query they no longer have access to show results for (results simply re-filter).
• Highlight HTML is sanitized before rendering (allow only <mark>); never dangerouslySetInnerHTML on unsanitized server output.
• Recents stored per user only; no cross-user enumeration; per-user rate limit on write.

Documentation

• docs/api/search.md addenda (suggestions + recent endpoints).
• docs/ui/search.md (new): interaction spec, keyboard map, group order, state matrix; note on deferred workspace/team/date filter UI (Phase 13).

Dependencies

• TASK-0097 (search + filters + cursor), TASK-0096 (index populated).
• Phase 2 sidebar, Phase 4 editor (scroll anchors, keymap), @factory/ui icons, @factory/shared error toast.

Definition of Done

• [ ] /search page with debounce, cancel, grouping, icons, highlights.
• [ ] Deep links for all 7 types incl. block scroll/flash and record row flash.
• [ ] Full keyboard map + ARIA listbox semantics; / and Cmd/Ctrl+K wired.
• [ ] Recent searches (20, Redis, per-user) with clear-one/clear-all.
• [ ] Suggestions while typing (8, permission-filtered).
• [ ] Loading/empty/error/rate-limit states all present and tested.
• [ ] Mobile 390 px verified; unit + integration + E2E green; lint + typecheck green.
• [ ] docs/ui/search.md + API addenda merged.

---

TASK-0099

Title

Hybrid AI search foundation: dense_vector mapping, chunker wiring, stubbed embedding pipeline (RAG in Phase 12)

Objective

Prepare the search stack for hybrid (keyword + semantic) search without changing Phase-8 behavior: add a bodyVector dense_vector field to the OpenSearch mapping, wire the existing chunker into the indexer, and stub the embedding pipeline as a job in the worker ai queue — so Phase 12 only implements the embedding provider call and the RAG endpoint.

Background

packages/search already scaffolds a chunker (src/chunker.ts) and RRF fusion (src/rrf.ts); the OpenSearch mapping lives in packages/search/src/mapping.ts (created by TASK-0096). The worker ai queue is scaffolded (apps/worker/src/processors/ai/); packages/ai is the future provider-agnostic AI gateway (Phase 12). Prisma has AiEmbedding (ai_embeddings: model/workspace-level cache of embeddings — verify fields when implementing; reuse, do not duplicate). This task deliberately does NOT ship semantic search: no UI, no query path change.

Requirements

1. Mapping: add to factory-{env} index mapping: bodyVector: { type: "dense_vector", dims: 1536, index: true, similarity: "cosine" } + chunkId: keyword + chunkIndex: integer + chunkModel: keyword (embedding model tag, e.g. text-embedding-3-small). Index settings: HNSW m: 16, ef_construction: 100 (documented, tunable via env SEARCH_KNN_M/SEARCH_KNN_EF_CONSTRUCTION).
2. Reindex compatibility: dims is immutable — the migration path is a new alias rollout: create factory-{env}-v2 with the new mapping, alias factory-{env} → v2, backfill from the search_documents mirror (reuse TASK-0096 reconcile path), then delete v1. Implement as a one-off script scripts/opensearch-reindex.ts + pnpm search:reindex (documented, idempotent, resumable in 500-doc batches).
3. Chunker wiring: during indexing (TASK-0096 mappers), split page/record bodies with chunker.ts (target 800 tokens, 15% overlap; chunks < 100 tokens merged with the next) and store chunk metadata on the doc (metadata.chunks: [{ chunkId, charStart, charEnd }] — chunk text itself stays in the body field for keyword search; vector docs are separate: one OpenSearch doc per chunk { id: "<type>:<refId>:chunk:<chunkId>", type: "chunk", ...parent fields, body: chunkText, chunkId, chunkIndex } with the same permission snapshot. Chunk docs are excluded from keyword search results (type: "chunk" never in the public types filter — keyword search over whole-doc body stays unchanged).
4. Embedding stub: apps/worker/src/processors/ai/embedding-pipeline.ts consuming queue ai, job embed:chunks { workspaceId, docType, refId, chunkIds[] }: the stub (a) validates the payload, (b) writes an AiEmbedding cache placeholder row only if a real provider is configured (env AI_EMBEDDING_PROVIDER absent → log "embedding disabled (Phase 12)" and skip), (c) leaves bodyVector null. Phase 12 replaces step (b) with the gateway call.
5. KNN query path stub: packages/search/src/query-builder.ts gains buildHybridQuery(keyword, vector, weights) using rrf.ts (reciprocalRankFusion(keywordHits, knnHits, k=60)) — fully unit-tested with synthetic hits, but NOT wired into POST /api/v1/search (that wiring is Phase 12 behind a capability flag).
6. Config: env additions in .env.example — SEARCH_KNN_M, SEARCH_KNN_EF_CONSTRUCTION, AI_EMBEDDING_PROVIDER (empty by default), AI_EMBEDDING_MODEL (default text-embedding-3-small), validated by @factory/config zod.

Acceptance Criteria

• After pnpm search:reindex in dev, the live alias points to a v2 index whose mapping includes bodyVector (1536, cosine, HNSW settings) and all v1 docs are present (count equality vs. mirror) — scripted + asserted by an integration test that inspects the mapping via the client.
• A 5 000-char page indexes as 1 whole-doc + N chunk docs (N ≥ 5, chunks within 100-token tolerance, overlap present — chunk boundaries asserted via metadata.chunks offsets); chunk docs carry the parent's permission snapshot (private page → chunk docs also private).
• POST /api/v1/search is behaviorally unchanged: same fixture dataset, same 10 queries, same top-10 ids before/after this task (regression test).
• embed:chunks job with AI_EMBEDDING_PROVIDER unset: job completes, logs "embedding disabled (Phase 12)", bodyVector remains null, no AiEmbedding rows created, no crash (unit/integration test).
• rrf.ts + buildHybridQuery: unit tests prove RRF ordering with synthetic keyword/knn hit lists (tie handling, k=60, weight blend 0.5/0.5) and that the hybrid DSL contains both multi_match and a knn clause.
• .env.example + @factory/config validation: missing SEARCH_KNN_M falls back to default 16 (documented); invalid value fails boot with a clear zod message (unit test).

Backend Changes

• packages/search/src/mapping.ts: v2 mapping + HNSW settings + alias helper ensureIndexAlias(version).
• packages/search/src/query-builder.ts: buildHybridQuery + RRF integration.
• packages/search/src/chunker.ts: implement (or complete) the chunker per spec (800 tokens / 15% overlap / merge < 100) — tokenization via a shared tokenizer (whitespace + CJK-aware, no new heavy dependency; justify any addition in the PR).
• apps/worker/src/processors/search-index/: chunk emission in mappers/*.ts (page + record only in Phase 8; comment/file chunking deferred — record decision).
• apps/worker/src/processors/ai/embedding-pipeline.ts (stub) + job type in apps/worker/src/queues.ts.
• scripts/opensearch-reindex.ts + package.json script search:reindex.
• @factory/config env schema additions; .env.example update.

Frontend Changes

None (explicitly no UI in Phase 8 for semantic search).

Database Changes

• AiEmbedding model already exists; verify its shape supports (workspaceId, docType, refId, chunkId, model, vector, createdAt) and add a migration ONLY if a field is missing (expected: none).
• No other schema changes.

API Changes

None (no endpoint change; hybrid query exposed in Phase 12).

Tests

• Unit: chunker boundary cases (short doc, CJK text, exactly-800, merge logic), RRF ordering (5 synthetic cases), config defaults/invalid.
• Integration: reindex script count equality + mapping assertion; chunk doc permission snapshot; embedding stub skip path; search regression (10 queries, identical top-10).

Security

• Chunk docs inherit the parent permission snapshot — verified by test; a private page must not leak through its chunks (the Phase-12 hybrid query must apply the same permission filter; contract noted in docs/architecture/search.md).
• Reindex script reads OpenSearch credentials from env only; no keys in logs; script refuses to run against an index it didn't create v2 for (guard flag).

Documentation

• docs/architecture/search.md section "Hybrid foundation": v2 mapping, chunk doc model, RRF contract, Phase-12 wiring checklist, reindex runbook.
• .env.example comments for the new vars.

Dependencies

• TASK-0096 (mapping v1, mirror, mappers), packages/ai scaffold (no calls).
• Phase 12 consumes this: embedding provider, RAG endpoint, hybrid wiring.

Definition of Done

• [ ] v2 mapping + alias rollout + reindex script working and tested.
• [ ] Chunker implemented; page/record chunk docs emitted with parent permission snapshots.
• [ ] Keyword search regression: zero behavior change (test proves it).
• [ ] Embedding stub job runs, logs, skips cleanly without a provider.
• [ ] RRF + buildHybridQuery unit-tested (Phase 12 just flips the wire).
• [ ] Env schema + .env.example updated; docs merged; CI green.

---

TASK-0100

Title

Search performance verification: index lag < 30 s, query p95 < 500 ms @ 100k docs, create→find E2E

Objective

Prove the Phase-8 performance exit criteria with repeatable suites: end-to-end index lag under 30 s, query p95 under 500 ms with 100k indexed documents, and the canonical E2E (create a page with a unique phrase → searchable within 30 s → click → correct page).

Background

TASKS-0096…0099 ship the pipeline, API, UI, and hybrid foundation. tests/performance/ is the home for k6/Artillery suites (Phase-0 bootstrap); tests/e2e/ hosts Playwright. The Docker dev environment provides OpenSearch/Redis/Postgres/MinIO. 100k-doc fixture generation must be fast enough for CI (target: seed in < 10 min) and resettable.

Requirements

1. Index lag measurement: E2E helper that (a) records t0 before creating a page (via API) whose body contains a globally unique phrase (perf-<runId>-<uuid>), (b) polls POST /api/v1/search every 2 s, (c) passes when the page appears in results, failing if not found within 30 s; reports actual lag. Same helper asserts a deleted page disappears from results within 30 s.
2. Query performance suite: tests/performance/search-search.k6.js (k6) — dataset: 100k docs seeded via tests/performance/fixtures/search-100k.ts (10k pages × 7 types mix: 40% pages, 25% records, 15% files, 10% comments, 5% databases, 5% users; realistic body sizes 200–2000 chars; 2 workspaces, 200 users, permission mix 50% workspace / 25% team / 25% private). Load profile: 100 VUs, 5 min, Poisson arrivals 50 req/s; assertions: p95 < 500 ms, p99 < 1 s, error rate < 0.1%; measured query tookMs also logged server-side for cross-check.
3. Seed/reset tooling: pnpm perf:seed:search (idempotent: drops perf-* fixtures first) and pnpm perf:reset:search; fixtures excluded from normal test DBs (tagged perf, separate workspace ids) so unit/integration runs stay fast.
4. E2E scenario (tests/e2e/search.spec.ts, scenario "index lag + click through"): logged-in user creates a page (UI) titled "Perf Probe {uuid}", body one sentence with the unique phrase; within 30 s the search box finds it (assert result row); clicking it lands on the page and the page title matches; delete the page → within 30 s no longer returned.
5. Baseline + regression gating: store the last run's p95/p99/lag in tests/performance/reports/search-last.json (committed on demand by a maintainer, not per-PR); CI runs the k6 suite with a reduced profile (10 VUs × 60 s) on merge to main, full profile nightly; a p95 regression

20% vs. baseline fails the nightly with a report link.

6. OpenSearch tuning notes: record the measured index refresh interval (default 1 s — keep; do NOT tune to 30 s for the lag SLA) and any JVM heap/thread_pool settings required to hit p95 at 100k, in docs/architecture/search.md "Performance" section.

Acceptance Criteria

• Index-lag E2E (req 1) passes 10 consecutive local runs with mean lag < 15 s and never > 30 s; deletion-side lag also < 30 s.
• k6 full profile on the dev Docker env (8 vCPU / 16 GB): p95 < 500 ms and p99 < 1 s over 100k docs (report attached to the task completion note).
• pnpm perf:seed:search seeds 100k docs in < 10 min and pnpm perf:reset:search leaves the index + mirror consistent (count equality via the TASK-0096 reconcile job).
• The "create → find → click → correct page" E2E passes in CI (reduced env), asserting the exact page id in the URL after click.
• CI wiring: reduced k6 profile on merge-to-main in the existing CI contract (install → lint → typecheck → unit → integration → build → security → E2E), nightly full profile job exists and is scheduled.
• tests/performance/reports/search-last.json written by the suite with p50/p95/p99, error rate, doc count, env fingerprint (versions).
• No fixture data leaks: after reset, a search for the fixture seed phrase returns 0 results (permission-filtered and unfiltered).

Backend Changes

• apps/worker: none beyond what 0096/0099 ship (verify worker consumes a 1k-event burst without backlog growth > 30 s — assert in the lag suite via consumer-lag metric search-index:lag).
• Optional micro-tuning only if measured needed (e.g. scroll_size defaults, track_total_hits cap already 10k): each change documented with the before/after p95 in docs/architecture/search.md.

Frontend Changes

• None functional; if the search page shows a > 500 ms spinner, ensure it displays tookMs in a dev-only footer flag (gated by NEXT_PUBLIC_SEARCH_DEBUG=true) for perf triage.

Database Changes

• None (fixtures are data, not schema).

API Changes

• None.

Tests

• E2E: index-lag + click-through scenario (req 4), deletion-lag scenario.
• Performance: k6 suite (req 2/5) + seed/reset scripts (req 3) + report writer; worker consumer-lag assertion during the burst.
• Integration: none new (covered by 0096/0097).

Security

• Performance fixtures use synthetic users/workspaces with no real PII; fixture passwords never stored (users created via seed API with generated hashes, perf tag, purged by reset).
• The lag E2E runs authenticated; it must pass the permission filter (a page visible to the creator is found; a private page of another user is never found even after 30 s — assert the negative).

Documentation

• docs/architecture/search.md "Performance" section: environment used, numbers, tuning decisions, SLA definitions (lag = event → first hit; p95 = API query latency).
• docs/development/performance.md addendum: how to run seed/suite locally and read the report.

Dependencies

• TASKS-0096…0099 (pipeline, API, UI, foundation) all complete.
• Phase-0 CI + Docker; k6 + Playwright already bootstrapped.
• Phase 15 (production hardening) reuses these suites at higher load.

Definition of Done

• [ ] Lag < 30 s proven (create + delete) by E2E, 10-run stability recorded.
• [ ] p95 < 500 ms / p99 < 1 s @ 100k docs proven by k6 (full profile report attached); reduced profile green in CI.
• [ ] Seed/reset tooling < 10 min, idempotent, consistent post-reset.
• [ ] Click-through E2E green in CI.
• [ ] Nightly job + baseline report wired; regression gate documented.
• [ ] Perf docs merged with env fingerprint and tuning decisions.