Search Architecture

OpenSearch 2.x as the single search and vector engine (ADR-004). Sibling docs: system · backend · ai · database.

Pipeline in one line: PostgreSQL (source of truth) → domain events + periodic reconciliation → search indexer (worker) → OpenSearch (full-text + kNN + metadata) → permission-filtered query API → UI.

---

0. Phase 8 Implementation Notes (recorded decisions)

The Phase-0 scaffold shipped three divergent search implementations: the pure @factory/search package (this doc's target), a hand-rolled worker indexer, and a hand-rolled API module. Phase 8 unifies the worker and the API onto @factory/search so there is one tested core. The scaffold also drifted from this doc in ways the implementation now corrects:

• Layout (resolves the doc-vs-code contradiction). The index is a single index (alias factory-content) holding every typed doc, tenant-isolated by the mandatory workspaceId filter — exactly §2. The Phase-0 scaffold's per-workspace factory-search-<wid> indices were the deviation; they are retired. contentIndex() (in @factory/search) returns the alias, so a §7.2 rollover swaps the backing index behind the same alias with no caller change.
• Field names are camelCase. The stored doc, the mapping and the query builders all use camelCase (workspaceId, updatedAt, teamId, createdBy, databaseId, parentDocId). The scaffold's queries.ts had drifted to a mix of snake_case + grants.* + title.raw; that drift is removed. title is text with a title.keyword sub-field; prefix (§4.2) and fuzzy (§4.3) target title.keyword / id.
• OpenSearch 2.x vector type. The doc's mapping snippet shows the 1.x knn_vector + knn: true form. The docker + CI OpenSearch is 2.15, where the current type is dense_vector (cosine) with an HNSW index_options (m, ef_construction). contentMapping() emits the 2.x form (recorded deviation). index: { knn: true, "knn.algo_param.ef_search": 100 } is set in the index settings.
• Grant snapshot source. §5.1 says the snapshot comes from "packages/permissions.grantSnapshot". The API does not use @factory/permissions; its single permission walk is apps/api/src/lib/authorize.ts::effectiveLevel (role baseline + Permission rows, most-specific-wins, maxed with the role). The indexer's GrantSnapshot is a denormalized projection of that walk's inputs: public (active Share(scope=PUBLIC, allowSearchIndexing)), users[]/teams[] (explicit Permission rows with level ≥ VIEW), and workspace (the page/resource is not isPrivate, not trashed, not archived, and the workspace role floor is ≥ VIEW). The Phase-8 exit criterion ("search visibility == guard visibility") is pinned by a property test.
• ⚠️ Recorded conflict — the live page guard does not enforce isPrivate. effectiveLevel and the page read path (page.service.ts::list/get) apply only the workspace role floor + Permission rows; they never check Page.isPrivate or createdBy. As of Phase 8 this is a latent Phase-3 authorization gap: any workspace member (including a guest) can read/list an isPrivate page of another user. The arch doc §5.1 and the Phase-8 exit criterion both require private pages to be workspace-invisible (visible only via an explicit team/user grant or a public share). Resolution per AGENTS.md rule 29 (arch doc wins): the search snapshot implements the intended model — workspace: true only for non-private, non-trashed, non-archived content — so search correctly isolates private pages even though the live page-read guard does not. Consequences: (a) search is stricter than the current page-read endpoint for private pages (it will not surface what a member could still open directly); (b) the property test "search visibility == guard visibility" is pinned against the intended model (the arch-documented rules), not the buggy live walk; (c) the page-read isPrivate gate is filed as a Phase-3 security follow-up (see the Phase-8 task doc § Security + ROADMAP residual) — deliberately not fixed in Phase 8 (it is a Phase-3 page-permission behavior change, out of scope, and would alter existing page-read behavior).
• Vector dimension is a parameter, not a constant. The default is 1536 (text-embedding-3-small, TASK-0099). dense_vector.dimension is immutable, so a model change rolls a new index forward via the alias (scripts/opensearch-reindex.ts); the vectorModel keyword is the migration guard (§7.3). The stub provider (64-dim) is dev/test only and is never written to the production index.
• Single search writer (resolves the three-writer drift). Phase-0 left three OpenSearch writers: search-index.processor.ts (stream consumer), files.processor.ts (direct factory_files doc + mirror upsert at S3 finalization), and ai.processor.ts (chunk vectors into the 5-index scheme). All now target the single alias factory-content. files.processor.ts no longer writes a search doc directly — the stream consumer is the only doc writer, keeping the mirror + contentHash invariant intact. Consequence: file.uploaded is published by the API at presign time (the File row is still UPLOADING, mimeType/size null), and the worker's S3 finalization publishes no second event (a pre-existing Phase-3 gap, unchanged). The file doc is built leniently (name known; mime/size may lag) and converges on the next permission.changed workspace resync or the 15-min search:reconcile contentHash sweep — acceptable because mimeType/size are metadata, not full-text body.
• Chunk docs carry the vector; the keyword path is unchanged. A chunk doc (type: "chunk", id <type>:<refId>:chunk:<chunkId>) holds body = chunk text + vector + the parent's grant snapshot. Keyword search excludes type: "chunk" implicitly (the public types filter never includes it), so whole-doc body keyword results are unchanged (TASK-0099). The worker's ai.processor.ts vector write is redirected to the chunk doc id in factory-content (partial doc update); TASK-0099 formalizes the embed:chunks job.
• Reconciliation = contentHash sweep, not a count heuristic. TASK-0096 §7 proposed counting search_documents rows vs OpenSearch _count per type. Arch doc §7.3/§1.3 specifies the stronger contentHash sweep: recompute SHA-1(title+body+permissions+metadata) from the source, compare against the mirror's contentHash, reindex drift. This subsumes the count check (a doc can be present but stale) and is what "the mirror is the reconciliation source of truth" (TASK-0096 §6) actually requires. Implemented as the search:reconcile repeatable job (15 min, batches of 500). The search_documents mirror therefore gains a contentHash String? column (migration; arch doc §1.3 already lists it — the Phase-0 schema lacked it).
• Typecheck baseline at HEAD (recorded per AGENTS.md rule 29). At the Phase-8 start commit, tsc --noEmit is already red in Phase-2–7 code this phase does not touch: @factory/worker 47 errors, @factory/api 146 (databases/query.service 34, pages/page.service 16, blocks batch tests 15, teams 11, comments 11, …) — the same class of pre-existing drift Phase 7's close-out measured ("web: 219 pre-existing drift errors in untouched files, unchanged count; api typecheck: 149 baseline, 0 in the modified auth files"). Phase-8 scope follows the documented Phase-7 convention: the search code the phase owns is typecheck-clean (packages/search 0; the rewritten worker/api search files 0); the pre-existing non-search baseline is unchanged — not faked, not expanded (fixing 253 cross-phase errors is out of Phase-8 scope). packages/search itself goes fully clean (0 errors, 72 tests) — stronger than net-zero.

The rest of this doc (pipeline §1, schema §3, query API §6, ops §7) is the contract the unified implementation meets; where this doc and the live code disagreed, the code now matches the doc and the divergence is noted here.

---

1. Indexing Pipeline

                    domain events (Redis stream factory:events)
   page.created ──┐  page.updated ──┐  record.created ──┐  comment.created ──┐  file.uploaded ──┐
                  ▼                 ▼                   ▼                    ▼                  ▼
   ┌────────────────────────────────────────────────────────────────────────────────────────────┐
   │ search-index consumer group (apps/worker, concurrency 8)                                   │
   │                                                                                            │
   │  1. load envelope → resolve source row (Postgres, read replica)                            │
   │  2. extract: title, body (text of content JSONB), metadata (creator, dates, type, team)    │
   │  3. evaluate visibility: packages/permissions.grantSnapshot(resource, workspace)           │
   │       → {public: bool, workspace: bool, teams: [id], users: [id], level}                    │
   │  4. build OpenSearch doc (schema §3) + search_documents mirror row (upsert)                │
   │  5. for page/block/record: also enqueue ai-index job (chunk + embed, §6 of ai-arch)        │
   │  6. OpenSearch bulk: index (upsert by id) / delete                                          │
   │  7. XACK the event                                                                        │
   └────────────────────────────────────────────────────────────────────────────────────────────┘
                  ▲                                        ▲
                  │                                        │
   reconciliation sweep (every 5 min, per workspace,      repair endpoint
   staggered) ────────────────────────────────────────────┘  (POST /v1/admin/search/reindex)
   • scan search_documents indexedAt older than (now - 5 min) whose source hash changed
   • scan source rows with updatedAt > lastSweepWatermark not present in the index
   • emit synthetic index/delete events for the drift set

1.1 Event → document rules

Event	Doc action	Notes
page.created / page.updated (metadata)	upsert page:{id}	title/description/indexed; body excludes child-page content (children are their own docs)
block.updated (realtime persist or REST)	upsert block:{id} and refresh page:{pageId} body (debounced 30 s per page in the consumer — coalesces keystroke storms)	the page doc's body is a concatenation of live blocks' text (truncated at 100 KB); block docs give per-block ranking for snippet UI
page.trashed / page.deleted	delete page:{id} + all descendant block:{id} (descendants fetched by tree walk, batched deletes)	trashed = deleted from index (search must not surface trashed content)
database.created / database.updated	upsert database:{id}	title + description only (rows are record: docs)
record.created / record.updated	upsert record:{id}	title = title property value; body = concatenated non-computed property values (select names, text, …); pageId present → the record page is a separate page: doc for its blocks
record.archived	keep indexed, archived: true (UI can filter)	archived ≠ deleted
comment.created	upsert comment:{id}	body = comment text (search covers comments per PRODUCT-SPEC §15)
file.uploaded	upsert file:{id} when metadata extraction produced text (PDF/docx via the file pipeline's extractor)	binary-only files index name + mime only
share.created (scope PUBLIC, allowSearchIndexing)	set public: true on the target doc	public search (future: workspace-public search index)
permission.changed	re-evaluate + re-stamp every doc in the affected workspace that grants or denies the changed grantee (bounded batch: docs whose permissions snapshot includes the grantee/resource chain)	this is the only push permission path; the rest is stamped at index time
user.*	upsert user:{id} (name, display name — for "search people" in mentions)

1.2 Idempotency & ordering

• Doc id = {type}:{refId} → every event upserts the same id; replay is safe.
• Envelope eventId is recorded in the mirror row (metadata.lastEvent); a consumer that sees an older occurredAt for the same id skips the write (out-of-order guard — the stream is per-consumer ordered, but late duplicates from XAUTOCLAIM happen).
• Source-row read happens after the event; the consumer always writes the current row state (last-write-wins by updatedAt), so a burst of page.updated events converges to the latest state without re-reading history.

1.3 Mirror table & why it exists

search_documents (Postgres) is the reconciliation anchor: it stores (type, refId, title, body, permissions, metadata, indexedAt, contentHash). The sweep compares contentHash (SHA-1 of title+body+permissions+metadata) against the source rows' current projection. OpenSearch is treated as a cache that may be wrong; Postgres is never wrong. The mirror also powers the admin repair UI (show me what's drifted) without querying OpenSearch.

1.4 Lag SLA (< 30 s)

• Definition: time from source-row commit (Postgres) to doc being searchable.
• Budget: event publish < 5 ms (in-tx XADD attempt, 3 retries → buffer) → consumer pickup < 5 s (concurrency 8, batch 32) → OpenSearch refresh_interval 1 s on the content index → total p95 < 30 s with 20× headroom.
• Measurement: the indexer stamps indexedAt; a monitor job samples 50 recent source rows per workspace per minute and measures indexedAt - updatedAt; the search_lag_seconds histogram (p50/p95/p99) is a Phase-15 alert target (page alert at p95 > 30 s for 10 min; index-lag dashboard per consumer group via XPENDING age).
• Backlog behavior: under backlog, consumers batch bigger (256) and the page-doc debounce widens to 60 s (keeps up with volume); lag grows visibly and alerts fire — the SLO is a capacity SLO, and the dashboard shows it, not a hidden failure.

---

2. Index & Cluster Layout

Index	Contents	Settings
factory-content-v{n} (alias factory-content)	all typed docs (§3 schema), single index	refresh_interval: 1s, 1 primary + 1 replica (2+), knn: true, HNSW on vector (m=16, ef_construction=100, cosinesimil), mapping below
factory-content-000001, -000002, …	rollover by size (30 GB) or age (90 d), alias swap	reindex via alias: build next index → reindex (filter by indexedAt range) → atomic alias move; reconciliation sweep covers the gap (≤ 5 min dual-write window where both indices are readable, query targets the alias)

Why one index for all types: (1) cross-type search is the product (one box, "search everything"); (2) the type keyword filter is nearly free; (3) one kNN graph across all content gives better RAG recall than per-type shards; (4) one reconciliation sweep. Isolation is enforced by the workspace_id filter (below), not by index boundaries.

---

3. Index Schema (OpenSearch mapping)

{
  "mappings": {
    "properties": {
      "workspace_id":    { "type": "keyword" },        // tenant filter — ALWAYS present in every query
      "type":            { "type": "keyword" },        // page|block|database|record|comment|file|user
      "ref_id":          { "type": "keyword" },        // source row id (plain ref)
      "title": {
        "type": "text",
        "analyzer": "factory_title",                   // standard + lowercase + trim
        "fields": {
          "keyword": { "type": "keyword", "ignore_above": 256 },
          "prefix":  { "type": "text", "analyzer": "factory_prefix" }
        }
      },
      "body": {
        "type": "text",
        "analyzer": "factory_body",                    // standard + english stemming
        "fields": {
          "raw": { "type": "text", "index": false }    // for highlight/return
        }
      },
      "vector": {                                       // AI embeddings (§ai-arch)
        "type": "knn_vector",
        "dimension": 1024,                              // = current embed model output
        "method": { "name": "hnsw", "space_type": "cosinesimil",
                    "engine": "lucene",
                    "parameters": { "m": 16, "ef_construction": 100 } }
      },
      "vector_model": { "type": "keyword" },            // which embedding model produced `vector`
      "created_by":      { "type": "keyword" },         // user id
      "created_by_name": { "type": "text", "fields": { "keyword": { "type": "keyword" } } },
      "updated_by":      { "type": "keyword" },
      "created_at":      { "type": "date" },
      "updated_at":      { "type": "date" },
      "archived":        { "type": "boolean" },
      "team_id":         { "type": "keyword" },
      "parent_id":       { "type": "keyword" },         // for page/block: enables "in page X" filters
      "database_id":     { "type": "keyword" },         // records/blocks-in-databases
      "mime_type":       { "type": "keyword" },         // files
      "size_bytes":      { "type": "long" },            // files
      "permissions": {                                  // ← query-time permission filtering (§5)
        "public":    { "type": "boolean" },
        "workspace": { "type": "boolean" },             // all members of the workspace can see
        "teams":     { "type": "keyword" },             // team ids granted
        "users":     { "type": "keyword" },             // user ids granted
        "level":     { "type": "keyword" }              // VIEW|COMMENT|EDIT|FULL_ACCESS (highest grant)
      }
    }
  },
  "settings": {
    "index": { "knn": true, "knn.algo_param.ef_search": 100, "number_of_replicas": 1 },
    "analysis": {
      "analyzer": {
        "factory_title": {
          "tokenizer": "standard",
          "filter": ["lowercase", "trim", "english_possessive_stemmer"]
        },
        "factory_body": {
          "tokenizer": "standard",
          "filter": ["lowercase", "stop_english", "english_possessive_stomper", "english_stemmer"]
        },
        "factory_prefix": {
          "tokenizer": "standard",
          "filter": ["lowercase", "trim"]
        }
      },
      "normalizer": {
        "keyword_lowercase": { "type": "custom", "filter": ["lowercase"] }
      }
    }
  }
}

Design notes.

• vector is optional per doc: page/block/record docs carry embeddings (AI chunker); comment/file/user docs don't — kNN queries simply rank among embedded docs (missing vector = not a kNN candidate, which is the right semantics for "semantic search").
• permissions is a denormalized snapshot, not a live reference — see §5 for how it stays correct.
• No per-workspace indices: tenant isolation is by filter (and every query builder in packages/search requires workspace_id in the filter clause — it's a parameter, not an option).

---

4. Query Shapes

All queries are built by packages/search/src/queries.ts; the mandatory wrapper:

function baseFilter(wid: string, viewer: ViewerCtx) {
  return {
    bool: {
      filter: [
        { term: { workspace_id: wid } },                 // tenant — non-negotiable
        // permission filter — §5:
        permFilter(viewer),
        { bool: { must_not: [{ term: { archived: true } }], should: [{ term: { archived: false } }, { bool: { must_not: [{ term: { archived: true } }], boost: 0 } }] } }
      ]
    }
  };
}

4.1 Full-text (default mode)

{
  "size": 50,
  "query": {
    "bool": {
      "must": [
        { "multi_match": {
            "query": "quarterly revenue forecast",
            "fields": ["title^3", "body"],
            "type": "best_fields", "operator": "or",
            "fuzziness": "AUTO",            // 0 for len≤3, 1 for 4–6, 2 for >6
            "prefix_length": 1,             // first char must match exactly (keeps recall sane)
            "max_expansions": 50
        } }
      ],
      "filter": [ /* baseFilter */ ]
    }
  },
  "highlight": { "fields": { "title": {}, "body": { "number_of_fragments": 2, "fragment_size": 160 } } },
  "_source": ["type","ref_id","title","created_by_name","updated_at","team_id","database_id"],
  "sort": [ { "_score": "desc" }, { "updated_at": "desc" } ]
}

• title^3 weights titles 3× (matches user mental model: page titles rank first).
• fuzziness: AUTO gives free typo-tolerance for the common case without a fuzzy pass.

4.2 Prefix (autocomplete / "search as you type")

{
  "query": { "bool": {
    "must": [
      { "bool": { "should": [
        { "prefix": { "title.prefix": { "value": "proj", "boost": 2 } } },
        { "match":  { "title":        { "query": "proj", "operator": "and" } } }
      ], "minimum_should_match": 1 } },
      { "term": { "type": [ "page", "record" ] } }      // autocomplete favors navigable results
    ],
    "filter": [ /* baseFilter */ ]
  } }
}

Served by GET /v1/search/suggestions?q= — capped size: 8, no highlight, p95 < 100 ms (prefix terms are cheap; results cached 300 s per (wid, q) in Redis, invalidated by the indexer).

4.3 Explicit fuzzy (typo-heavy queries, "did you mean")

{ "fuzzy": { "title": { "value": "recieve", "fuzziness": 2, "prefix_length": 1 } } }

Used when the full-text pass returns < 3 results: the UI runs a fuzzy retry and labels results "close matches". A top-5 term-fuzzy "did you mean" suggestion (OpenSearch suggestions API, factory-content term index on title.keyword) is offered in the UI.

4.4 Faceted filter combinations (UI filter chips)

"filter": [
  { "terms": { "type": ["page", "record"] } },          // type chips
  { "terms": { "team_id": ["t_1"] } },                   // team chip
  { "range": { "updated_at": { "gte": "2025-01-01" } } }, // "updated after"
  { "term":  { "created_by": "u_7" } },                  // "created by me"
  { "term":  { "database_id": "db_3" } }                 // "in database"
]

Facet counts come from a companion aggs on type + team_id (top-20) in the same request (one round-trip).

4.5 Hybrid AI search (BM25 + kNN + RRF)

Full design in ai, §5–§6; the query shape:

{
  "knn": {
    "field": "vector",
    "query_vector": [ /* embed(query) via AI gateway */ ],
    "k": 50,
    "num_candidates": 200
  },
  "query": { /* same bool as §4.1, size 50 */ },
  "size": 20
}

OpenSearch fuses both (BM25 score and kNN cosine) — the app then applies reciprocal rank fusion on the returned rank lists for the display list, and the top chunks feed the LLM for the answer-with-citations path. RRF: score(d) = Σ 1/(60 + rank_i(d)) over the two ranked lists (k=60; standard RRF constant).

---

5. Permission Filtering at Query Time

Principle: the index stores a visibility snapshot per document; every query filter downs to what the viewer may see — before scoring, not after. Filtering after the fact (fetch 100, hide 90) would (a) leak existence via timing/pagination, and (b) waste the result window.

5.1 Grant snapshot (stamped at index time)

The indexer calls packages/permissions.grantSnapshot(resource):

// returns the closure of all grants that affect this resource:
interface GrantSnapshot {
  public: boolean;       // a Share with scope PUBLIC exists (and allowSearchIndexing)
  workspace: boolean;    // workspace-visible (default for non-private workspace pages:
                         //   members with role floor ≥ VIEW see it)
  teams: string[];       // team ids with an explicit grant (or team-ownership: pages
                         //   whose page.teamId is set and not isPrivate)
  users: string[];       // user ids with an explicit grant (granteeType=user, level ≥ VIEW)
  level: 'VIEW' | 'COMMENT' | 'EDIT' | 'FULL_ACCESS';  // highest level in the closure
}

Computation: the same evaluation walk the guard uses (resource → parent pages → workspace), but from every grantee's perspective, producing the union:

• workspace: true when the resource's role-floor grants VIEW to the workspace role (i.e. not isPrivate and no explicit NONE from a broader scope).
• users[]/teams[] from explicit permissions rows (level ≥ VIEW).
• public: true only via an active, unexpired Share(scope=PUBLIC).

The snapshot is written into search_documents.permissions (audit) and the OpenSearch doc's permissions fields (queryable).

5.2 The query-time filter

function permFilter(v: { userId: string; teamIds: string[]; roleFloor: boolean }) {
  return {
    bool: {
      should: [
        { term:  { 'permissions.public': true } },                    // public share
        { term:  { 'permissions.workspace': true } },                 // workspace member (any role ≥ GUEST sees workspace-visible)
        { terms: { 'permissions.teams': v.teamIds } },                // my teams
        { term:  { 'permissions.users': v.userId } }                  // explicit to me
      ],
      minimum_should_match: 1
    }
  };
}

• The guard has already established v (viewer is a workspace member; their team list + role come from the cached membership, not per-doc work).
• A doc is visible iff at least one of: public, workspace-visible, team grant, user grant. This exactly matches the guard's "max along the chain, no NONE-veto applies because the snapshot only contains VIEW-or-better grants" semantics — packages/permissions has one implementation of the walk, tested with the property test "search visibility == guard visibility" (for 200 random grant topologies: evaluate(viewer, resource) ≥ VIEW ⟺ doc passes permFilter — this is the Phase 8 exit-criterion integration test).
• Staleness window: snapshots refresh on (a) any source change event (re-stamp on upsert), (b) permission.changed events (targeted re-stamp, §1.1), (c) the 5-min reconciliation sweep (full re-stamp for docs whose grant closure changed — detected by cheap set-diff against a permissions_version watermark per workspace). Worst case a permission change is visible to search ≤ 5 min; the guard is never stale, so search can at worst over-surface a doc for < 5 min, and opening it still 403s correctly. (Under-surfacing — hiding something the user may see — is impossible: grant additions re-stamp via permission.changed immediately.)
• Record docs inherit the record page's snapshot (record = page, ADR-006 — one walk). Block docs inherit their page's snapshot (blocks are never separately shared).

---

6. Query API & Result Shape

POST /v1/search (SearchQuery schema in @factory/shared):

interface SearchQuery {
  q: string;                        // 1–200 chars
  mode?: 'fulltext' | 'hybrid';     // default fulltext; hybrid enables kNN+RRF (+answer if ask=true)
  types?: ('page'|'block'|'database'|'record'|'comment'|'file'|'user')[];
  filters?: {
    teams?: string[];
    createdBy?: string;
    updatedAfter?: string;          // ISO date
    databases?: string[];
    archived?: boolean;
  };
  page?: string;                    // cursor (opaque, = search_after on _score+updated_at+id)
  pageSize?: number;                // 1–100, default 25
  ask?: boolean;                    // hybrid only: append an LLM answer (RAG)
  conversationId?: string;          // optional: keep ask context across follow-ups
}

Response:

{
  "data": {
    "results": [
      { "type": "page", "id": "pg_…", "title": "Q3 Revenue Forecast",
        "snippet": "…<em>revenue</em> targets for <em>Q3</em>…",
        "url": "/acme/p/pg_…",
        "icon": "📊", "createdByName": "Ana", "updatedAt": "…",
        "score": 12.4, "source": "bm25+vec" }
    ],
    "facets": { "types": [{"type":"page","count":142}, …], "teams": […] },
    "cursor": "eyJzY29yZSI6MTIuNH0=",
    "total": 347,
    "answer": {                     // only when ask=true
      "text": "The Q3 forecast targets $4.2M ARR…",
      "citations": [
        { "docId": "page:pg_…", "title": "Q3 Revenue Forecast", "url": "/acme/p/pg_…",
          "quote": "revenue targets for Q3 are…" , "relevance": 0.91 },
        { "docId": "record:rec_…", "title": "Forecast Review — Jul 12", "url": "…", "quote": "…" }
      ],
      "model": "gpt-4o", "tokensIn": 4120, "tokensOut": 210, "latencyMs": 2380
    },
    "latencyMs": 212
  }
}

• total is track_total_hits: 10000 (UI shows "347" up to 10k, then "10k+").
• Latency budget: fulltext p95 < 500 ms (Phase 8 SLO) — measured at OpenSearch took + network; hybrid adds embedding (~150 ms, cached 600 s per query text in Redis) + kNN (ef_search 100 ≈ 30–80 ms at 10M docs) → hybrid p95 < 900 ms; the ask answer is streamed separately (SSE answer.token events after the results arrive) so results render in < 900 ms and the answer streams behind them (time-to-first-token < 1.5 s, full answer < 6 s p95).

---

7. Reconciliation & Repair

7.1 The sweep (every 5 min, continuous, per workspace in round-robin)

per workspace (staggered by hash(wid) % 300s):
  1. watermark = sweep_state(wid).lastWatermark
  2. candidates =
       source rows with updated_at > watermark          (pages, records, databases,
       UNION files with status change, comments created>watermark, users updated)
       UNION search_documents WHERE indexed_at < now() - 30 min   (stale docs)
  3. for each candidate: compute contentHash(title,body,permissions,metadata)
       hash == mirror hash  → skip
       hash != mirror hash  → emit synthetic 'reindex' event (normal consumer path)
       doc missing in OpenSearch (byId check, batched) → emit 'reindex' event
  4. deletes: mirror rows whose source row is gone (trashed+purged / deleted)
       → emit synthetic delete event
  5. advance watermark; persist sweep_state; emit metrics
       search_repair_fixed_total{kind=stale|missing|orphan}

The sweep is the self-healing layer: any missed event (crash between commit and XADD, consumer poison-drop, manual DB edit) converges within ≤ 5 min + 1 s refresh. Cost: bounded by updated_at delta (typical workspace: dozens of rows) — the sweep is cheap by construction, not by luck.

7.2 Manual repair

• POST /v1/admin/search/reindex (workspace ADMIN): {scope: "workspace" | {type, refId}[], force: bool} → enqueues reindex events for the scope (no watermark change) — used after the drift table shows damage, after bulk imports, and after a vector-model change (force re-embed).
• GET /v1/admin/search/drift (workspace ADMIN): top-50 drifted docs (source hash vs index hash) — the operator triage view.
• Full reindex: build a new factory-content-v{n+1}, replay from Postgres (the mirror table drives it — no source re-projection bugs), alias swap, delete old. Runbook in infrastructure/ (Phase 15).

7.3 Vector-model migration

Embedding models change (vector_model is per-doc): new model → new index (different dimension) → background re-embed job (ai-index consumer, priority queue, 50 docs/s) → alias swap → old index retained 7 days. During migration, kNN runs against the old model's docs (alias still on old index) — search quality is stable, no mixed vectors ever compared (cosine across dimensions is nonsense; the guard is structural: vector_model term filter on the query side during the transition, removed at swap).

---

8. Operations & Metrics

Metric	Alert
search_lag_seconds (p50/p95/p99, sampled §1.4)	p95 > 30 s for 10 min
search_indexer_pending (XPENDING per group)	> 10k (consumer stuck)
search_indexer_errors_total	rate > 1/min
search_query_duration_seconds (per mode)	p95 > 500 ms (fulltext) / > 900 ms (hybrid)
search_result_empty_rate (queries returning 0)	> 40% over 1 h (index broken or filter bug)
os_cluster_status / os_disk_used_percent	yellow/red; disk > 85%
search_repair_fixed_total	> 100/min sustained (event path is losing events)
search_perm_stamp_total (permission re-stamps)	informational (spike = permission storm)

Capacity: 10M docs ≈ 40 GB index (title+body 1 KB avg avg + 1024-dim float32 vector ≈ 4 KB/doc where embedded ≈ 2/3 of docs) → 30 GB rollover every ~7M docs; cluster sized at 3 data nodes × 100 GB for the 10M-doc platform target (Phase 15 load test: 10M docs, 100 qps mixed fulltext/hybrid, p95 targets hold).

---

9. Hybrid foundation (TASK-0099)

The kNN leg of §4.5 was designed but not built; TASK-0099 lands the foundation (chunk docs, weighted fusion, embedding stub, alias-rollover runbook) without wiring it into POST /search (that wiring is Phase 12, behind a capability flag). Recorded decisions below; the spec-vs-reality divergences are listed, not silently resolved.

9.1 Chunk doc model

A long page or record body is additionally indexed as one doc per chunk. Chunk docs live in the SAME factory-content index, alongside whole docs:

{
  "docId": "page:pg_1#c2",     // <parent docId>#c<n>  — RECONCILED: spec's
  "type": "chunk",             //   <type>:<refId>:chunk:<chunkId> form was
  "id": "pg_1",                //   rejected (id parsing + mirror keying)
  "chunkId": "c2",             // the chunk's own stable id
  "chunkIndex": 2,             // 0-based
  "parentDocId": "page:pg_1",
  "body": "…chunk text…",
  "workspaceId": "…",          // + the parent's full grant snapshot, url,
  "grants": { "…same as parent…" }, // title, createdAt/updatedAt — chunk docs
  "archived": false            // are tenant-scoped + permission-filtered exactly
                               // like their parent
}

• Chunking (packages/search/src/chunking.ts, char-based, DEFAULT_CHUNK_OPTIONS maxChars 800 / overlap 120, paragraph + code-fence aware). RECONCILED: the spec said 800 tokens / 15% overlap; char-based 800/120 was kept (no tokenizer dependency; a later Phase-12 tokenizer can be swapped behind the same ChunkOptions interface).
• Threshold (worker expandChunkDocs): a whole doc is expanded only when body.length > CHUNK_BODY_THRESHOLD (1600 = 2 × maxChars) and the chunker yields ≥ 2 chunks. The spec is silent on a threshold; 1600 keeps small bodies whole-doc only (chunking a 1-chunk body would just duplicate the doc).
• Type scope: page + record only in Phase 8. comment/file/database/block chunking is a Phase-12 loose end.
• No mirror rows: chunk docs are derived from their parent, so they do NOT get a search_documents row (the mirror is whole-doc-only, keyed [type, refId]). Deletion is therefore parent-driven: SearchClient.deleteChunksOf(parentDocId(s)) runs a _delete_by_query?conflicts=proceed on parentDocId + type:"chunk" (the chunk count is unknown to the mirror, so per-doc _delete ids can't be enumerated).
• Stale-chunk invariant (worker indexDocs): before writing, the chunks of every page/record parent present in the batch are cleared first (shrink/grow stays exact). A single-doc repair that passes only one chunk does not clear, so it can never clobber a surviving sibling.

9.2 Keyword search stays whole-doc (the must_not exclusion)

Chunk docs must never surface from keyword search (keyword hits are deduped to the parent by displayType, and a raw chunk hit would be a fragment, not the doc).

RECONCILED (and load-bearing): excluding chunks via the public types filter is insufficient — the live API omits the types clause entirely when the caller sends no types filter, so the exclusion must be structural. Every keyword query body (full-text, prefix, fuzzy, and the hybrid BM25 leg) therefore carries a mandatory bool.must_not: [{ term: { type: "chunk" } }] (queries.ts → chunkExclusion()). The kNN bodies carry NO exclusion — chunk docs are the kNN leg's target. Whole-doc keyword recall is unaffected: a chunk's text is also contained in the parent's whole body.

9.3 Weighted reciprocal-rank fusion (packages/search/src/hybrid.ts)

buildHybridFusion(keywordHits, knnHits, weights, k = RRF_K, limit) fuses the two rank lists into one, RRF-weighted per leg:

score(d) = w_kw · Σ_kw 1/(k + rank_kw(d))  +  w_vec · Σ_knn 1/(k + rank_knn(d))

HybridWeights {kw, vec} is normalized (divides by kw+vec), defaults 0.5/0.5 (DEFAULT_HYBRID_WEIGHTS). Contract property (unit-tested): at 0.5/0.5 every fused score is exactly ½ of fuseHybrid's (RRF, k=60) score for the same doc, in the same order — so the weighted blender is a strict generalization of the plain RRF and existing RRF fixtures are preserved. The spec's query-builder.ts/ buildHybridQuery was not created: queries.ts already had buildHybridBody (the OpenSearch knn + bool hybrid DSL) and buildHybridKnnOnlyBody, and rrf.ts the plain fusion; the missing piece was the weighted fusion, which is what hybrid.ts adds.

9.4 embed:chunks stub job (queue ai)

apps/worker/src/processors/ai-embedding.ts (flat file, RECONCILED from the spec's ai/embedding-pipeline.ts — matches the repo's flat processor layout). Job: { type:"embed:chunks", workspaceId, docType, refId, chunkIds[] }, dispatched from ai.processor.ts before the AI_API_KEY gate (its skip gate is its own):

• provider unset (AI_EMBEDDING_PROVIDER empty, the default): logs embedding disabled (Phase 12), completes, no AiEmbedding row, no index write — the chunk docs' vector stays null and the kNN leg has nothing to match.
• provider set: writes one placeholder AiEmbedding row per chunk (chunkRef = page:pg_1#c2 = the chunk's content doc id, model, dimension = SEARCH_VECTOR_DIMENSION, metadata {stub:true, docType, refId}), without any provider or index call. Phase 12 replaces the placeholder with the real gateway call + vector merge.
• RECONCILED AiEmbedding shape: spec's (workspaceId, docType, refId, chunkId, model, vector) → the landed model (workspaceId, chunkRef, chunkText, vectorId, model, dimension, metadata), unique [workspaceId, chunkRef]. No migration (the model pre-exists; chunkRef carries <docType>:<refId>#<chunkId>, so docType/refId are recoverable from the row key).

Config additions (worker config.ts zod + .env.example): AI_EMBEDDING_PROVIDER (default ""), AI_EMBEDDING_MODEL (default text-embedding-3-small), plus SEARCH_VECTOR_DIMENSION / SEARCH_KNN_M / SEARCH_KNN_EF_CONSTRUCTION surfaced in .env.example.

9.5 v1 → v2 alias rollover (scripts/opensearch-reindex.ts)

dense_vector dimensions are immutable per index, so a model/dimension change (or the v1 → v2 mapping change) must roll a new index forward through the alias — never re-map in place. The runbook:

pnpm search:reindex --i-know-this-swaps-the-alias [--to v2] [--dimension 1536]
                    [--dry-run] [--strict-equal] [--cleanup-v1]

• Guard: --i-know-this-swaps-the-alias is required (a bare --dry-run is read-only) — this command re-points the live factory-content alias.
• Steps: resolve current alias target → pick target index (--to, else current+1) → create it with the §3 mapping (dim/m/efConstruction from --dimension / SEARCH_KNN_M / SEARCH_KNN_EF_CONSTRUCTION) → backfill from the search_documents mirror in 500-doc _bulk batches (cursor-paged, idempotent re-runs) → _refresh + count-equality vs the mirror (--strict-equal hard-fails a mismatch, else warns) → atomic POST /_aliases swap (one call: remove old + add new; never remove-then-add) → optional --cleanup-v1 delete.
• Self-contained (no new deps): the root has no @factory/* hoist, so the mapping is re-declared in the script (mirror of packages/search/src/mapping.ts — keep the two in sync) and OpenSearch is driven with raw fetch; Postgres uses the repo's existing @prisma/client.
• Backfill caveat (Phase-12 loose end): the mirror holds whole docs only, so a freshly reindexed index contains whole docs; chunk docs reappear as the worker re-indexes pages/records (or on the next content event). A source-table rebuild (url/grants/createdByName straight from Postgres) would close this gap.

9.6 Phase 12 wiring checklist (what this task intentionally left out)

• [ ] Real embedding provider call in embed:chunks (+ vector merge onto the chunk doc via the existing search.updateDoc(docId, { vector }) path).
• [ ] Hybrid query path behind a capability flag on POST /search (AI_EMBEDDING_PROVIDER set ⇒ hybrid is offered), wiring buildHybridBody + buildHybridFusion (this task ships the fusion; not the endpoint).
• [ ] Reindex backfill from the SOURCE tables (url/grants/createdByName) instead of the mirror; chunk re-embed after rollover.
• [ ] Real AiEmbedding vectors (replace the placeholder rows).
• [ ] Chunking for comment/file/database/block (page + record only for now).
• [ ] Token-based chunking (swap the char-based chunkText behind ChunkOptions).

10. Performance (TASK-0100 — measured, 2026-09-29)

Measured with the k6 suite (tests/performance/search-search.k6.js) over the 100,000-doc fixture (seeded by tests/performance/fixtures/search-100k.ts). The environment-fingerprinted report the suite writes is tests/performance/reports/search-last.json (kept untracked; committed on demand by a maintainer, never per-PR — CI uploads it as an artifact).

10.1 The 100k fixture (deterministic)

type	count	share
page	41,937	~42%
record	26,211	~26%
file	15,726	~16%
comment	10,484	~10%
database	5,242	~5%
user	400	0.4%

Body sizes 200–2000 chars (deterministic PRNG, same corpus on every run), 2 workspaces (perf-search-a/perf-search-b), 200 users, permission mix workspace/team/private. Fixture docs carry deterministic ids + a marker the seeder/reset uses for idempotent cleanup; they live in dedicated perf-* workspaces so normal test DBs/indexes stay clean.

10.2 SLA definitions (and where each is gated)

SLA	Target	Where enforced
Query latency, full profile	p95 < 500 ms, p99 < 1 s, error < 0.1% (100 VUs / 5 min / 50 req/s @ 100k docs)	k6 options.thresholds (DoD defaults) — nightly CI run; a breach fails the lane
Query latency, reduced profile	p95 < 1000 ms, p99 < 2000 ms, error < 0.1% (10 VUs / 60 s / 10 req/s — 2×-flake headroom)	k6 options.thresholds — merge-to-main CI gate
Index lag (create/delete → searchable/unsearchable)	< 30 s, both sides	Playwright e2e tests/e2e/specs/search-index-lag.spec.ts (hard 30 s deadline per poll loop)
1k-event burst drain	backlog 0 within 30 s	same e2e (stream-side burst, search-index:lag metric)

The reduced profile is a smoke/gate profile: its thresholds are 2× the DoD values so a single noisy CI runner does not fail a merge, while a genuine regression (order-of-magnitude) still trips it.

10.3 Measured numbers (reduced profile, 2026-09-29)

Environment: dev Docker stack — OpenSearch 2.15.0 (single node, default JVM heap 512 m, default settings), PostgreSQL 16, Redis 7, NestJS API under tsx, k6 v0.57.0, machine 4 vCPU / 15 GB (smaller than the DoD's 8 vCPU / 16 GB reference environment — see the full-profile note below).

Reduced profile (10 VUs / 60 s / 10 req/s) over 100,000 fixture docs:

metric	p50	p95	p99
client latency (ms)	22.7	32.2	41.9
server tookMs (ms)	17	25	33

Error rate 0. The S6 baseline run on the same machine measured p50 28.5 / p95 44.9 / p99 51 ms (run-to-run spread; both far under the 1000 ms reduced threshold and the 500 ms DoD target).

Full profile (100 VUs / 5 min / 50 req/s — the DoD shape), same environment: p50 24.8 ms / p95 49.9 ms / p99 92.8 ms (server tookMs p50 18 / p95 38 / p99 69), error rate 0, docCount 100,000 — a ~10× margin under the 500 ms / 1 s DoD thresholds, measured on a machine smaller than the DoD's 8 vCPU / 16 GB reference environment (4 vCPU / 15 GB), so the reference-hardware acceptance is implied rather than separately run.

10.4 OpenSearch tuning notes (recorded per the task spec)

• index.refresh_interval stays at the 1 s default — no tuning applied. The 30 s lag SLA is met at 1 s (measured e2e: create-side lag ~4.2 s, delete-side ~69 ms, burst drain 36 ms). The task spec explicitly forbids relaxing it to 30 s "for the lag SLA" — doing so would trade away live visibility for a number the system already hits by a wide margin.
• No JVM heap or thread_pool tuning required to hit p95 at 100k docs. The single-node dev container runs the default 512 m heap (OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m) and default thread_pool settings; cluster settings were left at defaults (_cluster/settings shows no search-related overrides). If a future corpus grows toward the 10 M-doc NFR-S5 size, the first dials to touch are the heap (1–2 GB) and thread_pool.search.size — but that is out of Phase-8 scope and unmeasured.
• No scroll_size or other micro-tuning was needed (the task spec allowed it only "if measured needed").

10.5 Running the suite

All from tests/performance/ (k6 binary v0.57.0 on PATH — CI installs it from GitHub releases; locally: any recent v0.57.x).

pnpm perf:seed:search    # idempotent: reset first, seed exactly 100,000 (verifies)
pnpm perf:reset:search   # drop the perf fixtures (OS + DB rows)
pnpm check:search        # k6 archive parse gate (catches syntax only)
pnpm search:100k         # FULL profile: 100 VUs / 5 min / 50 req/s, DoD thresholds
pnpm search:100k:reduced # REDUCED: 10 VUs / 60 s / 10 req/s, 2×-flake thresholds

Env knobs: K6_API_URL, K6_JWT_SECRET (must equal the API's JWT_SECRET — the suite mints real HS256 tokens for the fixture users), K6_SEARCH_WORKSPACES (default perf-search-a,perf-search-b), K6_VUS / K6_DURATION_SECONDS / K6_RATE, K6_P95_MS / K6_P99_MS / K6_ERROR_RATE, K6_VERSION (fingerprint label, e.g. ci). The API must be started with RATE_LIMIT_API raised (default 100 req/min/user; load runs use 100000) — see the CI perf-search job for the canonical invocation.

Reports: k6 writes reports/search-{full,reduced}.json + search-summary.json into the package dir (k6 CWD); handleSummary additionally writes the fingerprinted reports/search-last.json (p50/p95/p99, server tookMs, error rate, doc count, OpenSearch version, profile + thresholds).

CI: the perf-search job in .github/workflows/ci.yml is self-contained (postgres/redis/opensearch services, pinned k6, API under tsx) — reduced profile on push to main (the merge gate), full profile on the nightly schedule + workflow_dispatch. See docs/development/performance.md for how to read the reports and the regression-gating intent.