realtime-architecture.mddocs/architecture/realtime-architecture.md← architecture33,529 B← Back to the app

Realtime Architecture (`apps/realtime`)

Node 20 + ws + Yjs. Sibling docs: system · frontend · database · security.

Position of Yjs in the truth hierarchy (ADR-003): Yjs is the source of truth while a page/database is open for editing; the persisted blocks table is the source of truth at rest. This document defines the wire protocol, the document model, persistence, presence, offline behavior, the latency budget, scaling, and failure modes.

1. Process & Room Model

apps/realtime (one node type, N replicas)
├── ws server            port 4001, wss://host/realtime (nginx upgrades /realtime → :4001)
├── RoomManager
│   ├── rooms: Map<roomKey, Room>            # in-memory, per node
│   ├── roomKey = sha256(workspaceId + ':' + resourceType + ':' + resourceId) % 32bit
│   └── ownership: node owns the rooms whose hash % instanceCount == nodeIndex
│        (consistent-hash ring; edge routes by room key, §6)
├── Room (one per open page/database)
│   ├── ydoc: Y.Doc                          # the editable state (blocks + meta)
│   ├── awarenessProvider: Yjs awareness     # presence (ephemeral)
│   ├── peers: Map<connId, Peer>             # sockets in this room
│   ├── persistTimer                         # debounced flush state
│   └── updateLog: RingBuffer<YjsUpdate>     # resume log, until acked-persisted
├── Hydrator            # Postgres → Y.Doc (blocks table → codec)
├── Persister           # Y.Doc → blocks table (debounced, batched upserts)
├── EventBus link       # 'page.updated' / 'block.updated' on persist; 'comment.created' push
└── AuthBridge          # verifies JWTs (packages/auth, shared secret) + permission checks

Room resources:

Room key suffix	Resource	What syncs
`page:{pageId}`	a page (or record page)	block tree (Yjs doc), presence, cursors
`database:{databaseId}`	a database (home view)	property schema + view config + record position/order ops (the record values are still REST — see §4.4)
`comment:{pageId}`	a page's comment panel	comment thread state (lightweight, non-CRDT message fan-out)

A page room and its database home room coexist when a DATABASE block is open: the database block inside a page renders the database's view, and edits to rows are REST mutations broadcast to both rooms' subscribers (the row grid is not CRDT-edited; the block tree that embeds it is).

1.1 Why one authoritative doc per resource

Yjs docs are per-connection merged, but ordering and persistence need a single writer per resource: the room's doc receives every update, so (1) persistence is serialized (one debounced flusher), (2) the updateLog for resume is consistent, and (3) two nodes never both persist the same page (which would double-emit page.updated). Room sharding (§6) guarantees exactly one owning node per room.

2. WebSocket Protocol

2.1 Connection & auth handshake

Client                                            Server
  │  GET /realtime?token=<JWT>&room=page:abc
  │      (Upgrade: websocket, Origin, Sec-WebSocket-Protocol: factory.v1,
  │       X-Realtime-Room: page:abc)                        │
  ├────────────────────────────────────────────────────────▶│
  │                          1. rate limit: 30 conns/min/IP, 100 conns/user
  │                          2. verify JWT (access token, 15 min TTL, HS256,
  │                             secret shared with api via env; issuer=factory)
  │                             → { userId, workspaceId claim }
  │                          3. permission check (packages/permissions, same engine
  │                             as the API guard):
  │                             page room  → level ≥ VIEW to join; EDIT to send doc updates
  │                             database   → same
  │                             (VIEW-only peers receive doc updates, cannot send)
  │                          4. join room (hydrate if needed — see §4.2)
  │◀────────────────────────────────────────────────────────┤
  │  101 Switching Protocols (Sec-WebSocket-Protocol: factory.v1, room:page:abc)
  │
  │◀── {"t":"hello","roomId":"page:abc","self":{"userId":"u_1","color":"#e91e63"},
  │     "docStateVector":base64,…,"awareness":[…existing peers…]}
  │
  │── {"t":"sync.request","stateVector":base64}        # client's Yjs state vector
  │◀── {"t":"sync.state","updates":[base64…]}          # doc delta to reach server state
  │     (+ full doc if the client is new: updates = encoded doc, size < 2 MB,
  │      else paged "sync.state" messages with `seq`)
  │
  │── {"t":"sync.ack"}                                 # client applied; room now active

Rules:

Token in the query string is acceptable at connect time only (standard for WS — there is no Authorization header in the upgrade from all browsers). The token is read from the upgrade request, never logged, and the connection is bound to the resulting userId; it is not stored beyond the connection lifetime.
A JWT that is valid but expired-by-< 5 min is accepted (tab left open across the 15 min boundary); the client refreshes via the BFF route and sends {"t":"auth.refresh","token":…} (server re-verifies, keeps the socket).
Rejected auth → close with code 4401 (unauthorized) or 4403 (forbidden); the client surfaces "You can't view this page" rather than a raw socket error.
Max connections per user: 100 (browsers open one per open tab/room; the cap catches leaks). A user can be in any number of rooms.

2.2 Message types (wire format: single-line JSON, Yjs updates as base64)

`t`	Dir	Schema	Purpose
`hello`	S→C	`{roomId, self:{userId,color}, awareness[]}`	post-upgrade greeting; server-side presence seed
`sync.request`	C→S	`{stateVector: base64}`	"give me what I'm missing" (Yjs state-vector protocol)
`sync.state`	S→C	`{updates: base64[], seq?: int}`	delta (or paged full doc)
`sync.ack`	C→S	`{}`	client applied; client is now in the doc
`update`	C→S	`{doc: 'blocks'\|'meta', update: base64}`	Yjs update from the client (only EDIT-capable peers)
`update.ack`	S→C	`{clock: number}`	server applied + broadcast; `clock` = server doc's Yjs clock (used by the offline queue)
`presence.set`	both	`{state: AwarenessPayload}`	full awareness state for self (join, field change)
`presence.patch`	both	`{state: AwarenessPayload, keys: string[]}`	partial awareness update (cursor moved — the hot path)
`presence.drop`	S→C	`{userId}`	peer left / pruned
`cursor.move`	C→S	`{blockId, offset, selection?: {fromBlockId,fromOffset,toBlockId,toOffset}}`	sugar over awareness for the cursor (debounced 50 ms on client)
`comments.event`	S→C	`{kind: 'created'\|'replied'\|'resolved'\|'reaction', payload}`	realtime comment fan-out (REST writes, socket pushes)
`notification.push`	S→C	`{notification: NotificationDto}`	in-app notification without a page open (user-level socket `room=user:{userId}`)
`record.updated`	S→C	`{databaseId, recordIds[], changedProperties[]}`	row grid refresh hint (REST mutations broadcast to database room)
`doc.persisted`	S→C	`{pageId, version}`	persist flush completed (client can clear "saving" UI early)
`ping` / `pong`	C↔S	`{}`	keepalive: server pings every 25 s; client must pong within 25 s (then close `4000`)
`error`	S→C	`{code: 'RATE_LIMITED'\|'UNAUTHORIZED'\|'FORBIDDEN'\|'TOO_LARGE'\|'INTERNAL', message}`	non-fatal; connection continues unless `code` is auth-related
`bye`	S→C	`{code, message}`	pre-close notice (server drain, room moved) — code `4001` "reconnect, room may have moved"

Yjs-specific messages use the standard Yjs sync protocol (state vectors + update messages) wrapped in our envelope; the base64 payloads are raw Y.encodeStateAsUpdate buffers. Max single-message size: 256 KB (a 256 KB update ≈ a few thousand block ops); clients chunk larger local edits (the offline queue chunks at 128 KB), and the server closes 4008 on oversize after an error TOO_LARGE.

2.3 The per-user control socket

Independent of page rooms, each authenticated user may hold one control socket (room=user:{userId}) that receives notification.push, comments.event (for pages the user has open but isn't editing), and record.updated for databases they've subscribed to ({"t":"subscribe","room":"database:abc"} over the control socket). The browser opens the control socket at app start (not per page), so notifications arrive with zero additional connections.

3. Connection Lifecycle

connect ─▶ auth ─▶ join room ─▶ hydrate (if cold) ─▶ sync handshake ─▶ ACTIVE
  │
  │  heartbeat: server ping 25 s / client pong 25 s
  │  awareness: peer joined → presence.set broadcast (≤ 1 frame per 100 ms per peer)
  │  updates:   update → apply → broadcast (fan-out, coalesced per event-loop tick)
  │  persist:   debounced flush (1 s quiet / 10 s max dirty) → 'doc.persisted'
  │
  ├── disconnect (clean): presence.drop, room may persist-final, release slot
  ├── idle: room with 0 peers → persist-final → doc evicted after 5 min LRU
  │         (updateLog cleared after successful persist)
  ├── socket dropped (network): client auto-reconnects (exponential backoff
  │   1 s → 2 s → 4 s … 30 s cap, full jitter); on reconnect it re-runs the sync
  │   handshake with its state vector — the server answers from the in-memory doc
  │   (hot room) or the updateLog (warm room) or a fresh hydrate (cold room)
  └── server drain (SIGTERM / room move): send `bye {code 4001}` → stop accepting
        updates → persist-final → close 1001. Client reconnects; room hash may now
        map to a different node → edge routes there (§6.2).

Presence pruning: the server heartbeats awareness (Yjs awareness does this via updateTimeout = 30 s). A peer with no presence.patch/ping for 60 s is pruned: its cursor disappears, presence.drop is broadcast. Pruning is per-room; the user's other rooms are unaffected.

4. Yjs Integration

4.1 Document model

One Y.Doc per room, two top-level types (see frontend, §3.4 for the codec that bridges to Block JSON):

Y.Doc (page room)
├── yXml   Y.XmlFragment   # ProseMirror document via y-prosemirror
│     # every block = one item; text = y.Text; attrs: {id, type, position}
└── yMeta  Y.Map           # keyed by block id:
      # {content: Json, properties: Json, position: string}
      # position lives in yMeta (NOT yXml order) so that:
      #   • reordering is a meta update (one small CRDT op, no tree rewrite)
      #   • persist ordering is deterministic (sort by position, not array order)

Y.Doc (database room)
├── ySchema  Y.Map         # property list + order (position), view list + config
│     # property add/rename/reorder = CRDT-merged (two users adding columns don't lose one)
└── yRows    Y.Map         # recordId → {position}
      # row creation/deletion/reorder = CRDT; property VALUES remain REST (below)

Awareness (yAwareness) is per-room and ephemeral — never persisted, never sent to Postgres; it survives only for the room's lifetime in memory.

4.2 Server hydration (Postgres → Y.Doc)

Cold room (first peer after eviction, or node restart):

1. SELECT id, parent_id, type, position, content, properties
   FROM blocks WHERE page_id = ? AND deleted_at IS NULL
   ORDER BY position                       -- one range-indexed scan
2. codec.decode(blocks) → yXml + yMeta     -- deterministic: sort by position,
                                           --   parentId from the tree
3. if the doc is a record page: nothing extra (properties are record_properties,
   fetched over REST by the client, not part of the ydoc)
4. room.ydoc = hydrated doc; log hydrate_ms metric (target < 200 ms for 10k blocks)

Hydration cost is bounded by block count: 10k blocks ≈ 8 MB scan ≈ 30–80 ms on a warm Postgres; 100k blocks ≈ 600 ms (acceptable one-time cost, hidden behind the "loading page" shell — the frontend shows metadata immediately and the doc streams in via the sync handshake's paged sync.state).

Hydrate-on-write race: a peer could send update during hydration. The room buffers incoming updates until hydration completes (max 5 s), then applies them to the hydrated doc — Yjs merge is commutative, so order doesn't matter, but applying to a half-built doc would corrupt the Y.XmlFragment structure.

4.3 Update persistence (debounced)

update arrives (per room, serialized on the room's event loop)
  1. doc.transact(apply update, 'remote')
  2. if !peer.canEdit → drop + error FORBIDDEN (defense in depth: guard already checked)
  3. broadcast to other peers (coalesced: all updates within one event-loop tick
     are encoded once per peer — Yjs merges before encode)
  4. mark room dirty; updateLog.push(update)   # until persisted
  5. persistTimer: clearTimeout; setTimeout(flush, 1000)
     hard cap: if dirty > 10 s → flush immediately

flush(pageId, actorSummary):
  1. blocks = codec.encode(doc)                  -- walk yXml + yMeta, position-ordered
  2. roomTouched = diffBlockSets(lastPersistedBlocks, blocks).ids()
     # captured PRE-rebase: the blocks THIS room edited (the flush wins on these)
  3. CAS persist against pages.version (baseVersion = room.lastVersion):
       UPDATE pages SET version = baseVersion + 1
         WHERE id = pageId AND version = baseVersion        -- count must be 1
       (plus the block batch upsert + soft-delete in the same tx)
     on CONFLICT (an external REST write bumped pages.version):
       -- the 3-way rebase (normative, TASK-0092): a concurrent REST writer raced us
       BASE   = room.lastPersistedBlocks          # what we last persisted
       OURS   = blocks                            # the CRDT doc (this flush's state)
       THEIRS = the store's current blocks        # what the REST writer just wrote
       externalOps = diffBlockSets(externalBase, THEIRS)   # the REST writer's changes
       absorb into the doc ONLY externalOps whose block.id ∉ roomTouched
         (a block both sides edited → the ROOM's (OURS) version wins; the REST op is
          dropped for that block, but its other blocks are kept)
       apply the absorbed ops to the doc (Y.transact origin 'realtime-flush-rebase')
       externalBase = THEIRS; retry the CAS persist (budgeted: 2 attempts)
       a SECOND conflict (a writer landed during the retry) → the doc is already in
         sync with the latest external state; advance room.lastVersion to it, keep the
         room DIRTY, and re-arm persistTimer (converges on the next flush — no clobber)
  4. room.lastVersion = the new version
     room.lastPersistedBlocks = the persisted blocks; room.updateLog.clear()
  5. EventBus.publish('page.updated', {pageId, version, blockDelta, ops})
       -- blockDelta = {inserted, updated, moved, deleted} over the diff (R3
       --    minimality: the ops are the minimal op set — unchanged blocks emit no op)
       -- actor: 'realtime-flush' (auditable, not impersonable — the public API has
       --    no path to set that actor)
       → search indexer, rollup recompute (if a database block changed), audit-flush
  6. broadcast doc.persisted {pageId, version}

Flush-rebase merge mechanism (normative, TASK-0092 R2): persistence is a CAS on pages.version, and a conflict is a merge, not a drop. The 3-way merge is (BASE = last persisted, OURS = the CRDT doc, THEIRS = the REST writer's blocks); flush wins on any block the room itself edited in this flush (roomTouched, captured before the rebase so a post-rebase encode can't accidentally mark an absorbed external op as ours), and the REST writer's changes to other blocks are absorbed into the doc (so a rejoiner and the next flush see the union — both writers survive, no clobber). The retry is budgeted (2 persist attempts) and a genuine second conflict converges on the next flush rather than clobbering. This is the reverse direction of the same CAS the REST batch endpoint uses: REST sees fresh state on retry (CONFLICT carries the new version, backend, §12) and the flush absorbs REST. The page.updated event's blockDelta is the additive Phase-3 field (the minimal op set — unchanged blocks are untouched).
Persistence is the only writer to blocks from the realtime path; the REST BlockModule batch endpoint remains for non-collaborative writes (import, AI generation, automation) and uses a CAS on pages.version to serialize against a concurrent flush (flush wins; REST sees fresh state on retry — CONFLICT carries the new version, see backend, §12).
Soft deletes: CRDT tombstoned blocks become deleted_at rows (kept for history; purged by retention). New blocks re-using an id is impossible (cuid, monotonic per creator).
Crash safety: a node crash loses at most 10 s of unpersisted updates (updateLog is in-memory). To bound this, flushes also happen on SIGTERM (drain) and the api's REST path can force-flush via an internal endpoint (admin).

4.4 What does NOT go through Yjs (and why)

Data	Path	Why
Record property values	REST (`PATCH /databases/:id/records/:rid`) + `record.updated` socket broadcast	values are server-validated (property type, relation cardinality, computed-property protection); CRDT-merging hostile values is a security hole — the server must veto
Comments	REST + `comments.event` push	append-mostly, small, needs mention fan-out logic (worker), not CRDT
Notifications	REST + `notification.push` on control socket	per-user, not per-resource
Database property schema values (select options etc.)	Yjs `ySchema` (merged) + debounced persist (same flusher)	schema edits are collaborative in practice (two admins adding columns)
File uploads	REST + `file.uploaded` event → search	out of scope for doc state

The rule: Yjs merges structure and text; REST validates semantics.

4.5 Resume protocol (after reconnect)

client reconnects (same page, has local ydoc from memory/IndexedDB)
  1. new socket → auth → join room
  2. client sends sync.request {stateVector: localSV}
  3. server:
       hot room:    delta = ydoc since localSV → 1–few sync.state messages
       warm room:   same, but the doc was re-hydrated from Postgres after a flush —
                    still correct (Postgres had everything ≤ last flush; the client's
                    own unflushed local ops are in its local doc and get merged)
       cold room:   hydrate first, then delta
  4. client applies delta → merge is idempotent (CRDT) → sync.ack
  5. client ships any OUTBOUND ops it queued while offline (offline queue, §5) —
     these may already be in the server doc (if another tab of the same user synced
     them) — the CRDT merge discards duplicates automatically

No "document version" handshake is needed beyond state vectors: Yjs state vectors are the resume protocol. The updateLog ring buffer (last ≤ 2000 updates or 10 MB, kept per room until the next successful flush) covers the window where the client's last ack is older than the last persisted state (rare; usually the delta from the live doc suffices).

5. Presence & Cursors

5.1 Awareness payload schema

Each peer's awareness state (Yjs Awareness.setLocalStateField):

interface AwarenessPayload {
  user: { id: string; name: string; color: string };   // color assigned from a
                                                       //   20-color palette by userId hash
  cursor?: {
    blockId: string;
    offset: number;               // text offset within the block
    selection?: {                 // non-trivial selections only
      fromBlockId: string; fromOffset: number;
      toBlockId: string;   toOffset: number;
    };
  };
  editingBlockId?: string;        // block currently focused (drives the "pencil" badge)
  page?: { pageId?: string };     // when present on the database room: which record page
  lastSeen: number;               // epoch ms (server heartbeat uses this)
}

Cursor update rate: client debounces cursor.move/presence.patch to 50 ms while typing/moving; server broadcasts patches immediately (they're tiny). A peer moving a cursor across a 100-peer page generates ≤ 20 patches/s — trivial.
Selections are rendered as per-user-color highlights (read-only layers in the ProseMirror decorations); cursors are carets with name tags (DOM overlay, absolutely positioned from the block's layout — recomputed on scroll/layout via IntersectionObserver, not per-frame).
Editing badge: the page's top bar shows avatars of peers (presence set) with a pencil for editingBlockId; the block itself gets a colored border (aria-live announcement "Ana is editing" once per join, see frontend, §10).

5.2 Presence guarantees

Presence is best-effort: no acks, no persistence. A flaky peer blinks out (drop at 60 s) and back in (re-set on reconnect). No UI state is ever required to be correct for presence to work.
Colors are deterministic per user (hash → palette), so a user is the same color in every room/tab (matches how their name tag looks elsewhere in the app).

5.3 Implemented (TASK-0093, CLOSED 2026-09-27 — recorded deviations)

Presence is implemented over the (b) additive-bridge control frames, not the §2.2 binary awareness relay. Recorded conflicts (the task file docs/tasks/phase-07-collaboration.md TASK-0093 "Implementation" section is the full record):

Wire: client → presence {cursor?, viewing?} + presence-leave (identity stripped); server → per-peer presence {pageId, user:{id,name,color}, cursor?, viewing?, lastSeen} + presence-leave. The binary 0x02 awareness relay (presence.set/patch/drop, hello.awareness[]) is reserved for TASK-0136 (the Yjs provider mount). viewing is a discrete boolean (the spec's viewing: null reads as "hidden" — false is implemented).
Server-stamped names: user.name = the handshake-resolved displayName ?? name (a client can't impersonate a name); user.color = the shared deterministic presenceColor(userId) (fnv-1a → the 20-name palette; the web renders hue = index × 18° — the app has no per-presence color tokens, so the hue is computed at runtime, no hex in components).
Join re-broadcast: on each join the room re-sends every peer's current state once (deterministic catch-up; the client's assert() also re-sends its own full state on reconnect). The 90 s prune + the explicit/socket-close leave both announce presence-leave (a peer's own tab never renders its own entry — the bar shows others; a leaving user's other tabs keep the entry alive — server keyed per userId).
Cap: beyond 50 peers the broadcast strips cursor/viewing (the avatars-only §5.1 rule — the 50/51 boundary is integration-tested).
Room key: presence frames carry the room key (page:<id> / database:<id>) in pageId; the web frame handler strips it (block frames carry the bare id).
Client cadence: caret moves coalesce at 120 ms (trailing flush; the spec's 50 ms was a target, not a measured requirement — well under the 500 ms budget); caret clears + viewing changes are immediate.

6. Scaling

6.1 Room sharding (consistent hash)

instanceCount is discovered at boot (K8s Service DNS / env RT_INSTANCE_COUNT + ordinal); owner(roomKey) = hash(roomKey) % instanceCount.
The edge (nginx) routes by room: the WSS handshake query carries room=…; nginx hash $arg_room consistent across the realtime upstream (K8s Service with externalTrafficPolicy: Local + a room-key annotation, or an Envoy-based filter in larger deployments). When the edge can't know the owner (room hash unknown to the proxy — page:abc is opaque), the fallback is the redirect dance:
1. client connects to node X; X computes owner(roomKey) ≠ me
2. X replies with a JSON frame {"t":"route","connectTo":"wss://…/realtime?node=<id>"} before closing (code 4002)
3. client reconnects to the named node (one extra RTT, only on imbalance/rebalance)
Rebalance: on scale-up, instanceCount changes → room ownership shifts. Nodes watch a Redis RT_FLEET key (each node lists itself with a 10 s TTL lease). On fleet change: a node losing a room does persist-final + bye 4001; the new owner hydrates on first reconnect. Max disruption = one reconnect per moved room, batched (rebalance moves rooms in 10% slices per 5 s to avoid a thundering herd).

6.2 Per-connection backpressure & limits

Limit	Value	Enforcement
New connections	30/min/IP, 100/user, 2000/node	counter at upgrade; 4429 close
Inbound message rate	100 msg/s sustained, burst 200	token bucket per socket; exceed → `error RATE_LIMITED`, then close `4008` after 10 s of excess
Inbound update size	256 KB/message, 2 MB/s sustained	size check + token bucket; `TOO_LARGE` error
Outbound queue	16 KB/socket in-flight	`ws` `onDrain`-gated send: if the socket's send buffer > 16 KB, the room pauses fan-out to that peer and queues (bounded 1 MB); on overflow the peer is dropped with `4009 "slow consumer"` and told to reconnect (its local doc has everything — sync on reconnect catches up)
Room size	500 peers/room (alert at 200)	beyond that, joins get `error ROOM_FULL` (503-class); product guidance: split huge docs
Node capacity	~2–5k steady rooms (measured)	HPA on `ws_connections` + `doc_flush_duration_p99`

Fan-out cost model: a keystroke from 1 of 10 peers = 9 encodes+sends of one small update. Yjs encodes per-recipient only when their state differs (state-vector check); in practice all peers receive the same bytes (batched with Buffer.concat). 100 peers × 20 updates/s = 2k small frames/s ≈ 20 MB/s — comfortable on a single node; this is why rooms, not nodes, are the unit of scaling.

6.3 Node sizing & health

Memory: a 10k-block doc ≈ 30–60 MB (Yjs doc + lastPersisted diff + updateLog). 3k docs × 40 MB ≈ 120 GB worst case — so the HPA targets connections, not memory, and cold rooms are evicted (5 min idle) to bound steady memory to ~active docs.
Health endpoints: GET /healthz (alive), GET /readyz (Postgres + Redis reachable), GET /metrics (Prometheus: rt_rooms, rt_peers, rt_update_bytes_total, rt_flush_duration_seconds histogram, rt_hydrate_duration_seconds, rt_slow_consumers_dropped_total).
Graceful drain (SIGTERM): stop accepting → bye to all sockets (staggered 10 ms apart) → persist-final every room (≤ 30 s budget; K8s terminationGracePeriodSeconds: 45) → exit 0. If persist-final can't finish in budget, the updateLog tail is written to a Redis RT_PENDING:{roomKey} key; the next owner drains it before accepting peers (crash-safe handoff).

7. Latency Budget (< 200 ms p95, peer-to-peer on the same node)

Segment	Budget (p95)	Notes
Client keystroke → Yjs op	< 5 ms	local, synchronous
Op → socket frame (client)	< 1 ms	local loopback to wss
TLS + network (same AZ)	< 10 ms	wss over 1 hop
Server auth/room lookup	< 1 ms	in-memory maps (per-socket auth cached after handshake)
Doc apply	< 5 ms	Yjs apply is O(update size); 10 KB update ≈ 1 ms
Broadcast encode (N peers)	< 10 ms for N ≤ 20	per-recipient encode only when SV differs
Network fan-out (same AZ)	< 10 ms
Peer apply + render	< 50 ms	ProseMirror DOM patch; heavy blocks defer
Total (1 → N, same node)	< 90 ms	leaves 110 ms margin to the 200 ms SLO

Cross-AZ adds ~30–80 ms (still < 200 ms with margin); cross-region is not supported (rooms are single-region by deployment). The measured SLO is enforced by a synthetic probe (two headless clients in tests/performance, histogram on update.ack RTT) in CI for every release.

Persistence latency is explicitly OUT of the 200 ms budget — it's debounced (≤ 10 s). The budget covers collaboration responsiveness, not durability.

8. Failure Modes & Mitigations

Failure	Detection	Behavior	User impact
Node crash (OOM/kill)	K8s restart	in-memory docs lost; ≤ 10 s unpersisted updates or the `RT_PENDING` handoff if drain completed	peers reconnect (backoff), re-sync from Postgres (correct state ≤ last flush); edits made in the last 10 s by crashed-node peers are lost only if their clients didn't re-send — mitigated: clients keep ops in the offline queue until `update.ack` arrives, so a reconnect re-sends them (CRDT merge discards dupes)
Postgres down	readyz / persist error	rooms keep editing in memory (doc stays authoritative); persist retries with backoff (1 s → 30 s cap); `rt_persist_errors_total` alerts; after 5 min of persist failure, new writes get `error PERSIST_DEGRADED` and clients switch to "offline mode" UI (keeps working locally)	editing continues; durability window grows (documented); no data loss while Postgres is up again (flush catches up)
Redis down	readyz	event emission buffers in-memory (bounded 10k events, oldest dropped + metric) — search/webhooks/notifications lag; auth unaffected (JWT stateless); rate limits fail-open to per-process counters	search index lag grows (SLA breach, alerted); no edit impact
Client network drop	pong timeout / `onclose`	client backoff reconnect; offline queue grows (IndexedDB); awareness peer pruned at 60 s	"Offline — N changes queued" badge; on reconnect, full merge
Slow consumer (bad network)	send buffer > 16 KB	fan-out paused → bounded queue → drop at 1 MB with `4009`	that peer reconnects; everyone else unaffected
Malformed/malicious Yjs update	apply throws / size limits	update dropped, `error`, peer flagged (3 strikes → socket closed `4008` + audit `realtime.abuse` + IP rate-limit bump)	abuser disconnected; room unaffected (Yjs apply is all-or-nothing)
Two nodes owning one room (bug during rebalance)	both persist → CAS conflict on `pages.version`	the second persister gets `CONFLICT`, re-reads, re-merges (Yjs merge of its doc with the fresh Postgres state) — no user edit is lost (both docs contain both edits after merge)	none observable
Full room (500 peers)	join check	`ROOM_FULL` to latecomers; page opens in read-only REST mode for them	latecomer sees a "too many editors" banner, can still view
Clock skew (client)	n/a	no client clock is trusted for ordering (Yjs Lamport clocks + fractional positions are monotonic per actor); `lastSeen` is advisory only	none

Crash-consistency statement: at any moment, Postgres holds a state that is a valid prefix-consistent view of the doc (≤ 10 s stale), and every client's doc is a superset merge of (Postgres state) ∪ (ops it has been acked) ∪ (its own pending ops). On any recovery, merging converges to the union of all edits — no edit is ever silently discarded; at worst, an unacked edit is replayed and deduplicated.

9. Security Notes (summary — detail in security)

JWT verification with the shared API secret; userId/workspaceId claims are never re-derivable by the client — the server re-checks membership and permission per room join (not per message — the per-message check is the canEdit flag set at join).
VIEW-only rooms: sockets receive sync.state/update frames but the server drops any update from them (guard + per-socket flag, double-checked).
Share-page viewers: public share links without a JWT join the room with a share-token principal (from Share row: scope must include VIEW; comments allowed iff allowComments), no userId claim — presence shows "Guest (shared)".
All sockets log: connect (userId, room, ip, ua), disconnect reason, drops with cause (slow consumer / abuse / auth failure) — the abuse path writes audit realtime.connection.abuse.
WSS only (nginx terminates TLS; plaintext ws is not exposed past the pod network).

Realtime Architecture (apps/realtime)