Automation Architecture

Phase 11 — the automation engine: the execution pipeline that the API surface (../api/automations.md) enqueues into, and the UI contract (../ui/automations.md). Scope: the run lifecycle, the event matcher, error classification and retry policy, dead-lettering, loop prevention, schedules, and the watchdog:

                       factory:events (Redis stream)
        ┌────────────────────────────────────────────────────────────┐
        │  record.created / record.updated / page.created /          │
        │  form.submitted                                           │
        └───────────────┬────────────────────────────────────────────┘
                        │  StreamConsumer (group "automations-matcher")
                        ▼
        matcher.ts  —  matchAndEnqueue: ACTIVE automations, scope match,
                       60 s dedupe, PENDING run row, counter bumps
                        │
                        ▼ enqueueRunJob (own Queue handle, add + close)
        BullMQ queue "automations"  —  job "run-automation"
                        │
                        ▼  automations.processor.ts
        pipeline.ts  —  handleRunAutomation: loop guard → action-count
                       guard → condition gate → 1..50 actions in
                       position order (per-action timeout + classify)
                        │
        ┌───────────────┼─────────────────────────────┐
        ▼               ▼                             ▼
   SUCCEEDED       RETRYING (backoff,        FAILED / DEAD_LETTER
   (+ skip)        re-enqueue same runId)    (+ notification, audit,
                                               executed event, counter)

Plus the schedule path (BullMQ repeatables, reconciled worker-side) and the watchdog (stale-run reclaimer, §7).

Source of truth for every shared constant (trigger/action enums, the condition schema, the 9 action schemas, cron validation, the status machine, depth cap, backoff schedule) is packages/shared/src/automation.ts — the API and the worker both import it; neither re-declares a constant.

---

1. Run lifecycle (state machine)

                 ┌──────────┐  worker picks up   ┌──────────┐
   matcher /     │          │ ──────────────────→ │          │
   API / ───────→│ PENDING  │                     │ RUNNING  │
   schedule /    │          │ ←────────────────── │          │
   manual        └──────────┘  scheduleRetry      └────┬─────┘
                                                       │
        ┌───────────────────┬──────────────────────────┼─────────────────────┐
        ▼                   ▼                          ▼                     ▼
  SUCCEEDED           SUCCEEDED               FAILED               DEAD_LETTER
  (actions done)      (skipped: conditions    (fatal error, or      (retryable error
                      self_trigger,           retryable at the      on the LAST
                      depth_limit)            cap — see §3)         attempt)

• Runs are rows in automation_runs (Prisma); the row is the state machine, never BullMQ. BullMQ jobs are cheap re-drive handles; the run id travels in the job payload and is stable across retries.
• scheduleRetry hops the row RUNNING → FAILED → PENDING (with retryCount+1) and re-enqueues the same runId with a real delay (BullMQ delay, not attempts — the worker manages retries itself so the backoff, the counter and the run row are all visible in one place).
• Terminal re-delivery (a job redelivered after the run already reached SUCCEEDED/FAILED/DEAD_LETTER) is dropped — the handler returns the existing status and does nothing.
• Backoff: runBackoffSeconds(n) = min(2^(n-1), 600) — attempt 1 waits 1 s, attempt 2 waits 2 s, … capped at 600 s. With the default AUTO_RUN_MAX_RETRIES = 8 a permanently-failing run dead-letters after ≈ 8 hops.

Every transition writes a structured line to automation_logs (run started, action i/n TYPE started|done|failed, run completed, skip reasons, run dead-lettered) — every run is logged, which is what the UI history panel renders.

2. The matcher (event → run)

matcher.ts matchAndEnqueue(deps, triggerType, entry) is called by the stream consumer for each factory:events entry (group automations-matcher, one consumer per worker process):

1. Workspace scope: the entry's workspaceId must equal the automation's.
2. Resource scope: for the four database trigger types the automation's databaseId ?? trigger.config.databaseId must equal the event's payload.databaseId; PROPERTY_CHANGED additionally requires the changed-property filter (config.properties) to contain "*" or intersect payload.changedProperties.
3. Status scope: only ACTIVE automations match.
4. Dedupe: a 600 s Redis SET NX EX on (automationId, entryId) makes stream replays and duplicate publishes harmless (late consumers replay from stream id 0 — MKSTREAM groups see the full history).
5. On a match: create the PENDING run row (input = truncated trigger event, 64 KB cap) + enqueueRunJob.

Every decision is observable: automations:matched:{trigger_type} and automations:skipped:{filter_mismatch|deduped} counters.

The trigger event itself is the flattened event (buildTriggerEvent(eventName, payload) = { type: eventName, payload, ...payload }) so the condition evaluator and the executors can read top-level fields (recordId, databaseId, …).

3. Error classification → retry policy

classifyError(err) (executors/classify.ts) maps any thrown value to { retryable, code }:

Source	retryable	code
FatalError (executor-declared)	no	its code
RetryableError (executor-declared)	yes	its code
Prisma transient SQLSTATE {40P01, 40001, 57P01, 55P03, 57014}	yes	DB_TRANSIENT_{state}
network errno {ECONNRESET, ETIMEDOUT, EPIPE, ECONNREFUSED, EAI_AGAIN}	yes	NET_{errno}
anything else	no (fail closed)	ACTION_TIMEOUT (TimeoutError) or UNEXPECTED

Fail-closed by default: an unknown error must not silently burn the retry budget. Executors declare intent explicitly — e.g. SEND_WEBHOOK maps HTTP 429/5xx to RetryableError("WEBHOOK_HTTP_RETRIABLE") and other 4xx to FatalError("WEBHOOK_HTTP_ERROR"); undici transport timeouts carry no errno (UND_ERR_CONNECT_TIMEOUT on the cause) and therefore classify as fatal UNEXPECTED — a dead network target fails once, not eight times.

Per-action execution wraps the executor in runWithTimeout(timeoutMs) (Promise.race); a timeout throws TimeoutError → fatal ACTION_TIMEOUT. retries/timeoutMs per action (optional, stored on the action row) override the defaults.

4. Dead-letter + notification

deadLetterRun (dead-letter.ts) is the terminal for a retryable error at the retry cap (and for the watchdog's crashed-run path, §7):

1. Row → DEAD_LETTER (the FAILED → DEAD_LETTER transition is asserted).
2. Log line run dead-lettered.
3. automation.executed event (depth 0 only — see §5) with status: "DEAD_LETTER" + error.code.
4. Notification to workspace admins (owner + ADMIN/OWNER members — one in-app Notification each; email only when SMTP_HOST is set): title Automation "X" dead-lettered, body Failed after N attempts: {code} — {message}.
5. Audit automation.dead_lettered (actor SYSTEM automation:{automationId}, resource automation_run).
6. Counter automations:dead_letter:{workspaceId}.

The automation.executed event (emitted for every terminal at depth 0, including skips) is the observability feed for other subsystems; its payload is { workspaceId, automationId, runId, status, triggerType, durationMs, error? }.

5. Loop prevention

Automations emit domain events (UPDATE_RECORD → record.updated, CREATE_PAGE → page.created) — which can re-trigger the automation itself. Two guards, evaluated at run start (before any action):

1. Self-trigger: the incoming triggerEvent.automationContext carries the emitting automation's id. If it equals the run's own id → the run is skipped (SUCCEEDED, output: { skipped: "self_trigger" }). An automation can never be re-driven by its own output.
2. Depth limit: each emission stamps automationContext.depth + 1; runs at depth ≥ AUTOMATION_MAX_DEPTH (5) are skipped ({ skipped: "depth_limit" }). Chains of different automations are therefore bounded to 5 hops.

Self-trigger beats depth-limit in the guard (a depth-1 self-match is reported as self_trigger).

automation.executed is depth-0 only. A run triggered at depth ≥ 1 emits no executed event — the root run's event already covers the chain, and emitting per-hop would N×-fan-out for N-deep chains. (This is the reason the integration spec asserts no executed event for a depth-1 run.)

Proof sketch (bounded termination): every execution emits at most one domain event per action; each emitted event is stamped depth+1; a run at depth ≥ 5 executes nothing; a self-match executes nothing. Hence the event DAG rooted at any trigger has depth ≤ 5 and no cycles.

6. Schedules

SCHEDULE automations get a BullMQ repeatable job keyed automations:run:{automationId} (the spec's repeatable-job id). Registration is worker-side: reconcileScheduleAutomations (schedule.ts) computes the set difference between ACTIVE SCHEDULE automations and the queue's repeatables and upserts/removes — at worker boot and every 5 minutes. The API never touches the queue (keeping the API horizontally scalable and queue-free). A fire creates a PENDING run with triggerEvent = { type: "schedule", firedAt, cron } and enqueues the normal pipeline. Cron validation (shared cronCheck) rejects intervals under 60 s.

7. Watchdog (crash safety)

runWatchdog (watchdog.ts) runs every 5 minutes in the worker. With STALE_MS = 15 min:

Found	Action
RUNNING + startedAt < now−15 min, retryCount < cap	→ PENDING, retryCount+1, re-enqueue (log watchdog reclaimed stale RUNNING run)
RUNNING + startedAt < now−15 min, retryCount ≥ cap	→ DEAD_LETTER with error WORKER_CRASHED (+ admin notification Automation "X" failed (worker crash) + audit automation.worker_crashed + dead-letter counter)
PENDING + createdAt < now−15 min	re-enqueue without bumping retryCount (log watchdog re-enqueued stale PENDING run)

This is what recovers from a SIGKILLed worker: the run row is stuck RUNNING, the watchdog flips it back to PENDING and the (re)started worker re-drives it. The integration spec covers this hermetically (SIGKILL a real BullMQ worker mid-run, age the row, run the watchdog, re-drive to SUCCEEDED).

Crash-safety caveat (recorded, not fixed): a run that is picked up and immediately killed before startedAt is written stays PENDING with no staleness clock until its createdAt ages 15 min — then the PENDING branch re-enqueues it (no retry bump). A second kill in the same window can thus loop the re-enqueue; the 15-min staleness + the worker lock (lockDuration 60 s) bounds the damage, and the PENDING branch is idempotent.

8. Non-idempotency caveats (recorded)

The pipeline is not idempotent by design: a SEND_WEBHOOK / CREATE_NOTIFICATION / SEND_EMAIL action executed twice sends twice. Retries therefore trade delivery-exactly-once for delivery-at-least-once. Mitigations that DO exist: (a) the matcher's 600 s (automationId, entryId) dedupe prevents double-queueing from duplicate stream events; (b) terminal re-delivery is dropped (§1); (c) the watchdog's PENDING branch is idempotent. A webhook endpoint that must be exactly-once needs its own dedupe (e.g. on the run id in a header — the payload interpolation already supports {{run.id}}).

9. Registry + testability seam

The 9 executors are registered in registry.ts (AUTOMATION_ACTION_TYPES → executor). The pipeline dispatches through executorFor(type) = testOverrides.get(type) ?? registry.get(type) — registerTestExecutor(type, executor) / clearTestExecutors() are the only production-code testability addition in Phase 11 (recorded per AGENTS.md §29): they let the integration spec drive a deterministic FatalError/RetryableError/never-resolving executor through the real pipeline (retries, backoff, DLQ, SIGKILL recovery) without network or AI side effects. The seam reuses the production type system: the override's type must be a real AUTOMATION_ACTION_TYPES entry and its config must pass the real Zod schema (validateAction re-validates before dispatch), so a test cannot bypass validation.

10. Concurrency & resources

• Queue automations, worker concurrency AUTOMATIONS_WORKER_CONCURRENCY (default 5). lockDuration 60 s, stalledInterval 30 s.
• The stream consumer uses a dedicated ioredis (subscribable mode — a client that is also a subscriber cannot be used for the queue); the worker likewise. The matcher opens a short-lived Queue handle per event (add + close) — no long-lived queue connection per consumer.
• Run storage caps: triggerEvent/input 64 KB, output 128 KB (truncateJson, with a truncated: true flag).
• Every metric the pipeline writes: automations:matched:*, automations:skipped:*, automations:actions:{type}:succeeded|failed, automations:dead_letter:{workspaceId} (all Redis counters).