Factory Observability (TASK-0135, phase-15)

Prometheus + Alertmanager + Grafana stack for the AI Software Factory, built on the OTel instruments shipped by packages/observability (the 14-metric spec set + 1 operational) and the realtime app's local ws_* registry.

Layout

infrastructure/observability/
├── prometheus/
│   ├── scrape.yaml            # scrape configs (mounted as prometheus.yml)
│   ├── recording-rules.yaml   # p95/route-group, error rate, queue, ws, search, ai
│   └── alerts.yaml            # the 6 spec alerts + 2 operational (BackupStale, SearchIndexLagHigh)
├── alertmanager/
│   └── alertmanager.yaml      # repeat_interval 4h; page → webhook+email, warn → email
├── grafana/
│   ├── dashboards/            # the 6 spec dashboards (fixed uid factory-prometheus)
│   │   ├── api-overview.json  realtime.json  workers-queues.json
│   │   ├── search.json        ai.json        infra.json
│   ├── provisioning/
│   │   ├── datasources/datasource.yaml   # uid factory-prometheus → prometheus.monitoring:9090
│   │   └── dashboards/dashboards.yaml    # folder "Factory", path /var/lib/grafana/dashboards
│   └── ...
├── scripts/
│   └── dashboard-completeness.mjs   # mechanical check of the panel-coverage contract
└── README.md

The k8s manifests that deploy this (namespace monitoring) live in infrastructure/kubernetes/observability/. Apply:

kubectl apply -k infrastructure/kubernetes/            # the apps + networkpolicy.yaml
kubectl apply -k infrastructure/kubernetes/observability/   # the stack (ns monitoring)

observability/ is a standalone apply root (NOT appended to the parent kustomization): the parent's namespace: factory transformer would override the stack's monitoring namespace (verified with kustomize v5.3.0), breaking the FQDN the factory-config configmap already points the OTLP exporter at (otel-collector.monitoring.svc.cluster.local). Only networkpolicy.yaml (the factory-namespace policies) was appended to the parent. Conflict recorded in the TASK-0135 report (AGENTS rule 29).

Metric catalog (every emitted family)

packages/observability instruments (15 — EXPOSED names)

Classic text format: counters expose with a _total suffix that is NOT in the instrument name — http_request_count exposes as http_request_count_total; ws_messages_total, ai_tokens_total, storage_ops_total, error_total already end in _total and are unchanged. Series carry attribute labels only (no resource labels) plus the scrape job/service_name static label from scrape.yaml; target_info{service_name=...} is the always-present identity gauge.

workspaceId is ABSENT (not "unknown") when unresolvable.

apps/realtime local registry (10 — merged into the same exposition)

The realtime app keeps its own Metrics class (phase 10) and MERGES its render with the OTel text on /metrics (mergeExposition: local families win on a name collision, so ws_messages_total is the LOCAL one — the OTel message counter is deliberately not wired to avoid a label-shape collision).

External (expected-present)

Alerts (8 total)

Routing (alertmanager): repeat_interval: 4h; page → platform incoming-webhook (self-dogfood) + email; warn → email only. PLACEHOLDERS to replace before staging: the webhook URL, the ops mailbox, and the SMTP relay (see the header comments in alertmanager/alertmanager.yaml).

Unpaneled / gap

Families emitted but intentionally NOT in a panel (the spec's completeness contract requires them to be listed here):

• uptime_seconds — process health is the pod liveness probe's job; no panel needed (the realtime "Active connections" stat covers the user-facing view).
• connected_clients — duplicate of ws_active_connections by design (the local registry counts the same states map); one panel is enough.
• events_processed_total — internal app counter, no spec panel or alert; available in Explore for debugging.
• ws_handshake_failures_total — no spec panel; a WsHandshakeFailuresHigh alert is the follow-up if upgrade rejections become an operational concern.
• target_info — identity gauge (drives the service-identity convention), not a panel target by design; query it in Explore to verify a scrape.
• probe_ssl_earliest_cert_expiry — the CertExpiringSoon alert covers it; a panel would need the blackbox exporter deployed first.

Spec-conflict gaps (AGENTS rule 29 — recorded, not guessed)

• Workers/Queues — DLQ count + retry rate panel: the spec's dashboard asks for them, but NO DLQ/retry metric exists in the spec's 14-metric set and BullMQ 5 exposes none via the shipped instruments. Failed jobs are visible today via error_total{code} (worker) + the /health probe. Follow-up: a queue_failed_jobs_total instrument if the gap matters.
• AI — budget-approaching gauge: the ai:usage-rollup repeatable writes budget usage to the DB but exports no Prometheus gauge (budget is outside the 14-metric set). The token-rate panels are the live proxy for burn.
• Infra — per-pod cgroup / PG pool / Redis hit rate / OpenSearch health gauges: exporter-dependent (cadvisor/kubelet, pg_stat_activity, INFO stats, /_cluster/health) and outside the 14-metric set. The panels use app-side proxies (db/redis p95, storage ops) instead.
• http_request_count naming: the spec's literal alert expr uses http_request_count; the 0.57.0 exporter exposes it as http_request_count_total. All alerts/dashboards use the EXPOSED name (the only deviation from the spec's literal expressions, forced by the exporter — recorded here per rule 29).

Verification status

• node infrastructure/observability/scripts/dashboard-completeness.mjs → the mechanical panel-coverage check (this section + the six dashboards).
• All staging-only verifications (live OTLP export, in-cluster Prometheus scrape, Grafana rendering, alertmanager delivery, NetworkPolicy deny-tests) are listed in the task report (AGENTS rule 29 — not runnable in this environment).