AI Software Factory — Kubernetes Deployment

Bare-Kubernetes manifests for the production stack (NestJS API, Next.js web, BullMQ worker, WebSocket realtime) plus the self-hosted data layer (Postgres, Redis). Ingress is for the nginx ingress controller; TLS is issued by cert-manager.

Deployment paths. This repo supports two deployment paths:

1. Kubernetes (recommended) — this directory, rendered by kustomize.
2. Terraform + EC2 — see ../terraform/ for a simpler single-AZ self-hosted stack (no k8s). Pick one per environment; the k8s path is the recommended production path.

Manifest map

Walkthrough

0. Prerequisites

• A cluster (EKS/GKE/minikube/k3s) with the nginx ingress controller and cert-manager (with a cluster issuer, e.g. letsencrypt-prod as used in ingress.yaml) installed.
• Access to ghcr.io for pulling images (a GitHub token with packages:read → kubectl create secret docker-registry ghcr-pull --docker-server=ghcr.io ... and add imagePullSecrets to the deployments if you make images private).
• DNS: factory.example.com A/ALIAS record pointing at the ingress controller's LB address.

1. Create the secrets FIRST

# Strategy A (recommended): SealedSecrets
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/... # install once
sealed-secrets create infrastructure/kubernetes/secret.example.yaml \
  -n factory > sealed-factory-secrets.yaml   # commit THAT, not the raw secret

# Strategy B: External Secrets Operator → values in AWS Secrets Manager/Vault
# Strategy C (throwaway only):
kubectl -n factory create secret generic factory-secrets \
  --from-literal=DATABASE_URL='postgresql://factory:REAL@factory-db:5432/factory?schema=public' \
  --from-literal=JWT_SECRET='$(openssl rand -hex 32)' \
  --from-literal=REFRESH_TOKEN_SECRET='$(openssl rand -hex 32)' \
  --from-literal=SESSION_SECRET='$(openssl rand -hex 32)' \
  --from-literal=MASTER_ENCRYPTION_KEY='$(openssl rand -hex 32)' \
  --from-literal=S3_ACCESS_KEY='...' --from-literal=S3_SECRET_KEY='...'
kubectl -n factory create secret generic factory-db \
  --from-literal=POSTGRES_USER=factory \
  --from-literal=POSTGRES_PASSWORD=REAL \
  --from-literal=POSTGRES_DB=factory

secret.example.yaml documents the full key set and the SealedSecrets / SOPS / ExternalSecrets options — never commit real values.

2. Set the image tag (kustomize images variable)

The manifests all carry image: .../api:TAG; kustomize substitutes the tag from the images: block in kustomization.yaml (single source of truth). Bump newTag (or let CI do it), e.g. to v1.2.3.

3. Apply

kubectl apply -k infrastructure/kubernetes/
kubectl -n factory rollout status deploy/api
kubectl -n factory get pods,svc,ingress,hpa

First boot sequence: postgres comes up → api pods stay not ready until /api/v1/health/ready passes (it checks DB connectivity). Migrations run on the api startup path (prisma migrate deploy); watch: kubectl -n factory logs -l app.kubernetes.io/name=api --tail=50 -f.

4. Verify

curl -fsS https://factory.example.com/api/v1/health
curl -fsS https://factory.example.com/api/v1/health/ready
curl -i https://factory.example.com            # → Next.js (or its maintenance page)
wscat -c wss://factory.example.com/ws          # handshake (token required after connect)

HPA notes

• api (2..10, CPU 70%): standard. Scale-down is stabilized (300 s window, 1 pod/120 s) to avoid flapping under bursty traffic.
• worker (2..8, custom metric): scales on bullmq_queue_depth (averageValue 50 jobs/pod). Requires the Prometheus Adapter mapping — the worker.yaml header contains a ready-to-paste adapter rule. Until that is wired, use the commented CPU fallback.
• realtime (1..6, CPU 70%): CPU is a weak proxy for connection count; the file documents how to add a ws_connections pods-metric once the server exposes the gauge.
• web: no HPA (static-ish Next.js; 2 replicas + PDB suffice; add one if you scale the frontend).
• HPA requires the metrics-server (CPU/memory) or the Prometheus adapter (custom metrics) to be running.

Blue-green / progressive delivery (Argo note)

These manifests are deliberately plain so they compose with Argo Rollouts or Argo CD:

• Argo CD: point an Application at this directory (path: infrastructure/kubernetes, namespace: factory); syncs are declarative and drift-detected. The images: tag bump becomes the deploy mechanism. The monitoring stack needs its OWN Application (path: infrastructure/kubernetes/observability, namespace: monitoring) for the same standalone-apply-root reason documented in the manifest map above.
• Argo Rollouts (blue-green / canary): replace the api/web Deployments with Rollout resources (same selectors, add strategy.rollingDeploy → blueGreen with an Analysis template that watches /api/v1/health/ready + error rate before promoting). The Services here are the active services — keep stable names and let rollouts manage preview/active service switching.

Operational notes

• Postgres is self-hosted here for completeness — for real production use managed RDS/Aurora/Cloud SQL (see the note in postgres/statefulset.yaml) and swap the DATABASE_URL secret value.
• PVCs: pgdata (20Gi, from the StatefulSet VCT) and factory-redis (5Gi, create separately — snippet in redis.yaml). Set an encrypted storageClassName on both for production.
• realtime sticky sessions: NOT required — cross-replica room fan-out goes through Redis pub/sub. realtime.yaml documents the ip_hash fallback if you ever run a single-replica-room implementation.
• PDBs protect api and web during node drains. worker/realtime are covered by their HPAs' minReplicas + maxUnavailable: 0 rollout.
• Secrets never live in git (see secret.example.yaml header for the SealedSecrets / SOPS / ExternalSecrets options). Monitoring-stack secrets, created the same way before the first apply: grafana-admin (admin-password) in monitoring — the Grafana deployment fails without it; the alertmanager SMTP relay is assumed open (or credentials go in the source config file — see observability/alertmanager.yaml header).
• Observability (TASK-0135): the stack lives in monitoring (kubectl apply -k infrastructure/kubernetes/observability/); config sources + the dashboard/alert documentation live in ../observability/. The apps' OTLP exporter target is factory-config's OTEL_EXPORTER_OTLP_ENDPOINT (flip OTEL_ENABLED to "true" to go live — default "false" keeps dev/CI silent). The worker HPA scales on queue_depth (the metric the worker itself emits — the Prometheus Adapter mapping is in worker.yaml's header).