Self-Hosting Admin Guide (v1.0)
Operator's companion to deployment.md. This guide is the
action layer: one-command paths, the environment/secret matrix, sizing,
backup configuration, and observability. The reference layer (topology,
nginx walkthrough, restore runbook, SSO/SCIM setup) stays in
deployment.md — when in doubt, the runbook there wins.
1. One-command deployment
1.1 Docker Compose (single server / trial)
Prereqs: Docker + compose plugin, Node ≥ 20 + pnpm ≥ 9 (to build the four app
images once), .env filled in (copy .env.example).
# data layer (postgres, redis, minio, opensearch, clamav)
pnpm docker:up
# build the app images once (tags them :local)
docker compose --profile apps build
# full stack: the four apps + infra
docker compose --profile apps up -d
Everything then runs under compose with healthchecks; the repo-root .env is
the single config source (compose's env_file). Reset all data with
docker compose down -v. The single-server production variant (nginx TLS in
front, mailpit/SMTP relay for mail) follows deployment.md §4 — the same
compose file, the apps profile, plus the nginx service from
infrastructure/nginx/nginx.conf.
1.2 Kubernetes (recommended production path)
Prereqs: a cluster with nginx ingress controller + cert-manager
(cluster issuer), DNS A/ALIAS for the public host, and access to the image
registry (GHCR pull secret if the images are private).
# 1 — secrets FIRST (the apply fails on missing secrets by design):
# factory-secrets + factory-db from infrastructure/kubernetes/secret.example.yaml
# (SealedSecrets / SOPS / ExternalSecrets — see that file's header)
kubectl apply -k infrastructure/kubernetes/
# 2 — the deployed tag is ONE variable (kustomization.yaml `images:` block):
kustomize edit set image ghcr.io/<org>/api=<tag> # or edit newTag by hand
kubectl apply -k infrastructure/kubernetes/
# 3 — verify
kubectl -n factory rollout status deploy/api
curl -fsS https://<host>/api/v1/health && curl -fsS https://<host>/api/v1/health/ready
First boot: postgres comes up → api pods stay not ready until
/api/v1/health/ready (DB connectivity) passes. Migrations run on api
startup (prisma migrate deploy) — the upgrade path is simply "deploy the
next tag"; rollback = redeploy the previous tag (manifests carry no
state; the schema is forward-only, so a schema rollback needs the restore
runbook, not a redeploy). See infrastructure/kubernetes/README.md for the
full manifest map and deployment.md §3 for the workload walkthrough.
2. Environment & secret matrix
Split of authority: non-sensitive → ConfigMap (factory-config,
infrastructure/kubernetes/configmap.yaml) / .env values; secrets →
the factory-secrets Secret (or .env under compose). In k8s, envFrom
applies configmap-then-secret, so a secret key always wins over a configmap
key of the same name.
2.1 Required secrets (factory-secrets)
| Key | What | Generation |
|---|---|---|
POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB |
DB credentials (also in factory-db for the StatefulSet) |
pick strong values |
DATABASE_URL |
assembled primary DSN (?schema=public; via PgBouncer in prod) |
from the above + host |
JWT_SECRET |
access-token HMAC | openssl rand -hex 32 |
REFRESH_TOKEN_SECRET |
refresh-token HMAC (rotation) | openssl rand -hex 32 |
SESSION_SECRET |
session/cookie signing | openssl rand -hex 32 |
MASTER_ENCRYPTION_KEY |
at-rest field encryption | openssl rand -base64 32 |
S3_ACCESS_KEY / S3_SECRET_KEY |
object-storage credentials | MinIO IAM / cloud |
SEARCH_USERNAME / SEARCH_PASSWORD |
OpenSearch admin (empty in dev, single-node, security off) | per deployment |
AI_API_KEY |
AI provider key (empty ⇒ provider disabled/mock) | provider |
SMTP_USER / SMTP_PASSWORD |
mail relay (empty ⇒ stub transport logs only) | mail provider |
2.2 Non-sensitive config (ConfigMap / .env highlights)
| Key | Dev default | Prod reference | Notes |
|---|---|---|---|
PUBLIC_BASE_URL / WEB_URL / API_URL / REALTIME_URL |
http://localhost:{3000,4000,4001} |
https://<host> (+ /api, wss://<host>/ws) |
what email links & the SSO bindings use; SSO_ORIGIN must be the origin serving /saml/*, /sso/* (the API origin in two-origin dev; empty in single-origin) |
DATABASE_REPLICA_URL |
empty | PgBouncer reader pool on the replica |
db.read() routes tree/listing/search here; permission + If-Match version reads are NEVER replica (@factory/database-client) |
DATABASE_POOL_SIZE |
20 | per-app share of the PgBouncer pool | PgBouncer (transaction mode) owns pooling; this is the app's share, not a Prisma option |
PERF_CACHE |
auto (off in dev) |
auto (on in production) |
Redis perf-cache gate |
SEARCH_URL / SEARCH_INDEX_PREFIX |
http://localhost:9200 / factory |
cluster endpoint / factory |
indices: factory-content, factory-docs, ai-chunks (by alias) |
S3_ENDPOINT / S3_BUCKET / S3_REGION / S3_FORCE_PATH_STYLE |
MinIO :9000 / factory-uploads / us-east-1 / true |
cloud S3 endpoint / bucket / region / false |
upload buckets are versioned (§4) |
JWT_EXPIRES_IN / REFRESH_TOKEN_TTL_DAYS / BCRYPT_ROUNDS |
15m / 30 / 12 |
same | session tuning |
REALTIME_HEARTBEAT_MS / YJS_STORAGE_PREFIX |
30000 / yjs |
same | 30 s WS heartbeat; Redis prefix for Yjs update logs |
WEBHOOK_RETRY_MAX / WEBHOOK_TIMEOUT_MS |
8 / 10000 |
same | webhook delivery policy |
RATE_LIMIT_WINDOW_MS / RATE_LIMIT_MAX |
60000 / 300 |
tune per tier | app-level rate limit (authoritative; nginx is first line) |
OTEL_ENABLED / OTEL_EXPORTER_OTLP_ENDPOINT |
false |
false in self-host (NFR-H5 opt-in), true in staging/prod-cloud → collector http://otel-collector.<obs-ns>.svc.cluster.local:4318 |
telemetry is opt-in per NFR-H5 |
AI_PROVIDER |
mock (dev) |
provider of choice | mock is deterministic (echo) — staging runs on it so release E2E is reproducible |
WEBHOOK_ALLOW_PRIVATE |
true (dev only) |
false |
allows webhook targets on private/RFC1918 ranges |
The full variable list with every default: .env.example (the API validates
it via zod — apps/api/src/config/env.ts).
3. Sizing
Two tables, by design: the manifest values (what the plain k8s manifests
ship today) and the NFR-H3 production floor (the target for a real
customer-facing deployment, deployment.md §1 + docs/product/non-functional-requirements.md).
3.1 App workloads
| Workload | Manifest (k8s base) | Prod floor (NFR-H3) | Scaling |
|---|---|---|---|
api |
2 replicas, req 250m/512Mi, lim 1/1Gi | 3 replicas, req 1000m/512Mi, lim 2000m/1Gi | HPA 2→10 (manifest) / 3→10 (floor), CPU 70% |
web |
2 replicas, req 200m/512Mi, lim 1/1Gi | 2 replicas, req 500m/512Mi | no HPA (static-ish Next.js) |
worker |
2 replicas, req 250m/512Mi, lim 1/1Gi | 2+ per queue family (search/files, webhooks/automation, email/ai/jobs — deployment.md §3.1), req 1000m/1Gi |
HPA 2→8 on bullmq_queue_depth (custom metric; CPU fallback in the manifest) |
realtime |
2 replicas, req 250m/512Mi, lim 1/1Gi | 2 replicas, req 1000m/1Gi | HPA 1→6 (manifest) / 2→6 (floor) on connections |
3.2 Data layer
| Component | Manifest (self-host) | Prod floor | Notes |
|---|---|---|---|
| Postgres primary | 1 replica StatefulSet, 20Gi PVC | 8 vCPU / 32 GB | PITR via wal-g (archive_command → S3 pg/wal_005/); daily base backup 05:00 UTC |
| Postgres replica | postgres/replica.yaml (hot standby, for db.read()) |
4 vCPU / 16 GB | reads serve tree/listing/search only |
| PgBouncer | postgres/pgbouncer.yaml, transaction mode |
2 replicas | DATABASE_URL routes through it; per-app pool_size = DATABASE_POOL_SIZE |
| Redis | 1 Deployment, AOF on PVC | 7, AOF appendfsync everysec, 16 GB |
not part of DR (rebuildable — §4) |
| OpenSearch | per deployment.md (single node in dev; 1–3 in prod) |
3-node cluster | daily snapshot 05:30 UTC → S3 repo factory-s3 |
| MinIO/S3 | minio/minio (compose) / managed S3 |
4-node erasure (self-host) or managed | upload buckets versioned |
Storage math for the uploads bucket: 500 MB per-file cap (deployment.md §5
nginx client_max_body_size 512m); budget disk on the PG primary for
WAL_S3_PREFIX growth ≈ write-rate × 26 h (the gate window, §4).
4. Backup configuration (PITR env)
The release pipeline's stage 9 hard gate (scripts/backup-health.ts)
reads the real artifacts and fails on staleness. Configuration that produces
those artifacts:
| Env | Value | Used by |
|---|---|---|
WAL_S3_BUCKET |
the backup bucket (factory-backups) |
wal-g WAL archiving (pg/wal_005/) + base backups (pg/basebackups_005/) |
WAL_S3_PREFIX |
pg |
object prefix for both |
WAL_S3_REGION |
e.g. us-east-1 |
wal-g / S3 SDK |
S3_ENDPOINT |
backup S3 endpoint | both cronjobs |
S3_ACCESS_KEY / S3_SECRET_KEY (backup identity) |
least-priv IAM (put/get on the backup prefix only — postgres/walg-iam-policy.yaml) |
both cronjobs |
OS_SNAPSHOT_REPO |
factory-s3 |
OpenSearch snapshot CronJob (05:30 UTC) |
OS_URL |
OpenSearch endpoint | snapshot + health check |
BACKUP_S3_PG_PREFIX |
pg |
backup-health |
BASE_MAX_HOURS / WAL_MAX_MINUTES / OS_MAX_HOURS |
26 / 10 / 26 |
backup-health thresholds (defaults; the WAL value is the RPO ≤ 5 min headroom) |
DEPLOY_STAGING / BACKUP_HEALTH_STRICT |
staging: true |
unreachable/empty endpoint ⇒ hard FAIL (dev: loud report-only skip) |
Verify locally: pnpm backup:health (with the env above). The restore
procedure itself — pick T, scratch namespace restore-drill, wal-g restore,
the 3 verification queries, RPO/RTO recording — is not repeated here; it
lives in deployment.md §8 (runbook + executed-drill log).
5. Observability
Stack (deployment.md §9): OTel Collector → Prometheus (metrics) +
Loki (logs) → Grafana (8 dashboards as code) + Alertmanager
(18 alert rules, critical pages on-call, warn to ops). Namespace
factory-observability, internal-only (NetworkPolicy).
| Step | What |
|---|---|
| 1. Flip the apps | OTEL_ENABLED=true + OTEL_EXPORTER_OTLP_ENDPOINT pointing at the collector (HTTP 4318). All four apps (factory-web, factory-api, factory-worker, factory-realtime) start exporting traces/metrics; pino logs gain requestId correlation. |
| 2. Scrape | Collector's prometheus receiver + the scrape config for the prometheus.io/* pod annotations; recording rules for p95-per-route-group, error rate, queue depth. |
| 3. Dashboards & alerts | Provision the 8 dashboards + the alert rules (deployment.md §9.2–9.3); alertmanager routes critical → on-call webhook + email (the platform's own automation + mail queue). |
| 4. Verify in staging | Every metric in the metric list is scraped (the /metrics conformance test), dashboards render real data, and a chaos step (stop the worker) fires QueueDepthHigh within 5 m. |
Self-host default is OTEL_ENABLED=false (NFR-H5, telemetry opt-in) — the
apps then cost nothing for it; flipping is a config change, not a deploy of
new app code.
6. Day-2 checklist
| When | Do |
|---|---|
| Every deploy | kustomize edit set image … <tag> → kubectl apply -k → rollout status → /api/v1/health/ready (migrations ride the api startup) |
| Daily | Confirm pg-backup (05:00 UTC) + opensearch-snapshot (05:30 UTC) CronJob success; run pnpm backup:health |
| Weekly (staging) | Refresh the synthetic Postgres clone; nightly perf-standard run |
| Quarterly (prod) | Full DR drill (deployment.md §8.5) — PITR to T−5 min + snapshot restore, 3 verification queries, record RPO/RPO deviations |
| On incident | Restore runbook deployment.md §8.1 (the scratch namespace is the only elevated procedure — two-person approval, org-admin role) |
| Certificates | Ingress cert < 14 d → the CertExpiringSoon alert; renew before expiry |
7. Related runbooks
deployment.md— topology, nginx, release process, DB migration/rollback, backup & restore runbook, SSO/SAML/OIDC + SCIM, retention.support-matrix.md— supported versions & EOL dates.infrastructure/kubernetes/README.md— manifest map + HPA notes.docs/api/— the REST API (OpenAPI atdocs/api/openapi.yaml).