AI Software Factory — Terraform (EC2 self-host path)
Infrastructure-as-code for the simple self-host deployment path of the AI Software Factory monorepo on AWS.
The two deployment paths — read this first
This repo supports exactly two ways to run in production:
| Path | Where | When to use |
|---|---|---|
| Kubernetes (recommended) | ../kubernetes/ |
Production. Container orchestration, HPA, rollouts, PDBs, ingress. Scales the team and the workload. |
| Terraform + EC2 + systemd (this directory) | infrastructure/terraform/ |
Simple self-host: one team, ~8-16 nodes, direct SSH debuggability, no container platform to operate. |
They deploy the same components (web, api, worker, realtime, Postgres, Redis, S3, OpenSearch) with the same names and ports. Pick ONE per environment; do not mix. The Terraform path is deliberately boring: EC2 ASGs behind one ALB, managed data services, everything else via SSM parameters.
Why EC2 + systemd instead of ECS/EKS? (documented choice)
ECS would add a container platform to operate for a workload that is four
long-running node processes; EKS duplicates the k8s path. With systemd, a
dead process restarts, logs are in journalctl -u factory -f, and an engineer
can psql/redis-cli/curl their way to a fix without a kubectl or docker
login. If you outgrow this, the k8s path is the upgrade — not ECS.
Architecture
Internet
│ 443 (TLS via ACM cert)
▼
┌─────────────────────┐
│ web ALB (public) │ edge SG
└─────────────────────┘
┌──────────────┬───────────────┬────────────────┐
/ws/* ▼ /api/* ▼ / ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ realtime │ │ api │ │ web │ app SG (private subnets)
│ ASG 2..6 │ │ ASG 2..8 │ │ ASG 2..10 │
│ :4001 (ws) │ │ :4000 │ │ :3000 │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
└───────────────┼────────────────┘
│ (data SG: only app→data)
┌─────────────┬──────┴───────┬────────────────┬──────────────┐
▼ ▼ ▼ ▼ ▼
┌────────┐ ┌────────────┐ ┌───────────┐ ┌────────────┐ ┌─────────┐
│ RDS │ │ ElastiCache│ │ S3 bucket │ │ OpenSearch │ │ SSM + │
│ PG 16 │ │ Redis 7 │ │ (uploads) │ │ 3 nodes │ │ KMS │
│ multi-AZ│ │ (AOF) │ │ ver+SSE │ │ VPC │ │ │
└────────┘ └────────────┘ └───────────┘ └────────────┘ └─────────┘
Network: VPC (2 AZs) → public subnets (ALB, NAT) + private subnets
(app ASGs + all data services). Private subnets are internet-less;
outbound via NAT gateway. S3 reachable via VPC endpoint.
File map
| File | Contents |
|---|---|
main.tf |
Backend (S3+Dynamo), AWS provider + default tags, locals, bootstrap state resources |
variables.tf |
Region, project, environment, node counts, instance types, CIDRs, DB/Redis/OS sizing, tags |
network.tf |
VPC, 2 public + 2 private subnets, IGW, NAT, route tables, S3 VPC endpoint |
security.tf |
Security groups (edge/app/data), KMS key, S3 bucket policy (TLS-only) |
compute.tf |
IAM role, ALB + listener + target groups, 4 launch templates + ASGs, cloud-init |
data.tf |
RDS PG16 multi-AZ, ElastiCache Redis, S3 uploads (versioned/SSE/lifecycle), OpenSearch 3-node domain |
outputs.tf |
Endpoints, DNS names, suggested .env assembly |
Plan / apply workflow
0. Prerequisites
- AWS CLI + credentials (IAM user/role with EC2/VPC/RDS/ElastiCache/S3/ OpenSearch/KMS/ACM/SSM/DynamoDB permissions for the stack).
- A DNS zone you control (Route53 or elsewhere) for
var.public_url(defaultfactory.example.com— change it). - Terraform ≥ 1.6.
1. Pre-create SSM parameters (app env)
The node cloud-init reads every app variable from SSM Parameter Store at
/${var.project}/${var.environment}/ (same names as .env.example).
Create them before first apply:
ENV=dev # matches var.environment
# Required (create the rest of .env.example the same way):
aws ssm put-parameter --name /factory/$ENV/DB_PASSWORD --type SecureString --value "$(openssl rand -base64 32)" --overwrite
aws ssm put-parameter --name /factory/$ENV/OS_ADMIN_PASSWORD --type SecureString --value "$(openssl rand -base64 32)" --overwrite
aws ssm put-parameter --name /factory/$ENV/JWT_SECRET --type SecureString --value "$(openssl rand -hex 32)" --overwrite
aws ssm put-parameter --name /factory/$ENV/REFRESH_TOKEN_SECRET --type SecureString --value "$(openssl rand -hex 32)" --overwrite
aws ssm put-parameter --name /factory/$ENV/SESSION_SECRET --type SecureString --value "$(openssl rand -hex 32)" --overwrite
aws ssm put-parameter --name /factory/$ENV/MASTER_ENCRYPTION_KEY --type SecureString --value "$(openssl rand -hex 32)" --overwrite
aws ssm put-parameter --name /factory/$ENV/S3_ACCESS_KEY --type SecureString --value "<s3-key>" --overwrite
aws ssm put-parameter --name /factory/$ENV/S3_SECRET_KEY --type SecureString --value "<s3-secret>" --overwrite
# Non-sensitive (create as String or skip — defaults are sane):
aws ssm put-parameter --name /factory/$ENV/S3_REGION --type String --value us-east-1 --overwrite
DB_PASSWORD and OS_ADMIN_PASSWORD are read by Terraform itself
(data "aws_ssm_parameter"), so they must exist before terraform plan.
2. Init + apply
cd infrastructure/terraform
export TF_VAR_aws_region=us-east-1
export TF_VAR_environment=dev # dev | staging | prod
# optionally: TF_VAR_public_url=https://factory.example.com etc.
terraform init # (state backend: S3 — see below for the bootstrap quirk)
terraform plan -out=plan.tfplan
terraform apply plan.tfplan
State backend bootstrap quirk: the S3 state bucket + DynamoDB lock table
are terraform-managed resources, but terraform init needs them to exist
first. First-time setup (one-off):
aws s3api create-bucket --bucket ai-software-factory-tfstate --region us-east-1
aws dynamodb create-table --table-name ai-software-factory-tflock \
--attribute-definitions AttributeName=LockID,AttributeType=S \
--key-schema AttributeName=LockID,KeyType=HASH \
--billing-mode PAY_PER_REQUEST
…then terraform init works and the main.tf resources adopt the existing
bucket/table (they become no-ops). If your org pre-provisions state storage,
comment out those two resources in main.tf.
3. DNS + certificate
terraform output -json acm_validation_records
# → add the CNAME records to your DNS zone, then:
terraform apply # ACM validation completes; ALB listener gets the cert
# → A/ALIAS record: factory.example.com → (terraform output web_alb_dns_name)
4. Verify
curl -fsS https://factory.example.com/api/v1/health/ready
curl -fsS https://factory.example.com/ # web app
# SSH to an app node for first-boot checks:
ssh -i <key> ubuntu@<node-ip> 'journalctl -u factory -f'
5. Day-2 operations
- Deploys: the cloud-init clones the repo and builds on boot. For
rolling updates, replace nodes:
aws autoscaling terminate-instance-in- autoscaling-group --instance-id i-XXX --should-decrement-desired-capacity(or scale down/up). For repeatable images, bake the build into a golden AMI and swap the launch template — the cloud-initgit cloneblock is a convenience for the simple path, not a release pipeline. - Secret rotation: update the SSM parameter, then roll the node group (env is read at boot).
- Scaling: change
node_count(desired) or the ASG min/max incompute.tfand re-apply. The api/worker/realtime groups have no HPA in this path (manual scaling); the k8s path has HPAs.
Cost notes (us-east-1, default variables, rough monthly)
| Resource | Qty | Approx. $/mo (on-demand) |
|---|---|---|
| EC2 (2 web t3.large + 2 api t3.large + 2 worker t3.medium + 2 realtime t3.medium) | 8 | ~$220 |
| EBS (8× 50 GiB gp3, encrypted) | 8 | ~$65 |
RDS db.r6g.large.multi-az (100 GiB gp3, PITR 7d) |
1+1 standby | ~$390 |
ElastiCache cache.t4g.large |
1 | ~$50 |
OpenSearch t3.small.search ×3 + 3×50 GiB gp3 |
3 | ~$110 |
| NAT gateway (2 AZs × 1 used) + data | — | ~$50 |
| ALB (LCU + hourly) | 1 | ~$20 |
| S3 (versioned uploads) | — | usage-based |
| KMS, ACM, SSM, CloudWatch (free tier) | — | ~$10 |
| Total (dev sizing, on-demand) | ~$900-1000 |
Cuts if you need them: dev environment → single-AZ RDS (drop .multi-az),
db.t4g.medium, 1 worker node, Spot for web/realtime, t3.nano search nodes.
What is NOT terraformed
- Kubernetes manifests —
../kubernetes/is the alternative deployment path and is NOT applied by this stack. If you choose the k8s path, you still use this Terraform for the data services (RDS/ElastiCache/S3/OpenSearch/VPC) and point the k8s secrets at these endpoints (see outputs.tfsuggested_env). - CI/CD: the GitHub workflows in
.github/workflows/build/push images and (on the k8s path) deploy; on this path, deploys are manual node replacement (see day-2 operations). - DNS zone records (you own the zone; only ACM validation records are emitted as an output).
- SSM parameter values (created manually — see step 1).
- IAM users/groups/roles for humans (only the EC2 instance role here).
- Backups beyond defaults: RDS snapshots (7d retention) and S3 versioning are on; a cross-region replication policy for S3/RDS is a follow-up task (Phase 15 hardening).
Teardown
terraform destroy # asks per-dependency; add -target if needed
# NOTE: force_destroy is OFF on the uploads bucket — empty it first if the
# bucket predates this stack (it cannot be destroyed with objects).