The Trinity Beast Infrastructure — CloudFormation Recovery Guide

Complete Infrastructure-as-Code Template — One Command to Recreate Everything
May 3, 2026 Region: us-east-2 (Ohio) 126 Resources 12 Parameters

Table of Contents

List of Diagrams

1. What Is This?

The short version: This is your insurance policy. If every single piece of The Trinity Beast Infrastructure were deleted tomorrow — every server, every database, every network — this one YAML file recreates all of it with a single command.

CloudFormation in Plain English

Think of CloudFormation as a blueprint for a building. When a contractor builds a house, they don't wing it — they follow a blueprint that describes every wall, every pipe, every electrical outlet. CloudFormation is the same idea, but for cloud infrastructure.

You write a YAML file that describes what you want: "I need a network, a database, three servers, a load balancer..." and AWS reads that file and builds everything for you. Every time. Exactly the same way. No clicking through consoles, no forgetting a step, no "it worked on my machine."

How It Works Infrastructure as Code
  1. You describe what you want in a YAML file (the "template")
  2. You run one commandaws cloudformation create-stack
  3. AWS builds everything — in the right order, with the right connections
  4. AWS tracks it all as a "stack" — you can update, monitor, or delete everything as a unit

The Trinity Beast template is 126 resources defined in a single file. That's the entire platform — networking, databases, servers, load balancers, DNS, CDN, monitoring, and scheduling. One file. One command. Everything.

Why this matters: Without this template, recreating The Trinity Beast would mean manually clicking through dozens of AWS console screens, remembering exact settings, and hoping you don't miss anything. With this template, it's one command and a 30-minute wait. That's the difference between a weekend of panic and a coffee break.

2. What's In the Template

The template defines 126 AWS resources organized into 9 layers. Each layer builds on the ones before it — you can't have servers without a network, and you can't have a load balancer without servers. CloudFormation handles the ordering automatically.

Total Resources
126
Infrastructure Layers
9
Parameters
12
Deploy Time
~30-45 min

Layer 1: Networking

VPC & Network Infrastructure ~30 Resources

The foundation. Think of the VPC as the building itself — it defines the walls and rooms. Subnets are the rooms. The Internet Gateway is the front door. Security groups are the locks on each room's door.

ResourceDetailsPurpose
VPC10.0.0.0/16 (65,536 IPs)The private network — your building
Public Subnets3 subnets across 3 AZsRooms with windows — internet-facing (load balancers live here)
Private Subnets3 subnets across 3 AZsInterior rooms — no direct internet access (databases, servers live here)
Internet Gateway1 IGW attached to VPCThe front door to the internet
Route Tables4 route tablesHallway signs — tell traffic where to go
Security Groups7 security groupsDoor locks — control who can talk to what
VPC Endpoints10 endpointsPrivate back doors to AWS services (no internet needed)
Flow LogsVPC Flow Logs → CloudWatchSecurity cameras — record all network traffic

Layer 2: IAM (Identity & Access Management)

Roles & Permissions 5 Roles

IAM roles are like employee badges — they define what each service is allowed to do. A server can read from the database but can't delete the network. Each role follows the principle of least privilege: only the permissions needed, nothing more.

RolePurpose
ECS Task Execution RoleLets ECS pull Docker images from ECR and write logs to CloudWatch
ECS Task RoleLets the running containers access Secrets Manager, S3, SES, and other AWS services
EventBridge RoleLets the scheduler (EventBridge) launch ECS tasks for the nightly sync job
Flow Logs RoleLets VPC Flow Logs write to CloudWatch Logs
Lambda Execution RoleLets the receipt Lambda function write logs and access needed services

Layer 3: Data Layer

Databases, Caches & Storage ~15 Resources

Where your data lives. Aurora is the main database (think: filing cabinets). ElastiCache is the speed cache (think: sticky notes on your desk for things you need instantly). ECR stores your Docker images. S3 stores your website files.

ResourceDetailsPurpose
Aurora Serverless v2PostgreSQL 17.7, 1–21 ACU, Optimized I/OMain database — writer + reader instances
ElastiCache for ValkeyValkey 7.3, db.r7g.large, 13 GB, TLSIn-memory cache — sub-millisecond reads
ECR Repositories5 repositoriesDocker image storage (like a private Docker Hub)
S3 Bucketcpmp-ministry-site-east2Website files, CloudFormation templates, backups
Secrets Managertrinity-beast-secretsEncrypted storage for DB passwords, Stripe keys, API keys

Layer 4: Compute

ECS Fargate & Lambda ~20 Resources

The workers. ECS Fargate runs your containers without you managing servers. Think of it as hiring workers and only paying for the hours they work — no building maintenance. Lambda runs small, one-off tasks (like processing a receipt after a purchase).

ResourceDetailsPurpose
ECS Clustertrinity-beast-fargate-clusterThe factory floor — organizes all services
Main Servicetrinity-beast-main-service — 8 vCPU / 32 GBLPO + LRS (SERVER_TYPE: APP_REPORT_SERVER)
Mirror Servicetrinity-beast-mirror-service — 8 vCPU / 32 GBLPO + LRS (SERVER_TYPE: APP_REPORT_SERVER)
LRS Servicetrinity-beast-lrs-service — 8 vCPU / 32 GBLPO + LRS (SERVER_TYPE: APP_REPORT_SERVER)
Webhook Servicetrinity-beast-webhook-service — 8 vCPU / 32 GBWebhook Push delivery engine (SERVER_TYPE: WEBHOOK_SERVER)
Sync Job Task Deftrinity-beast-sync-jobNightly database sync task definition
Lambda Functiontrinity-beast-receipt — Go (provided.al2023)Post-checkout receipt processing
SQS Queuetrinity-beast-queued-usage-logs — StandardDecoupled usage log write pipeline
Lambda Functiontrinity-beast-queued-writer — Go (provided.al2023)SQS consumer — batch-inserts usage logs into Aurora

Layer 5: Load Balancing

ALB & NLB ~12 Resources

Load balancers are traffic cops. They stand at the front door and direct incoming requests to the right server. The ALB handles web traffic (HTTP/HTTPS). The NLB handles UDP traffic (real-time price feeds).

ResourceListenersPurpose
ALB (Trinity-Beast-TCP-ALB)Port 80, 443, 8080, 9090Web traffic — API, LRS reports, HTTPS redirect
NLB (Trinity-Beast-UDP-NLB)UDP 2679, 2680Real-time UDP price feeds
Target Groups4 target groupsRoute traffic to the right ECS service

Listener breakdown: Port 80 redirects to 443 (HTTPS). Port 443 serves the API. Port 8080 routes to the main service. Port 9090 routes to LRS services. Ports 8081 and 9091 serve dedicated health checks (isolated from production traffic). UDP 2679 and 2680 deliver real-time price data. The Webhook service (BeastWebhook) does not use the ALB or NLB — it pushes prices outbound to subscribers via UDP datagrams and signed HTTPS POSTs.

Layer 6: DNS (Route 53)

Domain Names & Records 10+ Records

DNS is the phone book of the internet. When someone types cpmp-site.org, DNS tells their browser which server to talk to. Route 53 manages all of this.

RecordPoints ToPurpose
cpmp-site.orgCloudFrontMain website
www.cpmp-site.orgCloudFrontWWW alias for website
api.cpmp-site.orgALBREST API endpoint
lrs.cpmp-site.orgALBLRS report server
udp.cpmp-site.orgNLBUDP price feed endpoint
MX recordSES inboundEmail receiving
SPF recordTXT recordEmail authentication — "yes, we're allowed to send email"
DMARC recordTXT recordEmail policy — tells receivers how to handle our email
SES verificationTXT/CNAME recordsProves we own the domain for sending email

Layer 7: CDN (CloudFront)

Content Delivery Network 1 Distribution

CloudFront is like having copies of your website in cities around the world. Instead of everyone connecting to Ohio, visitors get served from the nearest location. Faster for them, less load on your servers.

SettingValue
OriginS3 bucket (cpmp-ministry-site-east2)
HTTP → HTTPSAutomatic redirect
TLS VersionTLS 1.2 minimum
CertificateACM certificate in us-east-1 (CloudFront requirement)

Layer 8: Monitoring & Alerts

CloudWatch Alarms & SNS 14 Alarms

Monitoring is your early warning system. CloudWatch watches metrics (CPU usage, error rates, database health) and SNS sends you a text or email when something goes wrong. You find out about problems before your users do.

CategoryAlarmsWhat They Watch
ECS CPUCPU alarms per serviceContainer CPU usage — alerts if consistently high
ECS Service CountRunning task countAlerts if a service has zero running tasks (it's down)
AuroraCPU, connections, storageDatabase health — CPU spikes, connection exhaustion
ElastiCacheCPU, memory, connectionsCache health — memory pressure, connection limits
ALB/NLB5xx errors, unhealthy targetsLoad balancer health — are requests failing?

Alert delivery: SNS topic sends to both email ([email]) and SMS ([phone]). You get notified both ways for every critical alarm.

Layer 9: Scheduling (EventBridge)

Automated Tasks 1 Rule

EventBridge is your cron job in the cloud. It runs tasks on a schedule without you lifting a finger.

RuleScheduleWhat It Does
trinity-beast-nightly-synccron(0 6 * * ? *) = 1:00 AM ESTLaunches the sync job ECS task to synchronize database data nightly

3. Parameters — What You Need to Provide

Parameters are the blanks you fill in when deploying. Think of them as the customization options on an order form. Most have sensible defaults — you only must provide two values.

Only 2 required: DBPassword and SecretValue are the only parameters you absolutely must provide. Everything else has a default that matches the current production setup.

ParameterPurposeDefaultSecret?Required?
DBPassword PostgreSQL master password for Aurora cluster None — you must provide this Yes (NoEcho) YES
SecretValue JSON string for trinity-beast-secrets (DB creds, Stripe keys, API keys) None — you must provide this Yes (NoEcho) YES
DBUsername PostgreSQL master username postgres No No
DBName Aurora database name CPMP_Backend_Aurora No No
DomainName Primary domain name cpmp-site.org No No
ACMCertificateArnEast2 ACM certificate ARN in us-east-2 (for ALB HTTPS) Current production cert ARN No No
ACMCertificateArnEast1 ACM certificate ARN in us-east-1 (for CloudFront) Current production cert ARN No No
AlertEmail Email address for critical alerts [email] No No
AlertSMS Phone number for SMS alerts [phone] No No
SESFromAddress SES sender address for receipt emails CPMP Mission <No-Reply@CPMP-Site.org> No No
LPOImageTag Docker image tag for LPO server latest No No
SyncImageTag Docker image tag for sync job latest No No

About SecretValue: This is a JSON string containing all the secrets your application needs. Format: {"DB_PASSWORD":"...","STRIPE_SECRET_KEY":"...","STRIPE_WEBHOOK_SECRET":"...",...}. Keep this stored securely outside of AWS — in a password manager, encrypted file, or similar. If you lose this, you'll need to regenerate all the API keys and passwords it contains.

4. How Resources Are Wired Together

CloudFormation doesn't just create resources in isolation — it wires them together using two key mechanisms: !Ref (reference another resource's ID) and !GetAtt (get a specific attribute like an endpoint URL). Here's what that looks like in plain English.

The Big Picture

Diagram 4.1 — Resource Wiring Overview
graph TB
    Internet["Internet"]

    subgraph DNS_CDN["DNS & CDN"]
        Route53["Route 53
cpmp-site.org"] CloudFront["CloudFront
Static Website"] S3["S3 Bucket
trinity-beast-website-east2"] end subgraph LoadBalancers["Load Balancers"] ALB["ALB
Trinity-Beast-TCP-ALB
TCP: 80, 443 → 8080, 9090"] NLB["NLB
Trinity-Beast-UDP-NLB
UDP: 2679, 2680"] end subgraph ECS["ECS Fargate Cluster — 4 Services"] Main["BeastMain · AZ 2a
APP_REPORT_SERVER
8 vCPU / 32 GB"] Mirror["BeastMirror · AZ 2b
APP_REPORT_SERVER
8 vCPU / 32 GB"] LRS["BeastLRS · AZ 2c
APP_REPORT_SERVER
8 vCPU / 32 GB"] Webhook["BeastWebhook
WEBHOOK_SERVER
8 vCPU / 32 GB"] endB"] end subgraph Data["Data Layer"] Aurora["Aurora PostgreSQL
Serverless v2
2–18 ACU"] MemDB["ElastiCache
Valkey 7.2
cache.r7g.2xlarge · 52 GB"] Secrets["Secrets Manager
trinity-beast-secrets"] end Lambda["Lambda
trinity-beast-receipt
Go / provided.al2023"] EventBridge["EventBridge
Nightly Sync
1 AM EST"] SyncJob["Sync Job
0.5 vCPU / 1 GB"] SNS["SNS
Critical Alerts
Email + SMS"] CW["CloudWatch
4 Dashboards
14 Alarms"] Internet --> Route53 Route53 -->|"cpmp-site.org"| CloudFront Route53 -->|"api / lrs"| ALB Route53 -->|"udp"| NLB CloudFront --> S3 Internet -->|"Stripe Webhook"| Lambda ALB -->|"TCP 8080"| Main ALB -->|"TCP 8080"| Mirror ALB -->|"TCP 9090"| LRS NLB -->|"UDP 2679"| Main NLB -->|"UDP 2679"| Mirror NLB -->|"UDP 2680"| LRS Main --> Aurora Main --> MemDB Mirror --> Aurora Mirror --> MemDB LRS --> Aurora LRS --> MemDB Webhook --> Aurora Webhook --> MemDB Main -.->|"!GetAtt Endpoint"| Aurora Main -.->|"!GetAtt Endpoint"| MemDB Main -.->|"Reads"| Secrets Lambda --> Aurora Lambda --> MemDB Lambda -.->|"Reads"| Secrets EventBridge -->|"cron 0 6 * * ?"| SyncJob SyncJob --> Aurora SyncJob --> MemDB CW -.->|"ALARM"| SNS %% Internet → DNS — white linkStyle 0 stroke:#e2e8f0,stroke-width:2px %% DNS → CDN — pink (website path) linkStyle 1 stroke:#f472b6,stroke-width:2px %% DNS → ALB — blue (TCP) linkStyle 2 stroke:#60a5fa,stroke-width:2px %% DNS → NLB — orange (UDP) linkStyle 3 stroke:#FF9900,stroke-width:2px %% CloudFront → S3 — pink linkStyle 4 stroke:#f472b6,stroke-width:2px %% Internet → Lambda — violet (Stripe) linkStyle 5 stroke:#a78bfa,stroke-width:2px %% ALB → ECS — blue (TCP) linkStyle 6 stroke:#60a5fa,stroke-width:2px linkStyle 7 stroke:#60a5fa,stroke-width:2px linkStyle 8 stroke:#60a5fa,stroke-width:2px %% NLB → ECS — orange (UDP) linkStyle 9 stroke:#FF9900,stroke-width:2px linkStyle 10 stroke:#FF9900,stroke-width:2px linkStyle 11 stroke:#FF9900,stroke-width:2px %% ECS → Aurora — red (database) linkStyle 12 stroke:#f87171,stroke-width:2px linkStyle 14 stroke:#f87171,stroke-width:2px linkStyle 16 stroke:#f87171,stroke-width:2px linkStyle 18 stroke:#f87171,stroke-width:2px %% ECS → ElastiCache — green (cache) linkStyle 13 stroke:#10b981,stroke-width:2px linkStyle 15 stroke:#10b981,stroke-width:2px linkStyle 17 stroke:#10b981,stroke-width:2px linkStyle 19 stroke:#10b981,stroke-width:2px %% !GetAtt / !Ref wiring — cyan (dashed) linkStyle 20 stroke:#22d3ee,stroke-width:1.5px linkStyle 21 stroke:#22d3ee,stroke-width:1.5px %% Secrets reads — rose linkStyle 22 stroke:#fca5a5,stroke-width:1.5px %% Lambda → Aurora — red linkStyle 23 stroke:#f87171,stroke-width:2px %% Lambda → ElastiCache — green linkStyle 24 stroke:#10b981,stroke-width:2px %% Lambda → Secrets — rose linkStyle 25 stroke:#fca5a5,stroke-width:1.5px %% EventBridge → SyncJob — yellow linkStyle 26 stroke:#facc15,stroke-width:2px %% SyncJob → Aurora — red linkStyle 27 stroke:#f87171,stroke-width:2px %% SyncJob → ElastiCache — green linkStyle 28 stroke:#10b981,stroke-width:2px %% CloudWatch → SNS — rose (alerts) linkStyle 29 stroke:#fca5a5,stroke-width:1.5px style Internet fill:#FF9900,color:#0f172a,stroke:#FF9900 style ALB fill:#1e293b,stroke:#60a5fa,color:#e2e8f0 style NLB fill:#1e293b,stroke:#FF9900,color:#e2e8f0 style Main fill:#064e3b,stroke:#10b981,color:#e2e8f0 style Mirror fill:#064e3b,stroke:#10b981,color:#e2e8f0 style LRS fill:#064e3b,stroke:#10b981,color:#e2e8f0 style Webhook fill:#064e3b,stroke:#10b981,color:#e2e8f0 style Aurora fill:#1e293b,stroke:#f87171,color:#e2e8f0 style MemDB fill:#1e293b,stroke:#10b981,color:#e2e8f0 style Lambda fill:#1e293b,stroke:#a78bfa,color:#e2e8f0 style CloudFront fill:#1e293b,stroke:#f472b6,color:#e2e8f0 style S3 fill:#1e293b,stroke:#f472b6,color:#e2e8f0 style Route53 fill:#1e293b,stroke:#e2e8f0,color:#e2e8f0 style Secrets fill:#7f1d1d,stroke:#fca5a5,color:#e2e8f0 style SNS fill:#7f1d1d,stroke:#fca5a5,color:#e2e8f0 style CW fill:#1e293b,stroke:#60a5fa,color:#e2e8f0 style SyncJob fill:#1e293b,stroke:#facc15,color:#e2e8f0 style EventBridge fill:#1e293b,stroke:#facc15,color:#e2e8f0
TCP Path (ALB) UDP Path (NLB) Website / CDN Lambda (Stripe) Aurora (Database) ElastiCache (Cache) !GetAtt / !Ref Wiring Secrets / Alerts EventBridge → Sync Job

Key Connections (What !Ref and !GetAtt Do)

Database → ECS Tasks

The Aurora cluster endpoint is automatically passed to all 4 ECS task definitions as the DB_HOST environment variable. When CloudFormation creates the Aurora cluster, it gets an endpoint like trinity-beast-aurora.cluster-xxxxx.us-east-2.rds.amazonaws.com. That endpoint is injected into every container so they know where the database is — no hardcoding needed.

Cache → ECS Tasks

The ElastiCache cluster endpoint is passed to all 4 ECS task definitions as the CACHE_URL environment variable. Same idea — CloudFormation creates the cache, gets the endpoint, and passes it to the containers automatically.

Security Groups → Each Other

Security groups reference each other to create a chain of trust:

  • The ALB security group allows traffic from the internet (ports 80, 443)
  • The ECS tasks security group allows traffic only from the ALB security group — not from the internet directly
  • The Aurora security group allows traffic only from the ECS tasks security group — only your servers can reach the database
  • The ElastiCache security group allows traffic only from the ECS tasks security group — same protection for the cache

This is defense in depth. Even if someone bypasses the load balancer, they can't reach the database directly because the security group only trusts traffic from the ECS containers.

DNS → Load Balancers → Services

Route 53 records use !GetAtt to point at the load balancers:

  • api.cpmp-site.org → ALB's DNS name (automatically resolved)
  • udp.cpmp-site.org → NLB's DNS name
  • cpmp-site.org → CloudFront distribution domain

If the load balancer's address changes (e.g., after a rebuild), the DNS records update automatically because they reference the resource, not a hardcoded IP.

Secrets Manager → ECS Tasks

The Secrets Manager secret ARN is passed to ECS task definitions. At runtime, ECS pulls the secret values and injects them as environment variables. Your application code never sees the raw secret — it just reads environment variables like DB_PASSWORD and STRIPE_SECRET_KEY.

5. How to Deploy

Deploying the stack means telling CloudFormation: "Build everything described in this template." Here's how, step by step.

Prerequisites

Before You Start Checklist
  1. AWS CLI installed and configured — run aws sts get-caller-identity to verify you're authenticated to account 211998422884
  2. ACM certificates — you need two SSL certificates already created and validated:
    • One in us-east-2 (for the ALB)
    • One in us-east-1 (for CloudFront — this is an AWS requirement, CloudFront only uses us-east-1 certs)
  3. Docker images in ECR — the template references ECR repos, but you need to push images to them after the repos are created (or have them ready to push)
  4. The secret values — your database password and the JSON secrets string (Stripe keys, etc.)

The Deploy Command

Important: Replace the placeholder values below with your actual secrets. Never commit real secrets to version control.

aws cloudformation create-stack \
  --stack-name trinity-beast-stack \
  --template-body file://trinity-beast-stack.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-2 \
  --parameters \
    ParameterKey=DBPassword,ParameterValue='YOUR_DB_PASSWORD_HERE' \
    ParameterKey=SecretValue,ParameterValue='{"DB_PASSWORD":"...","STRIPE_SECRET_KEY":"...","STRIPE_WEBHOOK_SECRET":"..."}' \
    ParameterKey=DBUsername,ParameterValue=postgres \
    ParameterKey=DBName,ParameterValue=CPMP_Backend_Aurora \
    ParameterKey=DomainName,ParameterValue=cpmp-site.org \
    ParameterKey=AlertEmail,ParameterValue='[email]' \
    ParameterKey=AlertSMS,ParameterValue='[phone]' \
    ParameterKey=LPOImageTag,ParameterValue=latest \
    ParameterKey=SyncImageTag,ParameterValue=latest

What's CAPABILITY_NAMED_IAM? This flag tells CloudFormation: "Yes, I know this template creates IAM roles, and I'm okay with that." It's a safety check — AWS wants you to explicitly acknowledge that you're granting permissions.

What Happens During Deployment

Deployment Timeline ~30-45 Minutes

CloudFormation reads the template, figures out the dependency order, and starts creating resources. Here's the rough timeline:

TimeWhat's Happening
0–2 minVPC, subnets, internet gateway, route tables created
2–5 minSecurity groups, VPC endpoints, IAM roles created
5–15 minAurora cluster spinning up (this is the slowest part)
5–15 minElastiCache cluster spinning up (runs in parallel with Aurora)
10–20 minECR repos, S3 bucket, Secrets Manager created
15–25 minECS cluster, task definitions, ALB, NLB created
20–30 minECS services start (4 services — they'll fail health checks until images are pushed)
25–35 minRoute 53 records, CloudFront distribution, CloudWatch alarms
30–45 minStack complete — all 126 resources created

Post-Deployment Steps

After the Stack Is Created Action Required

The stack creates the infrastructure, but some things need to be done manually after:

  1. 1Push Docker images to ECR — your ECS services are running but have no code yet
    # Authenticate Docker to ECR
    aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin 211998422884.dkr.ecr.us-east-2.amazonaws.com
    
    # Tag and push the LPO server image
    docker tag trinity-beast-lpo-server:latest 211998422884.dkr.ecr.us-east-2.amazonaws.com/trinity-beast-lpo-server:latest
    docker push 211998422884.dkr.ecr.us-east-2.amazonaws.com/trinity-beast-lpo-server:latest
    
    # Tag and push the sync job image
    docker tag trinity-beast-sync-job:latest 211998422884.dkr.ecr.us-east-2.amazonaws.com/trinity-beast-sync-job:latest
    docker push 211998422884.dkr.ecr.us-east-2.amazonaws.com/trinity-beast-sync-job:latest
  2. 2Deploy Lambda code — upload the Go binary to the Lambda function
    aws lambda update-function-code \
      --function-name trinity-beast-receipt \
      --zip-file fileb://bootstrap.zip \
      --region us-east-2
  3. 3Upload website to S3
    aws s3 sync ./website/ s3://cpmp-ministry-site-east2/ --delete
  4. 4Invalidate CloudFront cache
    aws cloudfront create-invalidation \
      --distribution-id E110PRKEIYQVLL \
      --paths "/*"
  5. 5Verify DNS resolution
    dig api.cpmp-site.org
    dig udp.cpmp-site.org
    dig cpmp-site.org
    curl -I https://api.cpmp-site.org/health

How to Check Stack Status

# Check overall stack status
aws cloudformation describe-stacks --stack-name trinity-beast-stack --region us-east-2

# Watch events in real-time (useful during creation)
aws cloudformation describe-stack-events \
  --stack-name trinity-beast-stack \
  --region us-east-2 \
  --query 'StackEvents[0:10].[Timestamp,ResourceType,LogicalResourceId,ResourceStatus]' \
  --output table

# List all resources in the stack
aws cloudformation list-stack-resources \
  --stack-name trinity-beast-stack \
  --region us-east-2

Success looks like: "StackStatus": "CREATE_COMPLETE". If you see CREATE_FAILED or ROLLBACK_COMPLETE, check the events for the specific resource that failed — CloudFormation will tell you exactly what went wrong.

6. How to Update

Need to change something? Edit the YAML file and tell CloudFormation to update. It's smart enough to figure out what changed and only touch those resources.

The Update Command

aws cloudformation update-stack \
  --stack-name trinity-beast-stack \
  --template-body file://trinity-beast-stack.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-2 \
  --parameters \
    ParameterKey=DBPassword,UsePreviousValue=true \
    ParameterKey=SecretValue,UsePreviousValue=true

Notice UsePreviousValue=true — for secret parameters, you don't need to re-enter them on every update. CloudFormation remembers the values from the last deployment.

How Updates Work

CloudFormation Change Detection

CloudFormation compares your new template against the current state and categorizes each change:

Change TypeWhat HappensExample
No Interruption Resource is updated in place — no downtime Changing a CloudWatch alarm threshold
Some Interruption Resource is briefly interrupted during update Changing an ECS task definition (rolling deploy)
Replacement Old resource is deleted and a new one is created Changing VPC CIDR block, Aurora engine version

⚠️ Replacement Warning: Some changes force CloudFormation to delete and recreate a resource. This can cause data loss. Before updating, always preview changes with a Change Set:

aws cloudformation create-change-set \
  --stack-name trinity-beast-stack \
  --template-body file://trinity-beast-stack.yaml \
  --change-set-name my-changes \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-2

# Review what will change
aws cloudformation describe-change-set \
  --stack-name trinity-beast-stack \
  --change-set-name my-changes \
  --region us-east-2

This shows you exactly what will be modified, added, or replaced — before anything happens. Always use Change Sets for production updates.

Common Safe Updates

Dangerous Updates (Require Replacement)

7. How to Recover from Disaster

The scenario: everything is gone. The AWS account was compromised, someone deleted the stack, or you need to rebuild from scratch in a new account. Here's the step-by-step playbook.

Estimated recovery time: 1–2 hours from "everything is gone" to "everything is running." The stack itself takes 30–45 minutes. The rest is pushing code and verifying.

Step-by-Step Recovery Playbook Disaster Recovery

1 Run create-stack with the template

This is the big one. One command creates all 126 resources:

aws cloudformation create-stack \
  --stack-name trinity-beast-stack \
  --template-body file://trinity-beast-stack.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-2 \
  --parameters \
    ParameterKey=DBPassword,ParameterValue='YOUR_DB_PASSWORD' \
    ParameterKey=SecretValue,ParameterValue='YOUR_SECRETS_JSON'

Then wait. Monitor progress:

aws cloudformation describe-stacks --stack-name trinity-beast-stack --region us-east-2 \
  --query 'Stacks[0].StackStatus' --output text

2 Wait for stack to complete

Watch for CREATE_COMPLETE. This takes 30–45 minutes. Aurora and ElastiCache are the slowest. Go get coffee.

3 Push Docker images to the new ECR repos

The ECR repos are empty — they're just containers waiting for images. Build and push:

# Authenticate
aws ecr get-login-password --region us-east-2 | \
  docker login --username AWS --password-stdin \
  211998422884.dkr.ecr.us-east-2.amazonaws.com

# Build and push LPO server
docker build -t trinity-beast-lpo-server:latest .
docker tag trinity-beast-lpo-server:latest \
  211998422884.dkr.ecr.us-east-2.amazonaws.com/trinity-beast-lpo-server:latest
docker push 211998422884.dkr.ecr.us-east-2.amazonaws.com/trinity-beast-lpo-server:latest

# Build and push sync job
docker build -t trinity-beast-sync-job:latest -f Dockerfile.sync .
docker tag trinity-beast-sync-job:latest \
  211998422884.dkr.ecr.us-east-2.amazonaws.com/trinity-beast-sync-job:latest
docker push 211998422884.dkr.ecr.us-east-2.amazonaws.com/trinity-beast-sync-job:latest

ECS will automatically pick up the new images and start the services.

4 Deploy Lambda code

# Build the Go binary for Lambda
GOOS=linux GOARCH=arm64 go build -o bootstrap cmd/receipt/main.go
zip bootstrap.zip bootstrap

# Deploy to Lambda
aws lambda update-function-code \
  --function-name trinity-beast-receipt \
  --zip-file fileb://bootstrap.zip \
  --region us-east-2

5 Upload website files to S3

aws s3 sync ./cpmp-redesign/ s3://cpmp-ministry-site-east2/ --delete

6 Invalidate CloudFront

aws cloudfront create-invalidation \
  --distribution-id E110PRKEIYQVLL \
  --paths "/*"

7 Update Route 53 nameservers at domain registrar

If the hosted zone was recreated, it gets new nameservers. You must update them at your domain registrar (wherever you bought cpmp-site.org).

# Get the new nameservers
aws route53 get-hosted-zone --id YOUR_HOSTED_ZONE_ID \
  --query 'DelegationSet.NameServers' --output text

Copy those 4 nameserver values and update them at your registrar. DNS propagation can take up to 48 hours, but usually completes within 1–2 hours.

8 Verify all services are healthy

# Check ECS services
aws ecs describe-services \
  --cluster trinity-beast-fargate-cluster \
  --services trinity-beast-main-service trinity-beast-mirror-service trinity-beast-lrs-service trinity-beast-webhook-service \
  --region us-east-2 \
  --query 'services[].{name:serviceName,running:runningCount,desired:desiredCount,status:status}'

# Check API health
curl -s https://api.cpmp-site.org/health | jq .

# Check ALB target health
aws elbv2 describe-target-health \
  --target-group-arn YOUR_TARGET_GROUP_ARN \
  --region us-east-2

# Check Aurora
aws rds describe-db-clusters \
  --db-cluster-identifier trinity-beast-aurora \
  --region us-east-2 \
  --query 'DBClusters[0].Status'

# Check ElastiCache
aws elasticache describe-cache-clusters \
  --cluster-name trinity-beast-cache \
  --region us-east-2 \
  --query 'Clusters[0].Status'

Database note: The stack creates an empty Aurora database. You'll need to run your database migrations to recreate the schema and seed data. If you have a database backup (RDS snapshot), you can restore from that instead — but that's a manual step outside the CloudFormation template.

8. What's NOT in the Template

Honesty time. The CloudFormation template is powerful, but it doesn't cover everything. Some things require manual steps, either because AWS doesn't support them in CloudFormation or because they involve external services.

These items require manual action after the stack is deployed. The template creates the infrastructure, but these pieces must be configured separately.

ItemWhy It's Not in the TemplateWhat You Need to Do
ACM Certificates Certificates require DNS validation — a chicken-and-egg problem (you need DNS to validate, but DNS is in the template) Request certificates in ACM for us-east-2 (ALB) and us-east-1 (CloudFront). Validate via DNS. Pass the ARNs as parameters.
SES Domain Verification & DKIM SES verification involves external DNS records and waiting for AWS to verify Verify cpmp-site.org in SES. Set up DKIM signing. Move out of SES sandbox if needed.
Stripe Webhook Configuration Stripe is an external service — CloudFormation can't configure it Log into Stripe Dashboard. Create webhook endpoint pointing to https://api.cpmp-site.org/webhook/stripe. Copy the webhook secret into your SecretValue parameter.
Database Schema & Seed Data CloudFormation creates the database engine, not the tables inside it After Aurora is up, connect and run your migrations: go run cmd/migrate/main.go. Seed any required reference data.
Docker Images The template creates ECR repos (the shelves) but not the images (the books) Build your Docker images locally or in CI/CD, then push to ECR. See Section 7, Step 3.
Website Content The template creates the S3 bucket (the filing cabinet) but not the files inside it Upload your website files: aws s3 sync ./cpmp-redesign/ s3://cpmp-ministry-site-east2/
Compute Savings Plan Savings Plans are billing commitments, not infrastructure — they can't be defined in CloudFormation Purchase a Compute Savings Plan through the AWS Cost Explorer console after your infrastructure is stable.
Actual Secret Values Secrets should never be stored in a template file — that would be a security risk You provide secrets as parameters at deploy time. Store them securely in a password manager or encrypted vault.

Think of it this way: The template builds the house — walls, plumbing, electrical, locks on the doors. But you still need to move in your furniture (Docker images), hang your pictures (website content), set up your mail forwarding (SES), and give the locksmith your key preferences (secrets). The house is ready, but it needs to be lived in.

9. Template Location

The CloudFormation template and supporting files are stored in multiple locations for redundancy.

Primary Locations

Local Repository Source of Truth
trinity-beast-lpo-server/deployments/cloudformation/trinity-beast-stack.yaml

This is the canonical version. All edits should be made here and then synced to S3.

S3 Backup Cloud Copy
s3://cpmp-ministry-site-east2/cloudformation/trinity-beast-stack.yaml

A copy in S3 for redundancy. You can deploy directly from S3 using --template-url instead of --template-body:

aws cloudformation create-stack \
  --stack-name trinity-beast-stack \
  --template-url https://cpmp-ministry-site-east2.s3.us-east-2.amazonaws.com/cloudformation/trinity-beast-stack.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-2 \
  --parameters ...
Resource Inventory 72 JSON Files
trinity-beast-lpo-server/deployments/cloudformation/inventory/

This directory contains 72 JSON files — one for every resource's current configuration as captured from the live AWS environment. These serve as a reference if you need to verify that the CloudFormation template matches what's actually deployed. They're snapshots, not live data.

Keep these in sync: When you update the template locally, remember to upload the new version to S3: aws s3 cp trinity-beast-stack.yaml s3://cpmp-ministry-site-east2/cloudformation/