The Trinity Beast Infrastructure — CloudWatch Dashboard & Alarm Notifications

Shows a per-service cost breakdown including ECS Fargate, Aurora Serverless, ElastiCache, Lambda, S3, CloudWatch, NAT Gateway, and data transfer. Each service is displayed with daily and monthly cost trends, making it easy to identify which component is driving spend.

Provides a high-level monthly cost summary with total spend, month-over-month trends, and projected costs. Designed for stakeholders who need a quick financial snapshot without per-service granularity.

Metrics published by the Live Price Oracle service:

Metrics published by the Live Report Service:

Alarms: Trinity-Beast-ALB-UnhealthyTargets, Trinity-Beast-NLB-UnhealthyTargets

1. Check ECS service health in the console — are tasks running or in a crash loop?
2. Review container logs in /aws/ecs/trinity-beast for startup errors or OOM kills
3. Verify target group health check path and expected response code
4. Check if a recent deployment introduced a breaking change
5. If tasks are running but unhealthy, check application health endpoint directly

Alarms: Trinity-Beast-ECS-CPU-High, ECS-CPU-High-Mirror, ECS-CPU-High-LRS

1. Check for a traffic spike — correlate with LPO/LRS request metrics on the Application Dashboard
2. Consider scaling the service — increase desired task count or adjust auto-scaling thresholds
3. Check for runaway goroutines or infinite loops in recent deployments
4. Review Container Insights for per-task CPU breakdown
5. If sustained, evaluate whether the task CPU allocation (vCPU) needs to be increased

Alarms: Trinity-Beast-Main-Service-Count-Low, Mirror-Service-Count-Low, LRS-Service-Count-Low

1. Check ECS service events for task stopped reasons (OOM, exit code, health check failure)
2. Review container logs for the last running task — look for panic, fatal, or OOM messages
3. Check if the ECR image exists and is pullable (image pull failures)
4. Verify the task execution role has required permissions
5. Manually start a new task if the service is not recovering automatically

Alarm: Trinity-Beast-Aurora-CPU-High

1. Check for slow queries — use Performance Insights or pg_stat_statements
2. Verify ACU scaling — is the cluster at max ACU and still under pressure?
3. Check if the nightly sync job is running and creating batch write pressure
4. Look for missing indexes on frequently queried columns
5. Consider increasing the max ACU limit if load is legitimate

Alarm: Trinity-Beast-Aurora-Connections-High

1. Check connection pool settings in the application — are pools sized correctly?
2. Look for connection leaks — connections opened but never returned to the pool
3. Verify that the sync job and Lambda are not opening excessive connections
4. Consider using RDS Proxy if connection pressure is persistent
5. Check if a recent deployment changed pool configuration

Alarms: Trinity-Beast-ElastiCache-CPU-High, ElastiCache-Memory-High, ElastiCache-Connections-High

1. Check for a cache stampede — many cache misses causing simultaneous upstream fetches
2. Review key eviction metrics — if memory is full, keys are being evicted prematurely
3. Check connection pool settings in the LPO service — are connections being reused properly?
4. Look for large keys or hot keys that may be causing uneven load
5. If memory is consistently high, consider scaling to a larger node type or adding shards
6. Review TTL settings — are cached items living too long and consuming memory?

Alarm: Trinity-Beast-S3-Size-Unusual-Growth

1. Check for unexpected uploads — review S3 access logs or CloudTrail for PutObject events
2. Look for log file accumulation — are old log exports or reports piling up?
3. Verify lifecycle policies are in place to expire or transition old objects
4. Check if the LRS report output is being stored without cleanup
5. Review bucket versioning — old versions may be consuming space