The Trinity Beast — Stress Test Plan v5.0

1. Test Philosophy — What Changed

The v4.1 plan had 12 tests across 3 phases — LPO Only, LRS Only, and Combined — each tested across 4 topologies (TCP Direct, TCP ALB, UDP Direct, UDP NLB). That was 12 tests requiring 4 infrastructure reconfigurations and 4.5 hours.

The v5.0 plan eliminates redundancy by recognizing three facts:

• Production always runs APP_REPORT_SERVER — testing LRS in isolation tells you a ceiling you'll never hit in production. The Combined phase already covers LRS under real-world conditions.
• NLB is Layer 4 with near-zero overhead — if you know the single-node UDP ceiling, the NLB ceiling is ~3× that. A separate direct-vs-NLB comparison adds time without insight.
• Public vs Partner is the real distinction — not LPO vs LRS. Public subscribers hit rate limiting, monthly caps, and TLS termination. Partners bypass all of that. These are the two paths worth measuring.

Public vs Partner Access Paths

2. Infrastructure Under Test

3. Progressive Load Ramp (13 Levels)

Each test follows a 13-level progressive load ramp. Concurrency doubles at each level while request volume scales proportionally. This reveals the exact concurrency threshold where performance degrades.

4. Real-Time Telemetry & Cluster Aggregation

Every container runs 24 atomic.Int64 counters that track the complete request lifecycle. The /admin/stress-stats endpoint returns a per-container snapshot. The /admin/cluster-stats endpoint reads all 3 snapshots from ElastiCache in a single pipeline call — one round-trip, sub-millisecond, guaranteed all 3 containers.

graph LR
    subgraph Containers
        M[BeastMain
24 counters] -->|"every 3s"| EC
        Mi[BeastMirror
24 counters] -->|"every 3s"| EC
        L[BeastLRS
24 counters] -->|"every 3s"| EC
    end
    EC[(ElastiCache
cluster:stats:*
TTL 30s)]
    EC -->|"1 pipeline read"| CS[/admin/cluster-stats/]
    CS --> TBCC[Command Center
Cluster Health]
    CS --> CW[CloudWatch
6 Metrics]

    style M fill:#334155,stroke:#64748b,color:#94a3b8
    style Mi fill:#334155,stroke:#64748b,color:#94a3b8
    style L fill:#334155,stroke:#64748b,color:#94a3b8
    style EC fill:#5a3a3a,stroke:#8a5a5a,color:#e2c8c8
    style CS fill:#2d5a4a,stroke:#4a9a7a,color:#cbd5e1
    style TBCC fill:#2d4a6f,stroke:#4a7ab5,color:#cbd5e1
    style CW fill:#3d3a5c,stroke:#6b6399,color:#cbd5e1
        

5. Test Matrix — 7 Tests, 2 Phases

graph TD
    START[Launch Stress Client
96 vCPU] --> P1[Phase 1: Single Container]
    P1 --> T1A[1a: TCP Direct
Raw TCP ceiling]
    P1 --> T1B[1b: UDP Direct
v6 engine ceiling]
    T1A --> RECONFIG[Reconfigure:
Distribute AZs
Enable Governor]
    T1B --> RECONFIG
    RECONFIG --> P2[Phase 2: Production Topology]
    P2 --> T2A[2a: TCP ALB
Public subscribers]
    P2 --> T2B[2b: UDP NLB
Partner — no rate limit]
    P2 --> T2C[2c: UDP NLB
Public — rate limited]
    T2A --> T2D[2d: Combined
LPO + LRS simultaneous]
    T2B --> T2D
    T2C --> T2D
    T2D --> T2E[2e: Endurance
30 min at 80% ceiling]
    T2E --> RESTORE[Restore Production
fresh-price profile]

    style START fill:#4a5568,stroke:#718096,color:#e2e8f0
    style P1 fill:#5a4a2d,stroke:#9a7a4a,color:#cbd5e1
    style P2 fill:#2d4a6f,stroke:#4a7ab5,color:#cbd5e1
    style RECONFIG fill:#3d3a5c,stroke:#6b6399,color:#cbd5e1
    style RESTORE fill:#2d5a4a,stroke:#4a9a7a,color:#cbd5e1
    style T1A fill:#334155,stroke:#64748b,color:#94a3b8
    style T1B fill:#334155,stroke:#64748b,color:#94a3b8
    style T2A fill:#334155,stroke:#64748b,color:#94a3b8
    style T2B fill:#334155,stroke:#64748b,color:#94a3b8
    style T2C fill:#334155,stroke:#64748b,color:#94a3b8
    style T2D fill:#334155,stroke:#64748b,color:#94a3b8
    style T2E fill:#334155,stroke:#64748b,color:#94a3b8
        

6. Phase 1 — Single Container Ceiling

Isolates a single container to find the raw per-node ceiling for each protocol. No load balancer, no TLS termination, no cross-AZ latency. Governor disabled. Stress client connects directly to the container IP. This establishes the baseline that Phase 2 builds on.

Measures the absolute single-container TCP ceiling. Stress client connects directly to the container IP on port 8080, bypassing the ALB. This is the purest throughput test — no load balancer overhead, no TLS termination.

Key metrics to watch: tcp_rps, syncmap_hit_pct (should be 99%+), db_in_use (should stay well below db_max_open), batch_flush_errors (should be 0).

The most anticipated test. v5 peaked at 44K RPS — bottlenecked by json.Marshal and single-socket WriteToUDP serialization. v6 introduces zero-alloc response building and multi-socket architecture. Target: 80K+ RPS.

Key metrics to watch: udp_rps, udp_drop_pct (the critical number — above 1% means workers can't keep up), bg_drop_pct (housekeeping saturation), CPU% in Container Insights.

7. Phase 2 — Production Topology

All 3 containers running APP_REPORT_SERVER (LPO + LRS), distributed across AZs (2a/2b/2c), governor enabled. This is the real-world configuration. Every test in this phase answers a question about how the system performs under production conditions.

Production TCP throughput for public subscribers. Traffic flows through the ALB with TLS termination, rate limiting active, monthly cap checks enforced. This is the number that goes on the marketing page — what a subscriber actually experiences.

Key metrics to watch: total_rps across all 3 nodes (via /admin/cluster-stats), rate_limit_hits (should be 0 with stress tier key), ALB TargetResponseTime p99.

Production UDP throughput for AWS Partners. Partner API keys bypass all rate limiting, monthly caps, and billing checks — both in the TCP and UDP handlers. NLB operates at Layer 4 with near-zero added latency. This measures the fastest possible path through The Trinity Beast.

Key metrics to watch: udp_rps cluster-wide, udp_drop_pct per node, rate_limit_hits (should be exactly 0 — partner keys skip the limiter).

Same infrastructure as 2b, but with a public-tier API key. Rate limiting and monthly cap checks are active on every packet. This measures the exact overhead of rate limiting on the UDP path — compare udp_rps against test 2b to quantify the cost.

Key metrics to watch: udp_rps (compare to 2b), rate_limit_hits (should be 0 with stress-tier QPS), the delta between 2b and 2c reveals the per-packet cost of rate limit checks.

The resource contention test. Two stress clients run simultaneously — one hammering /price (LPO) and one hammering /reports/usage (LRS). Both services compete for the same CPU, memory, DB connections, and cache pool. The gap between this number and test 2a reveals exactly how much LRS overhead costs under load.

Key metrics to watch: tcp_rps + lrs_requests (both should be tracked), db_in_use (combined workload may approach db_max_open=180), batch_flush_errors.

Sustained load at 80% of the ceiling found in test 2a. Runs for 30 continuous minutes. This tests for memory leaks, connection pool exhaustion, goroutine accumulation, cache TTL edge cases, and Aurora ACU scaling behavior over time. If the system is stable at 80% for 30 minutes, it's production-ready.

Key metrics to watch: Memory% trend in Container Insights (should be flat, not climbing), db_wait_count (should stay at 0), Aurora ACU (should stabilize, not keep climbing), errors_5xx (must remain 0 for the full 30 minutes).

8. Application Parameter Profiles

Streamlined from 16 profiles to 8. Each test uses a profile specifically optimized for its protocol and topology. Profiles are applied instantly via /admin/system-mode?mode=<name>.

Key Tuning Principles

• db_max_idle = db_max_open — eliminates connection churn (the fix that dropped p99 from 1,266ms to 8.9ms)
• cache_min_idle ≈ 33% of cache_pool_size — optimal pre-warm ratio
• All pool sizes are multiples of 3 — one pool per container, evenly distributed
• UDP profiles use batch_size=100 — smaller batches reduce GC pressure, freeing CPU for packet processing
• Direct profiles use longer flush intervals — single container handles all writes
• ALB/NLB profiles use smaller pools — load distributed across 3 containers
• Combined profile uses db_max_open=180 — extra headroom for LPO + LRS sharing connections

Profile Matrix

All stress profiles share: QPS=100K, Burst=100K, TTL=300s, log_level=error, config_poll=300, cache_max_retries=1, cache_dial_ms=500, cache_write_ms=500.

9. Race Day Monitoring

Five data sources monitored simultaneously during each test. No X-Ray (adds latency). All monitoring is passive — zero impact on the system under test.

10. Pre-Flight & Post-Test Checklists

Pre-Flight (Before Each Test)

Post-Test (After All Tests Complete)

11. Engine Optimizations (v5 → v6)

The v6 engine addresses every bottleneck identified during v5 testing. These are code-level changes targeting the CPU-bound UDP hot path.

12. Estimated Timeline

Total estimated time: ~2.5 hours including one infrastructure reconfiguration between phases.