The Trinity Beast — ElastiCache Operations & Recovery

1. Design Philosophy

Three Rules of Cache Operations

1. The system MUST function without Valkey. The price API — the revenue-generating core — operates entirely from local sync.Map and live exchange WebSocket feeds. Valkey adds speed, never correctness.
2. Everything in Valkey is rebuildable. Every key family has an authoritative source outside Valkey (Aurora, S3, live traffic, or computed on-demand). Running the sync job restores 100% of operational state.
3. Valkey is eager to return, not required to be present. Application code checks for nil client, uses timeouts on every call, and falls through gracefully when unavailable. When Valkey comes back, the system resumes using it immediately — no restart required.

Diagram 1.1: Valkey's Role in TBI Architecture

graph TB
    subgraph "Revenue Path (Zero Valkey Dependency)"
        WS["WebSocket Feeds\n6 Exchanges"] --> SM["sync.Map\nLocal Cache"]
        SM --> API["/price API"]
        API --> Customer
    end

    subgraph "Operational Layer (Valkey-Powered)"
        SM -.->|"flush every 30s"| VK[("Valkey\n52 GB")]
        VK --> LRS[LRS Reports]
        VK --> Search[Full-Text Search]
        VK --> TX[Translation State]
        VK --> HP[Honeypot Queue]
        VK --> AG[Adaptive Governor]
        VK --> CS[Cluster Stats]
    end

    subgraph "Authoritative Sources"
        Aurora[("Aurora PostgreSQL")] -.->|"nightly sync"| VK
        S3[("S3 Bucket")] -.->|"search rebuild"| VK
        Traffic[Live Traffic] -.->|accumulates| VK
    end

    style SM fill:#064e3b,stroke:#10b981,color:#e2e8f0
    style VK fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    style Aurora fill:#451a03,stroke:#f59e0b,color:#e2e8f0
    style API fill:#064e3b,stroke:#10b981,color:#e2e8f0
        

2. Infrastructure Specifications

3. Dependency Classification Map

Every Valkey key family is classified by its role in the system and what happens when it disappears.

Diagram 3.1: Dependency Classification

graph LR
    subgraph "Class A: Performance Acceleration"
        A1["price:*"]
        A2["apikey:*"]
        A3["app:config"]
        A4["report:config"]
        A5["report_count:*"]
        A6["errmsg:*"]
        A7["public:site-assets"]
        A8["tx:params"]
    end

    subgraph "Class B: Operational State"
        B1["usage_logs:* indexes"]
        B2["usage_log:* hashes"]
        B3["search:index:*"]
        B4["report_usage_logs:*"]
        B5["docs:registry:*"]
        B6["report:text:*"]
    end

    subgraph "Class C: Coordination"
        C1["cluster:stats:*"]
        C2["adaptive:*"]
        C3["newsletter:lock:*"]
        C4["digest:lock:*"]
        C5["receipt:session:*"]
    end

    subgraph "Class D: Intelligence"
        D1["honeypot:*"]
        D2["autoops:threats:daily"]
        D3["autoops:support:*"]
        D4["autoops:bedrock:spend:daily"]
    end

    style A1 fill:#064e3b,stroke:#10b981,color:#e2e8f0
    style A2 fill:#064e3b,stroke:#10b981,color:#e2e8f0
    style B1 fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    style B3 fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    style C1 fill:#2e1065,stroke:#a78bfa,color:#e2e8f0
    style D1 fill:#451a03,stroke:#f59e0b,color:#e2e8f0
        

4. Complete Key Inventory

4.1 Class A — Performance Acceleration (Aurora/S3 Backed)

4.2 Class B — Operational State (Sync Job Rebuilt)

4.3 Class C — Coordination (Self-Healing)

4.4 Class D — Intelligence (Accumulates from Live Traffic)

4.5 Translation Engine State (Class A — Aurora Backed)

5. Graceful Degradation Matrix

What happens to each feature when Valkey is unavailable:

6. BeastReconciler — Valkey State Reconciler

BeastReconciler (trinity-beast-sync-job, 1 AM EST nightly) is the single recovery mechanism for Valkey. It is a first-class member of the ECS cluster alongside BeastMain, BeastMirror, BeastLRS, BeastWebhook, and BeastTranslate. After a cold start (new node, failover, or recovery), running BeastReconciler once restores 100% of operational state.

Diagram 6.1: BeastReconciler — Valkey State Reconciler Flow

flowchart TD
    Start([BeastReconciler Start]) --> CheckFirst{"First Run?
usage_logs:index exists?"}

    CheckFirst -->|"Yes - Cold Start"| Full["Full Historical Load
93 days from Aurora"]
    CheckFirst -->|"No - Incremental"| Inc["Incremental Load
since last_timestamp"]

    Full --> Prune["Prune Old Data
remove entries > 93 days"]
    Inc --> Prune

    Prune --> RUL["Sync Report Usage Logs
Aurora to Valkey"]
    RUL --> Keys["Sync API Keys
Aurora to Valkey hashes"]
    Keys --> Params["Sync App Params
Aurora to app:config hash"]
    Params --> ErrMsg["Sync Error Messages
Aurora to errmsg:* strings"]
    ErrMsg --> TxCost["Sync Translation Costs
Calculate averages"]
    TxCost --> TxParams["Sync Translation Params
Aurora to tx:params hash"]

    TxParams --> Rhema["Sync Rhema Knowledge
S3 to autoops:support:knowledge:b64"]
    Rhema --> DocReg["Sync Doc Registry
S3 listing to docs:registry:*"]
    DocReg --> Reports["Generate Report Text
S3 HTML to report:text:*"]
    Reports --> Search["Rebuild Search Index
CloudFront to search:index:*"]

    Search --> Health["Verify Valkey Health
PING + DBSIZE + baseline check"]
    Health --> End([Complete])

    style Full fill:#7f1d1d,stroke:#fca5a5,color:#e2e8f0
    style Inc fill:#064e3b,stroke:#10b981,color:#e2e8f0
    style Rhema fill:#2e1065,stroke:#a78bfa,color:#e2e8f0
    style DocReg fill:#2e1065,stroke:#a78bfa,color:#e2e8f0
    style Health fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
        

6.1 BeastReconciler Responsibilities — Current (12)

6.2 BeastReconciler Responsibilities — New (Recovery Expansion)

6.3 Manual Recovery Commands

For situations where BeastReconciler hasn't run yet, or you need immediate recovery of specific subsystems:

# Trigger BeastReconciler immediately (doesn't wait for 1 AM)
aws ecs run-task --cluster trinity-beast-fargate-cluster \
  --task-definition trinity-beast-sync-job \
  --overrides '{"containerOverrides":[{"name":"sync-container","environment":[{"name":"FORCE_FULL_SYNC","value":"true"}]}]}' \
  --network-configuration '...' --region us-east-2

# Rebuild search index only (fast — ~30 seconds)
bash scripts/kcc.sh build-search

# Rebuild doc registry from S3 (future command)
bash scripts/kcc.sh rebuild-doc-registry

# Verify Valkey health and key count
bash scripts/kcc.sh valkey-health

7. Disaster Recovery Runbook

Diagram 7.1: Cold-Start Recovery Sequence

sequenceDiagram
    participant Op as Operator
    participant ECS as ECS Containers
    participant BR as BeastReconciler
    participant VK as Valkey - New Node
    participant Aurora as Aurora
    participant S3 as S3

    Note over VK: Valkey node replaced - cold start

    Op->>BR: Trigger FORCE_FULL_SYNC=true
    BR->>Aurora: SELECT * FROM usage_logs (93 days)
    BR->>VK: Batch HSET + ZADD (39K+ entries)
    BR->>Aurora: SELECT * FROM api_keys
    BR->>VK: HSET apikey:* (all active keys)
    BR->>Aurora: SELECT * FROM application_parameters
    BR->>VK: HSET app:config
    BR->>Aurora: SELECT * FROM error_messages
    BR->>VK: SET errmsg:*
    BR->>S3: Read rhema/knowledge-base.txt
    BR->>VK: SET autoops:support:knowledge:b64
    BR->>S3: ListObjects docs/
    BR->>VK: SET docs:registry:* + SADD index
    BR->>S3: Read daily-reports/ (30 days)
    BR->>VK: SET report:text:*
    BR->>ECS: POST /admin/build-search-index
    ECS->>VK: SET search:index:* (12 languages)
    BR->>VK: PING + DBSIZE (verify)

    Note over VK: Full operational state restored

    ECS->>VK: Containers auto-resume writes
        

7.1 Scenario: Valkey Node Replacement

When ElastiCache replaces the node (maintenance, failure, or manual action):

1. Immediate effect: All ECS containers detect connection failure. RedisClient calls timeout (3s) and return errors. Application falls through to Aurora/sync.Map for all Class A keys.
2. Auto-recovery (containers): go-redis has MaxRetries: 3 and auto-reconnects. Once the new node is available, existing connections fail but new ones succeed. No container restart needed.
3. Data recovery: Run BeastReconciler with FORCE_FULL_SYNC=true. Duration: ~4 minutes for 39K usage logs + all supporting data.
4. Verification: bash scripts/kcc.sh valkey-health — confirms DBSIZE meets baseline (~42,000 keys).

7.2 Scenario: Valkey Temporarily Unreachable (Network/VPC Issue)

1. During outage: Price API continues normally. LRS reports return 503. Search returns empty or stale cache. Honeypot stops accumulating.
2. On reconnect: Everything auto-resumes. No sync needed — data accumulated during the outage is in Aurora (via SQS → queued-writer). Next nightly sync backfills the gap in Valkey.
3. Optional: If LRS reports are urgently needed, trigger an incremental BeastReconciler run (no FORCE_FULL_SYNC needed — it detects existing high-water mark).

7.3 Scenario: Valkey Data Corruption (Partial Key Loss)

1. Detect: DBSIZE below baseline, or specific subsystem returning unexpected errors.
2. Diagnose: Check which key families are missing: EXISTS usage_logs:index, EXISTS app:config, DBSIZE.
3. Fix: Run targeted recovery or full BeastReconciler run depending on scope. BeastReconciler is idempotent — running it on a partially populated Valkey is safe.

7.4 Recovery Time Objectives

8. Health Monitoring & Alerting

8.1 CloudWatch Alarms (Active)

8.2 Application-Level Health Checks

• LRS Health Endpoint: /lrs/health — calls PING on Valkey. Returns 503 if unreachable.
• Admin Valkey Endpoint: POST /admin/valkey with {"command":"PING"} — direct connectivity test.
• KCC Daily Collect: Reports Valkey item count, hit rate, memory usage, and CPU in the daily dashboard.

8.3 Baseline Metrics (Healthy State)

9. Validation Results — First Production Run

9.1 Test Conditions

• Mode: FORCE_FULL_SYNC=true (simulates cold-start recovery — rebuilds everything from scratch)
• Trigger: Manual ecs run-task with environment override
• Baseline: Valkey already populated (~42,000 keys). Full overwrite validates idempotency.
• Traffic: Pre-production (no customer impact risk)

9.2 Results by Function

9.3 Performance

9.4 Key Observations

• Idempotent by design. Running against an already-populated Valkey produced zero conflicts. The job overwrites cleanly — safe to run at any time, from any state.
• Sub-3-second full recovery. A complete cold start — 40,571 usage logs, 7 API keys, 86 parameters, 252 error messages, 243 translation jobs, 42 doc registry entries, search index trigger, Rhema knowledge base — all reconciled in 2.36 seconds.
• Cost-per-chunk calculation working. After adding the execution_mode column to translation_jobs, the per-model cost averages compute correctly from 7-day actuals with floor/ceiling clamping and 9% infrastructure markup.
• S3 as authoritative source confirmed. Both the Rhema knowledge base and doc registry rebuild successfully from S3 — validating that the recovery path is Aurora + S3, never Valkey itself.

10. Retired Key Families

Key families that are being phased out or have been removed: