Warren EC2 Capacity Report

Incident analysis — August 31, 2026. All timestamps UTC unless noted.

1. Memory Evidence

Cgroup readings (raw, from kernel)

Critical memory pressure events (last 24h)

The gateway's diagnostic subsystem fired critical memory pressure alerts throughout the day. These are the 10 highest RSS readings observed:

Note: sampled RSS values are lower than the cgroup peak (6.48 GiB) because diagnostics fire every ~30 seconds. The true peak occurred between samples.

2. Downstream Failures

2a. Silent message loss

When a new message arrives on a session that already has a queued-but-unprocessed message, the gateway prunes the older message to prevent consecutive user turns. No error is returned to the sender. The message disappears without trace.

2b. Miro MCP timeouts

The Miro MCP server requires a 30-second initialization handshake over streamable-http. When the event loop is blocked by memory pressure, the handshake cannot complete.

Network test (curl, bypassing gateway):

Correlation with memory pressure: Every Miro MCP timeout in the last 24 hours occurred during or immediately after a critical memory pressure window. The event loop P99 delay peaked at 75.4 seconds — 2.5× the MCP initialization timeout of 30 seconds.

2c. DM session wedge

At 22:54 UTC, critical memory pressure triggered an internal abort-for-restart. Two active agent runs were killed mid-flight. The Victor DM session (38b68fc1) recovered into a zombie state — permanently queued at q=1, idle, last=reply_operation:ended — where it accepted no new processing. Every subsequent message was pruned as an orphan.

The session remained wedged for 35+ minutes until the diagnostic investigation in this thread began. No self-recovery mechanism fired.

2d. Exec denials correlating with memory pressure

Subagent sessions spawned during the Miro deck work hit exec denied: host=gateway security=deny errors. These cluster tightly with critical memory pressure events:

14 of 15 exec denials occurred within 2 minutes of a critical memory pressure event. The tightest correlation: 01:28:49 deny against 01:28:51 critical — a 2-second gap.

Caveats: The 00:52 outlier sits 32 minutes from the nearest critical event and may not be pressure-related. The 23:01–23:32 window shows 7 consecutive critical events with zero exec denials — this likely reflects no exec attempts during that period (the diagnostic investigation was running from a different session with exec enabled), not a return to health.

Why this matters: This symptom cost the most investigation time. Warren observed exec denials on subagent sessions, concluded the sessions had lost tool policy, and proposed workarounds for a problem that didn’t exist. The subagents were spawned because memory pressure made the Miro MCP unavailable from the main session. The exec denials were legitimate policy restrictions on those subagents — but the reason the subagents existed at all was the cascading failure from memory pressure. The same misdiagnosis pattern as Miro and GWS: a real symptom attributed to the wrong cause, sending investigation down the wrong path.

3. Event Loop Impact

The Node.js event loop — which services all gateway operations including Slack socket mode, MCP handshakes, SQLite transactions, and model response streaming — showed sustained degradation during memory pressure windows:

4. Unrelated: Google Workspace

Unrelated Fixed

GWS API calls failed on Aug 31 morning because the agent called gws (OpenClaw plugin CLI, no OAuth configured) instead of ~/bin/gws-sa (service account wrapper installed by Victor on Aug 28). The service account credentials, token refresh script, and API access all work correctly. A standing rule was added to AGENTS.md to prevent recurrence. Not a capacity issue.

5. System Context