Incident analysis — August 31, 2026. All timestamps UTC unless noted.
When memory fills up, work goes into a queue. Warren gets slow.
Some tasks need an external connection — the Miro MCP server, for instance, needs a handshake to complete within 30 seconds. Under memory pressure the event loop stalls, sometimes for over a minute. The handshake never finishes and times out.
Warren then reports that he’s lost access to Miro.
He hasn’t. The OAuth token is valid, the server is healthy, all 57 tools are available. Network latency to the endpoint is under half a second. He simply couldn’t complete the handshake in time.
The same thing happened this morning with Google Workspace: reported as missing credentials, when the credentials were intact and untouched since installation.
The consequence isn’t just that Warren is slow. It’s that he diagnoses incorrectly.
He reports problems that don’t exist. Anyone investigating then goes looking at credentials, permissions, and integrations that were never broken — while the real cause is that the box ran out of room to work.
Meanwhile, 96 user messages were silently discarded across 15 sessions in 24 hours, with no error shown to anyone who sent them.
This setting was tuned for the DGX Spark, where 6 GiB was 5% of 121 GB physical memory — harmless. It was carried over unchanged on Aug 20 when the service unit was copied to EC2. Nobody adjusted it for the new box’s RAM.
On this t3.large, 6 GiB is 79% of 7.6 GB physical. V8 is permitted to grow to 6 GiB before triggering major garbage collection. When it finally collects at that size, the GC pause freezes the Node.js event loop — up to 75 seconds in today’s worst case. That’s 2.5× the 30-second MCP handshake timeout, which is why Miro initialization fails under load.
Removing this line lets the adaptive default take over: 3,909 MiB (50% of physical). V8 would cap at ~3.9 GiB and collect earlier, with shorter pauses. The spikes would still happen but would plateau lower — likely under the 30s MCP timeout instead of over it.
This is testable before any instance resize. It costs nothing, requires only removing line 25 from the service unit and running systemctl --user daemon-reload. The next scheduled restart is at 04:00 PDT.
The memory pattern is spike-and-drop, not leak. The idle floor between active windows is flat at 1.40–1.62 GiB for hours with no growth. Concurrent sessions drive spikes; when they end, RSS falls back to baseline.
Concurrent Opus 4-6/4-8 sessions each allocate 1–2 GiB of heap for context windows, model payloads, and tool schemas. Two concurrent sessions fit. Three push past the pressure threshold. The box has capacity for one heavy session at a time with headroom, but not the concurrent workload it actually runs.
The heap setting is the first thing to fix, but the underlying capacity question remains: is this box large enough for the concurrent workload? Before spending on a resize, the config fix is testable at zero cost and the next restart is already scheduled.
16.3 GB peak / 121 GB available
4 TB disk · GPU compute
21h uptime before peak
6.48 GB peak / 7.6 GB available
No swap · No cgroup limit
1.12 GB to OOM killer at peak
The gateway's diagnostic subsystem fired critical memory pressure alerts throughout the day. These are the 10 highest RSS readings observed:
Note: sampled RSS values are lower than the cgroup peak (6.48 GiB) because diagnostics fire every ~30 seconds. The true peak occurred between samples.
When a new message arrives on a session that already has a queued-but-unprocessed message, the gateway prunes the older message to prevent consecutive user turns. No error is returned to the sender. The message disappears without trace.
| Session ID | Messages Lost |
|---|---|
09f275e7 | 18 |
67218c44 | 16 |
38b68fc1 (Victor DM) | 16 |
886d5327 | 12 |
ef6e7934 | 5 |
c575b9d5 | 5 |
| + 9 more sessions (1–4 each) | |
The Miro MCP server requires a 30-second initialization handshake over streamable-http. When the event loop is blocked by memory pressure, the handshake cannot complete.
Network test (curl, bypassing gateway):
Correlation with memory pressure: Every Miro MCP timeout in the last 24 hours occurred during or immediately after a critical memory pressure window. The event loop P99 delay peaked at 75.4 seconds — 2.5× the MCP initialization timeout of 30 seconds.
At 22:54 UTC, critical memory pressure triggered an internal abort-for-restart. Two active agent runs were killed mid-flight. The Victor DM session (38b68fc1) recovered into a zombie state — permanently queued at q=1, idle, last=reply_operation:ended — where it accepted no new processing. Every subsequent message was pruned as an orphan.
The session remained wedged for 35+ minutes until the diagnostic investigation in this thread began. No self-recovery mechanism fired.
Subagent sessions spawned during the Miro deck work hit exec denied: host=gateway security=deny errors. These cluster tightly with critical memory pressure events:
| Exec Denied | Nearest Critical | Gap |
|---|---|---|
| 19:07:00 | 19:08:31 | 1m 31s |
| 19:09:04 | 19:08:31 | 33s |
| 22:43:54 | 22:45:37 | 1m 43s |
| 22:45:45 | 22:45:37 | 8s |
| 22:56:00 | 22:56:38 | 38s |
| 00:52:52 | 00:20:46 | 32m ← outlier |
| 01:19:19 | 01:17:59 | 1m 20s |
| 01:28:49 | 01:28:51 | 2s |
| 01:34:55 | 01:34:06 | 49s |
| 01:40:29 | 01:39:17 | 1m 12s |
| 01:41:23 | 01:39:17 | 2m 6s |
| 01:43:37 | 01:44:33 | 56s |
| 01:44:30 | 01:44:33 | 3s |
| 01:45:16 | 01:44:33 | 43s |
14 of 15 exec denials occurred within 2 minutes of a critical memory pressure event. The tightest correlation: 01:28:49 deny against 01:28:51 critical — a 2-second gap.
Caveats: The 00:52 outlier sits 32 minutes from the nearest critical event and may not be pressure-related. The 23:01–23:32 window shows 7 consecutive critical events with zero exec denials — this likely reflects no exec attempts during that period (the diagnostic investigation was running from a different session with exec enabled), not a return to health.
Why this matters: This symptom cost the most investigation time. Warren observed exec denials on subagent sessions, concluded the sessions had lost tool policy, and proposed workarounds for a problem that didn’t exist. The subagents were spawned because memory pressure made the Miro MCP unavailable from the main session. The exec denials were legitimate policy restrictions on those subagents — but the reason the subagents existed at all was the cascading failure from memory pressure. The same misdiagnosis pattern as Miro and GWS: a real symptom attributed to the wrong cause, sending investigation down the wrong path.
The Node.js event loop — which services all gateway operations including Slack socket mode, MCP handshakes, SQLite transactions, and model response streaming — showed sustained degradation during memory pressure windows:
| Metric | Normal | Under Pressure |
|---|---|---|
| Event loop P99 delay | <25ms | 75,430ms |
| Event loop utilization | 0.09–0.15 | 1.0 (saturated) |
| CPU core ratio | 0.15 | 1.39 |
| Heartbeat interval | 30s | 97s (delayed 31–33s) |
| SQLite transaction hold | Not logged | Repeated "slow" warnings |
Fixed
GWS API calls failed on Aug 31 morning because the agent called gws (OpenClaw plugin CLI, no OAuth configured) instead of ~/bin/gws-sa (service account wrapper installed by Victor on Aug 28). The service account credentials, token refresh script, and API access all work correctly. A standing rule was added to AGENTS.md to prevent recurrence. Not a capacity issue.
| Property | Value |
|---|---|
| Instance type | t3.large (confirmed from IMDS) |
| vCPU | 2 |
| RAM | 7.6 GiB |
| Swap | None configured |
| Disk | 48 GB |
| OS | Ubuntu 24.04 x86_64 |
| Node.js | v22.23.2 |
| Boot | Aug 20, 15:06 UTC |
| Daily restart | 04:00 PDT via openclaw-backup.service |
| Restart history | NRestarts=0, no StartLimitBurst triggers, clean stop/start every cycle |
| IAM identity | t-and-c-ec2-role (instance role only; no IAM user credentials on disk) |
| Backpressure | None — gateway accepts all inbound work regardless of memory/CPU state |