Document the 2026-03-22 multi-agent cascading timeout incident, root cause
analysis, and all new/changed defaults. Update env vars table.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Emergency fix for multi-agent (ClawTeam) burst scenario that caused cascading
failures: when multiple Opus agents ran concurrently, the old consecutive-count
breaker (threshold=3) tripped within seconds, blocking ALL subsequent requests
globally — including non-ClawTeam agents and new sessions.
Changes:
- Circuit breaker now uses a sliding time window (default 5min) instead of
consecutive failure count. 6 failures in 5min triggers open, vs old 3-in-a-row.
- Half-open state allows 2 concurrent probe requests (was 1), improving recovery
speed when API comes back.
- Graduated backoff: cooldown doubles on each re-open (120s→240s→300s cap),
resets fully on first success.
- Increased default timeouts: Opus first-byte 60s→90s, Sonnet 45s→60s,
overall 120s→300s, max concurrent 5→8.
- /health endpoint now exposes per-model breaker state, window stats, and
marks status as "degraded" when any breaker is open.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Circuit breaker: consecutive timeouts (default 3) mark model as degraded for 60s,
failing fast instead of blocking gateway with repeated timeout attempts
- Per-model timeout tiers: Opus 60s base, Sonnet 45s, Haiku 30s, plus adaptive
scaling by prompt size (~15s/100k chars for Opus)
- Structured JSON logging for spawns, timeouts, breaker state changes
- Late close guard: prevents double-settle when timeout fires before proc.close
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Reduced default CLAUDE_TIMEOUT from 300s to 120s for faster fallback
- Added CLAUDE_FIRST_BYTE_TIMEOUT (default 30s): aborts early if Claude
CLI produces no output, preventing silent hangs
- First-byte timing logged for every request for observability
- Health endpoint now reports firstByteTimeout in config
- Default timeout: 300s → 120s (CLAUDE_TIMEOUT)
- New: first-byte timeout at 30s (CLAUDE_FIRST_BYTE_TIMEOUT) — aborts early
if Claude CLI produces no stdout, triggering faster fallback
- Pool disabled by default (POOL_SIZE=0) — on-demand spawning is now the
default; set CLAUDE_POOL_SIZE>0 to re-enable
- Health endpoint now reports timeout and firstByteTimeout values
- Startup log shows pool mode (disabled/on-demand vs active)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Major architectural upgrade:
- Replace pool system with on-demand spawning (eliminates crash loops,
DEGRADED states, and stdin timeout errors from v1.x)
- Add session management with --resume support (reduces token waste on
multi-turn conversations)
- Expand default allowed tools (Bash, Read, Write, Edit, Glob, Grep,
WebSearch, WebFetch, Agent) with configurable CLAUDE_ALLOWED_TOOLS
- Add system prompt pass-through (CLAUDE_SYSTEM_PROMPT)
- Add MCP config support (CLAUDE_MCP_CONFIG)
- Add concurrency control (CLAUDE_MAX_CONCURRENT, default 5)
- Add periodic auth health monitoring
- Add session API endpoints (GET/DELETE /sessions)
- Improve /health with full diagnostics (stats, sessions, errors, config)
- Fix timeout race condition (graceful SIGTERM → SIGKILL)
- Fix ERR_HTTP_HEADERS_SENT by checking headersSent in all response helpers
- Document coexistence with Claude Code interactive mode (Telegram, IDE)
- No conflict with CC: different ports, protocols, and process models
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The proxy previously defaulted to bare "claude" command which fails in
macOS LaunchAgent where PATH is minimal. Now resolves the binary at
startup via: CLAUDE_BIN env → well-known paths → which fallback.
Fails fast with clear error if binary cannot be found.
Also enhances /health to report claudeBinary and claudeBinaryOk status.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add optional PROXY_API_KEY env var for Bearer token auth
- Return 401 for missing/invalid token on all endpoints except /health
- Skip auth when PROXY_API_KEY is not set (backwards compatible)
- Log auth status on startup
- Rework replenishPool to accept isCrash flag: initial fills spawn
immediately with no delay; only crash-triggered respawns apply backoff
- Exponential backoff on crash: 2s, 4s, 8s, 16s, 32s, capped at 60s
- After 5 consecutive fast crashes (< 10s lived) within a 60s rolling
window, mark model as "degraded" and stop all respawning to prevent
CPU spin loops
- Degraded state resets automatically if a subsequent spawn lives > 10s
- stderr from crashed pool workers is captured and logged on exit
- Bump version to 1.6.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- setup.mjs: install launchd plist (macOS) or systemd user service (Linux) after proxy starts; logs to ~/.openclaw/logs/proxy.log
- Add uninstall.mjs to stop and remove the auto-start entry
- README: Quick Start (Node.js) first, Security section, Docker moved to Server/Advanced
- Bump version to 1.5.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The v1.3.0 process pool refactor changed --allowedTools to --tools ""
which disabled all CLI tools. This caused agents to output raw <tool_use>
XML as text instead of executing tools. Restored --allowedTools with
Bash, Read, Write, Edit, Glob, Grep in both spawnWarm() and callClaude()
cold-start paths. Bumped version to 1.3.1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Changes since 1.0.0:
- SSE streaming support for OpenClaw compatibility
- --allowedTools flag for non-interactive tool approval
- Recovery and troubleshooting docs in README
- setup.mjs auto-configuration script
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
claude -p in non-interactive mode cannot show approval prompts,
so tools like gh/git were blocked. Add --allowedTools to enable
tool execution without manual approval.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Routes OpenClaw requests through `claude -p` CLI, letting you use
Claude Pro/Max subscriptions as a model provider without API keys.
- SSE streaming + non-streaming responses
- Auto-setup script for OpenClaw configuration
- Supports Opus 4.6, Sonnet 4.6, Haiku 4
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
taodengandClaude Opus 4.6