mirror of
https://github.com/dtzp555-max/ocp.git
synced 2026-07-21 21:15:09 +00:00
fix: -p spawn-home isolation (③ latency 3x) + concurrency queue/429 (⑥) + ocp restart env + /ocp plugin compat (#144)
* fix(server): isolate default -p spawn in credential-free scratch HOME (latency ③) The default (-p/stream-json) spawn inherited the operator's real HOME (global ~/.claude plugins/skills/hooks) and ran with cwd=~/ocp (project CLAUDE.md/skills), loading heavy host context on EVERY request. Measured: pure API floor for haiku "hi" ≈ 1–2s; same CLI in the operator's real HOME/cwd ≈ 10–28s; a clean minimal HOME + CLAUDE_CODE_OAUTH_TOKEN ≈ 3–7s with auth intact. When an OAuth token is resolvable (and OCP_SPAWN_REAL_HOME!=1), spawnClaudeProcess now runs claude under a credential-free minimal scratch HOME (<HOME>/.ocp/spawn-home, no .credentials.json / settings.json / plugins) with cwd = that neutral dir and the resolved token in CLAUDE_CODE_OAUTH_TOKEN (env token is authoritative for -p). Mirrors the TUI path's resolveTuiHome() env-token mode. Falls back to real HOME + inherited cwd when no token resolves (zero regression); OCP_SPAWN_REAL_HOME=1 is a kill-switch. The token is resolved ONCE (memoized via getSpawnHomeMode, reusing getOAuthCredentials, the same resolver the /usage probe uses) and never logged. Adds a startup log line and an additive /health `spawn` block so the operator can confirm isolation is on. ALIGNMENT.md Rule 2 justification: this is an INFRA / PROCESS-MANAGEMENT change (HOME/cwd/env isolation of the spawned process). It does NOT mirror a cli.js wire operation, introduces no new endpoint/header, and adds no API token to the wire path — so there is no cli.js function to cite. CLAUDE_CODE_OAUTH_TOKEN / CLAUDE_CODE_DISABLE_* are existing claude-CLI env contracts already used by the TUI path and getOAuthCredentials. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(server): bounded wait-queue + HTTP 429 for -p concurrency overflow (⑥) spawnClaudeProcess used `if (activeRequests >= MAX_CONCURRENT) throw` → the client got an opaque 500 AND the rejection was uncounted (a 15-concurrent stress run returned 7×500 while /health stats.errors stayed 0). The TUI path already had a bounded-queue semaphore (TuiSemaphore); the -p path did not. Now the -p path reuses TuiSemaphore as `claudeSemaphore = new TuiSemaphore(MAX_CONCURRENT, { maxQueue: CLAUDE_MAX_QUEUE })`. Requests beyond MAX_CONCURRENT WAIT (up to CLAUDE_MAX_QUEUE, default 16) instead of being rejected; only when the queue is ALSO full does the request get HTTP 429 + Retry-After (rate_limit_error, NOT 500), a distinct `concurrency_queue_full` log, and a stats.queueRejections counter surfaced on /health. callClaude / callClaudeStreaming now acquire a slot (acquireClaudeSlot) before spawning; the release fn is wired into the existing idempotent cleanup() so the slot is freed on EVERY exit path (close/error/timeout/abort) — the #37/#40 slot-leak guard. MAX_CONCURRENT semantics (max concurrent claude procs) are unchanged; only overflow handling changed from throw-500 to queue-then-429. claudeSemaphore.limit is kept in sync with runtime /settings maxConcurrent changes. Live-verified (fake claude, MAX_CONCURRENT=1 MAX_QUEUE=1): 3 concurrent → exactly one 429 (Retry-After: 7, rate_limit_error body) + two 200s, on BOTH the non-streaming and streaming paths; /health stats.queueRejections=1 while stats.errors=0; after drain activeRequests=0 / inflight=0 / queued=0 and a follow-up request returns 200 (no slot leak). 2 new unit tests (247 passed, 0 failed). ALIGNMENT.md Rule 2 justification: this is an INFRA / PROCESS-MANAGEMENT change (a concurrency queue + backpressure status code in front of the existing spawn). It does NOT mirror a cli.js wire operation, adds no new endpoint or wire header, and introduces no API token — so there is no cli.js function to cite. (Retry-After is a standard HTTP response header on OCP's own 429, not a claude wire header.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ocp): macOS restart uses bootout+bootstrap so plist env is re-read The macOS restart path used `launchctl kickstart -k gui/$uid/dev.ocp.proxy`, which only re-execs the process and reuses launchd's CACHED environment — so a plist EnvironmentVariables edit (CLAUDE_BIND, CLAUDE_CODE_OAUTH_TOKEN, etc.) was silently ignored until a full unload/reload. This is the documented pit-index footgun. `ocp restart` (macOS) now does a full `launchctl bootout` + `bootstrap` of the agent via a new `_launchd_reload` helper, which re-reads the plist EnvironmentVariables so env changes take effect. Success is keyed on the bootstrap (the env-reloading load), not the bootout (which may legitimately fail if the agent is not currently loaded). A missing plist returns failure so the `elif` chain falls through to the legacy label and then to the Linux `systemctl --user restart` path unchanged (systemctl already re-reads its EnvironmentFile). Updates `ocp restart` help text and adds a README Troubleshooting subsection ("Env var change doesn't take effect after restart") with the manual bootout+bootstrap commands and a ps-based verification one-liner. Verified `_launchd_reload` with a stubbed launchctl: missing plist → rc=1 (falls through, no launchctl call); bootout-fail + bootstrap-ok → rc=0; bootstrap-fail → rc=1; call order is bootout then bootstrap. `bash -n ocp` clean; npm test 247/0 (server.mjs untouched). ALIGNMENT.md Rule 2 justification: this changes ONLY the `ocp` CLI wrapper's local service-restart mechanism (launchctl invocation). It does not touch server.mjs, the wire path, any endpoint/header, or any API token — so there is no cli.js function to cite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(readme): document new env vars CLAUDE_MAX_QUEUE / CLAUDE_QUEUE_RETRY_AFTER / OCP_SPAWN_REAL_HOME Release_kit contract (CLAUDE.md Iron Rule 5.5: "new env var → README § Environment Variables table") requires the three env vars added by the perf/concurrency fixes to be in the README table. Adds rows for CLAUDE_MAX_QUEUE (16) + CLAUDE_QUEUE_RETRY_AFTER (5) (FIX ⑥ -p wait-queue + 429) and OCP_SPAWN_REAL_HOME (FIX ③ spawn-home isolation kill-switch), each cross-referencing the additive /health.concurrency and /health.spawn fields. Addresses the independent reviewer's MEDIUM finding. Docs only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ocp-plugin): add openclaw.extensions for OpenClaw 2026.5.27 compat + sync version (②/ocp) ocp-plugin/package.json's openclaw object lacked the 'extensions' field that OpenClaw 2026.5.27 requires to install/load a plugin (matches the sibling olp plugin). Without it the daemon refused to load the local path plugin, breaking /ocp Telegram commands. Version synced 3.12.0 -> 3.16.2 to match the manifest. Plugin-layer change; no server.mjs / ALIGNMENT cli.js surface touched. --------- Co-authored-by: dtzp555 <dtzp555@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -844,6 +844,29 @@ ocp restart
|
|||||||
openclaw gateway restart
|
openclaw gateway restart
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Env var change (e.g. `CLAUDE_BIND`, `CLAUDE_CODE_OAUTH_TOKEN`) doesn't take effect after restart
|
||||||
|
|
||||||
|
On **macOS**, `ocp restart` does a full `launchctl bootout` + `bootstrap` of the agent, which **re-reads the plist `EnvironmentVariables`** — so an env change you made (in `~/Library/LaunchAgents/dev.ocp.proxy.plist`) actually takes effect:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ocp restart
|
||||||
|
```
|
||||||
|
|
||||||
|
This is deliberate: the older `launchctl kickstart -k` only re-execs the process and **reuses launchd's cached environment**, so plist env edits would be silently ignored. If you ever restart the agent by hand, use bootout+bootstrap, not `kickstart -k`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
launchctl bootout gui/$(id -u)/dev.ocp.proxy 2>/dev/null
|
||||||
|
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/dev.ocp.proxy.plist
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify the new value reached the running process:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ps -E -p "$(launchctl print gui/$(id -u)/dev.ocp.proxy 2>/dev/null | awk '/pid =/{print $3}')" | tr ' ' '\n' | grep CLAUDE_
|
||||||
|
```
|
||||||
|
|
||||||
|
On **Linux**, `systemctl --user restart` already re-reads the unit's `EnvironmentFile`, so no special handling is needed.
|
||||||
|
|
||||||
### Usage shows "unknown"
|
### Usage shows "unknown"
|
||||||
|
|
||||||
Usually caused by an expired Claude CLI session. Fix:
|
Usually caused by an expired Claude CLI session. Fix:
|
||||||
@@ -903,7 +926,9 @@ See [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) and ADR 0007 PR-
|
|||||||
| `CLAUDE_BIN` | *(auto-detect)* | Path to claude binary |
|
| `CLAUDE_BIN` | *(auto-detect)* | Path to claude binary |
|
||||||
| `CLAUDE_TIMEOUT` | `600000` | Request timeout (ms, default: 10 min) |
|
| `CLAUDE_TIMEOUT` | `600000` | Request timeout (ms, default: 10 min) |
|
||||||
| `CLAUDE_HEARTBEAT_INTERVAL` | `0` | Streaming SSE keepalive interval (ms). `0` = disabled. See "Streaming heartbeat" section. |
|
| `CLAUDE_HEARTBEAT_INTERVAL` | `0` | Streaming SSE keepalive interval (ms). `0` = disabled. See "Streaming heartbeat" section. |
|
||||||
| `CLAUDE_MAX_CONCURRENT` | `8` | Max concurrent claude processes |
|
| `CLAUDE_MAX_CONCURRENT` | `8` | Max concurrent claude processes (`-p`/stream-json path) |
|
||||||
|
| `CLAUDE_MAX_QUEUE` | `16` | Max requests **waiting** for a `-p` concurrency slot. Beyond `CLAUDE_MAX_CONCURRENT`, requests queue (up to this cap) instead of being rejected; when the queue is **also** full, the request gets `HTTP 429` + `Retry-After` (not an opaque 500). Surfaced on `/health.concurrency` + `/health.stats.queueRejections`. |
|
||||||
|
| `CLAUDE_QUEUE_RETRY_AFTER` | `5` | Seconds advertised in the `Retry-After` header on a `-p` concurrency-overflow `429`. |
|
||||||
| `CLAUDE_MAX_PROMPT_CHARS` | `150000` | Prompt truncation limit (chars) |
|
| `CLAUDE_MAX_PROMPT_CHARS` | `150000` | Prompt truncation limit (chars) |
|
||||||
| `CLAUDE_SESSION_TTL` | `3600000` | Session expiry (ms, default: 1 hour) |
|
| `CLAUDE_SESSION_TTL` | `3600000` | Session expiry (ms, default: 1 hour) |
|
||||||
| `CLAUDE_CACHE_TTL` | `0` | Response cache TTL (ms, 0 = disabled). Set to e.g. `300000` for 5-min cache |
|
| `CLAUDE_CACHE_TTL` | `0` | Response cache TTL (ms, 0 = disabled). Set to e.g. `300000` for 5-min cache |
|
||||||
@@ -915,6 +940,7 @@ See [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) and ADR 0007 PR-
|
|||||||
| `PROXY_ADVERTISE_ANON_KEY` | *(unset)* | When `=1`, advertise `PROXY_ANONYMOUS_KEY` in the public `/health` body for remote zero-config discovery. Default off — `/health` is unauthenticated, so this exposes the shared key to any LAN-reachable device (issue #109). Localhost always sees it regardless. |
|
| `PROXY_ADVERTISE_ANON_KEY` | *(unset)* | When `=1`, advertise `PROXY_ANONYMOUS_KEY` in the public `/health` body for remote zero-config discovery. Default off — `/health` is unauthenticated, so this exposes the shared key to any LAN-reachable device (issue #109). Localhost always sees it regardless. |
|
||||||
| `CLAUDE_TUI_MODE` | `false` | **Opt-in.** Set to `"true"` to serve requests via interactive `claude` (no `-p` / `--output-format` → `cc_entrypoint=cli`, subscription pool). **Single-user only** — see [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) for the security constraint. |
|
| `CLAUDE_TUI_MODE` | `false` | **Opt-in.** Set to `"true"` to serve requests via interactive `claude` (no `-p` / `--output-format` → `cc_entrypoint=cli`, subscription pool). **Single-user only** — see [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) for the security constraint. |
|
||||||
| `CLAUDE_CODE_OAUTH_TOKEN` | *(unset)* | OAuth bearer token (highest-precedence credential source for the `-p` path). **Recommended for TUI-mode hosts:** when set (and `OCP_TUI_HOME` unset), OCP runs the interactive `claude` in a **credential-isolated home** (`$HOME/.ocp-tui/home`, no `credentials.json`) so this long-lived token is the only credential and is authoritative — interactive `claude` otherwise *prefers* `~/.claude/.credentials.json` over the env var, so a stale one shadows the token and its single-use refresh token gets corrupted by the spawn/teardown cycle (the permanent `Please run /login` 401 — see [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) and ADR 0007 PR-D). The token appears in the pane command (ps-visible) — acceptable for the single-user A-path; the multi-user B-path is refused at boot. |
|
| `CLAUDE_CODE_OAUTH_TOKEN` | *(unset)* | OAuth bearer token (highest-precedence credential source for the `-p` path). **Recommended for TUI-mode hosts:** when set (and `OCP_TUI_HOME` unset), OCP runs the interactive `claude` in a **credential-isolated home** (`$HOME/.ocp-tui/home`, no `credentials.json`) so this long-lived token is the only credential and is authoritative — interactive `claude` otherwise *prefers* `~/.claude/.credentials.json` over the env var, so a stale one shadows the token and its single-use refresh token gets corrupted by the spawn/teardown cycle (the permanent `Please run /login` 401 — see [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) and ADR 0007 PR-D). The token appears in the pane command (ps-visible) — acceptable for the single-user A-path; the multi-user B-path is refused at boot. |
|
||||||
|
| `OCP_SPAWN_REAL_HOME` | *(unset)* | Kill-switch for the default `-p`/stream-json **spawn-home isolation** (latency fix). When unset and an OAuth token is resolvable, OCP runs the per-request `claude` spawn in a **credential-free minimal scratch home** (`$HOME/.ocp/spawn-home`, no `.credentials.json`/`settings.json`/plugins) with a neutral cwd and the env token — so it loads none of the operator's heavy global `~/.claude` (plugins/skills/hooks) or the project `CLAUDE.md`, cutting per-request latency (measured ~10–28s → ~3–7s). Set to `"1"` to force the legacy real-`HOME` spawn (no cwd override) even when a token exists. With **no** resolvable token, OCP falls back to the real `HOME` automatically (zero regression). Active mode is shown at startup and on `/health.spawn`. |
|
||||||
| `CLAUDE_TUI_WALLCLOCK_MS` | `120000` | (TUI-mode) Maximum time in ms to wait for the native transcript to signal turn completion. Increase for long Opus thinking turns. |
|
| `CLAUDE_TUI_WALLCLOCK_MS` | `120000` | (TUI-mode) Maximum time in ms to wait for the native transcript to signal turn completion. Increase for long Opus thinking turns. |
|
||||||
| `OCP_TUI_CWD` | `$HOME/.ocp-tui/work` | (TUI-mode) Scratch working directory where interactive claude sessions run. Transcripts land under `<HOME>/.claude/projects/<encoded-cwd>/`. Created automatically. |
|
| `OCP_TUI_CWD` | `$HOME/.ocp-tui/work` | (TUI-mode) Scratch working directory where interactive claude sessions run. Transcripts land under `<HOME>/.claude/projects/<encoded-cwd>/`. Created automatically. |
|
||||||
| `OCP_TUI_HOME` | *(auto)* | (TUI-mode) `HOME` claude runs under. **When unset, OCP picks it for you:** if `CLAUDE_CODE_OAUTH_TOKEN` is set → a **credential-isolated** scratch home `$HOME/.ocp-tui/home` (no `credentials.json`, env-token auth — **recommended**); if no env token → the operator's real home (legacy shared `credentials.json`). Setting this to an **explicit** path overrides the auto-default. The credential handling at that path still follows the env token: **with** the env token it is credential-free (env-token auth, no `credentials.json` written); **without** the env token (and the path ≠ real home) it uses the legacy symlinked-credentials scratch mode, which carries the credential-fork caveat — see ADR 0007. |
|
| `OCP_TUI_HOME` | *(auto)* | (TUI-mode) `HOME` claude runs under. **When unset, OCP picks it for you:** if `CLAUDE_CODE_OAUTH_TOKEN` is set → a **credential-isolated** scratch home `$HOME/.ocp-tui/home` (no `credentials.json`, env-token auth — **recommended**); if no env token → the operator's real home (legacy shared `credentials.json`). Setting this to an **explicit** path overrides the auto-default. The credential handling at that path still follows the env token: **with** the env token it is credential-free (env-token auth, no `credentials.json` written); **without** the env token (and the path ≠ real home) it uses the legacy symlinked-credentials scratch mode, which carries the credential-fork caveat — see ADR 0007. |
|
||||||
|
|||||||
@@ -573,21 +573,42 @@ Usage:
|
|||||||
ocp restart Restart the Claude proxy service
|
ocp restart Restart the Claude proxy service
|
||||||
ocp restart gateway Restart the OpenClaw gateway
|
ocp restart gateway Restart the OpenClaw gateway
|
||||||
(briefly disconnects all Telegram/Discord bots)
|
(briefly disconnects all Telegram/Discord bots)
|
||||||
|
|
||||||
|
Note (macOS): restart does a full launchctl bootout + bootstrap, NOT
|
||||||
|
`kickstart -k`. bootout+bootstrap re-reads the plist's EnvironmentVariables,
|
||||||
|
so an env change you made (e.g. CLAUDE_BIND, CLAUDE_CODE_OAUTH_TOKEN) actually
|
||||||
|
takes effect. `kickstart -k` only re-execs the process and reuses launchd's
|
||||||
|
cached env, so env edits would be silently ignored. (Linux systemctl already
|
||||||
|
re-reads its EnvironmentFile on restart.)
|
||||||
EOF
|
EOF
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# macOS only: reload a launchd agent via bootout + bootstrap so plist
|
||||||
|
# EnvironmentVariables are re-read (kickstart -k would reuse the cached env).
|
||||||
|
# Args: <uid> <label> <plist-path>. Returns 0 iff bootstrap succeeds.
|
||||||
|
_launchd_reload() {
|
||||||
|
local uid="$1" label="$2" plist="$3"
|
||||||
|
[[ -f "$plist" ]] || return 1
|
||||||
|
# bootout may legitimately fail if the agent is not currently loaded — that's fine,
|
||||||
|
# we only require the subsequent bootstrap to succeed (the load that re-reads env).
|
||||||
|
launchctl bootout "gui/$uid/$label" 2>/dev/null || true
|
||||||
|
launchctl bootstrap "gui/$uid" "$plist" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
cmd_restart() {
|
cmd_restart() {
|
||||||
if [[ "${1:-}" == "gateway" ]]; then
|
if [[ "${1:-}" == "gateway" ]]; then
|
||||||
echo "Restarting gateway..."
|
echo "Restarting gateway..."
|
||||||
openclaw gateway restart 2>&1
|
openclaw gateway restart 2>&1
|
||||||
else
|
else
|
||||||
echo "Restarting proxy..."
|
echo "Restarting proxy..."
|
||||||
# Try current service name, then legacy, then manual restart
|
# Try current service name, then legacy, then manual restart.
|
||||||
|
# macOS: bootout+bootstrap (re-reads plist EnvironmentVariables — see cmd_restart_help).
|
||||||
|
# Linux: systemctl --user restart already re-reads its EnvironmentFile.
|
||||||
local uid
|
local uid
|
||||||
uid=$(id -u)
|
uid=$(id -u)
|
||||||
if launchctl kickstart -k "gui/$uid/dev.ocp.proxy" 2>/dev/null; then
|
if _launchd_reload "$uid" "dev.ocp.proxy" "$HOME/Library/LaunchAgents/dev.ocp.proxy.plist"; then
|
||||||
true
|
true
|
||||||
elif launchctl kickstart -k "gui/$uid/ai.openclaw.proxy" 2>/dev/null; then
|
elif _launchd_reload "$uid" "ai.openclaw.proxy" "$HOME/Library/LaunchAgents/ai.openclaw.proxy.plist"; then
|
||||||
true
|
true
|
||||||
elif systemctl --user restart ocp-proxy 2>/dev/null; then
|
elif systemctl --user restart ocp-proxy 2>/dev/null; then
|
||||||
true
|
true
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "ocp",
|
"name": "ocp",
|
||||||
"version": "3.12.0",
|
"version": "3.16.2",
|
||||||
"description": "Slash commands for the OpenClaw Proxy",
|
"description": "Slash commands for the OpenClaw Proxy",
|
||||||
"main": "index.js",
|
"main": "index.js",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
@@ -9,6 +9,7 @@
|
|||||||
"openclaw": {
|
"openclaw": {
|
||||||
"type": "plugin",
|
"type": "plugin",
|
||||||
"id": "ocp",
|
"id": "ocp",
|
||||||
"pluginManifest": "openclaw.plugin.json"
|
"pluginManifest": "openclaw.plugin.json",
|
||||||
|
"extensions": ["./index.js"]
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
+241
-18
@@ -20,7 +20,10 @@
|
|||||||
* CLAUDE_MCP_CONFIG — path to MCP server config JSON file
|
* CLAUDE_MCP_CONFIG — path to MCP server config JSON file
|
||||||
* CLAUDE_SESSION_TTL — session TTL in ms (default: 3600000 = 1h)
|
* CLAUDE_SESSION_TTL — session TTL in ms (default: 3600000 = 1h)
|
||||||
* CLAUDE_MAX_CONCURRENT — max concurrent claude processes, -p/stream-json path (default: 8)
|
* CLAUDE_MAX_CONCURRENT — max concurrent claude processes, -p/stream-json path (default: 8)
|
||||||
|
* CLAUDE_MAX_QUEUE — max requests waiting for a -p slot before HTTP 429 (default: 16)
|
||||||
* OCP_TUI_MAX_CONCURRENT — max concurrent interactive TUI turns, TUI-mode path (default: 2)
|
* OCP_TUI_MAX_CONCURRENT — max concurrent interactive TUI turns, TUI-mode path (default: 2)
|
||||||
|
* OCP_SPAWN_REAL_HOME — "1" forces the -p spawn to use the real HOME (disables the
|
||||||
|
* latency spawn-home isolation; default: isolated when a token exists)
|
||||||
* CLAUDE_BREAKER_THRESHOLD — failures in window before circuit opens (default: 6)
|
* CLAUDE_BREAKER_THRESHOLD — failures in window before circuit opens (default: 6)
|
||||||
* CLAUDE_BREAKER_COOLDOWN — base ms to wait before retrying after circuit opens (default: 120000)
|
* CLAUDE_BREAKER_COOLDOWN — base ms to wait before retrying after circuit opens (default: 120000)
|
||||||
* CLAUDE_BREAKER_WINDOW — sliding window duration in ms (default: 300000 = 5min)
|
* CLAUDE_BREAKER_WINDOW — sliding window duration in ms (default: 300000 = 5min)
|
||||||
@@ -31,7 +34,7 @@
|
|||||||
import { createServer } from "node:http";
|
import { createServer } from "node:http";
|
||||||
import { spawn, execFileSync } from "node:child_process";
|
import { spawn, execFileSync } from "node:child_process";
|
||||||
import { randomUUID, timingSafeEqual } from "node:crypto";
|
import { randomUUID, timingSafeEqual } from "node:crypto";
|
||||||
import { readFileSync, readdirSync, accessSync, existsSync, constants, chmodSync, statSync } from "node:fs";
|
import { readFileSync, readdirSync, accessSync, existsSync, constants, chmodSync, statSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
|
||||||
import { fileURLToPath } from "node:url";
|
import { fileURLToPath } from "node:url";
|
||||||
import { dirname, join } from "node:path";
|
import { dirname, join } from "node:path";
|
||||||
import { homedir } from "node:os";
|
import { homedir } from "node:os";
|
||||||
@@ -272,6 +275,15 @@ const SYSTEM_PROMPT = process.env.CLAUDE_SYSTEM_PROMPT || "";
|
|||||||
const MCP_CONFIG = process.env.CLAUDE_MCP_CONFIG || "";
|
const MCP_CONFIG = process.env.CLAUDE_MCP_CONFIG || "";
|
||||||
let SESSION_TTL = parseInt(process.env.CLAUDE_SESSION_TTL || "3600000", 10);
|
let SESSION_TTL = parseInt(process.env.CLAUDE_SESSION_TTL || "3600000", 10);
|
||||||
let MAX_CONCURRENT = parseInt(process.env.CLAUDE_MAX_CONCURRENT || "8", 10);
|
let MAX_CONCURRENT = parseInt(process.env.CLAUDE_MAX_CONCURRENT || "8", 10);
|
||||||
|
// FIX ⑥ (concurrency): bound on requests WAITING for a -p concurrency slot. Beyond
|
||||||
|
// MAX_CONCURRENT, requests queue (up to CLAUDE_MAX_QUEUE) instead of being rejected; when the
|
||||||
|
// queue is ALSO full, the request gets HTTP 429 + Retry-After (not an opaque 500). See
|
||||||
|
// claudeSemaphore / acquireClaudeSlot below.
|
||||||
|
const CLAUDE_MAX_QUEUE = parseInt(process.env.CLAUDE_MAX_QUEUE || "16", 10);
|
||||||
|
// Retry-After seconds advertised on a 429 backpressure response. A claude turn is typically a
|
||||||
|
// few seconds to tens of seconds; a small constant nudge keeps well-behaved clients from
|
||||||
|
// hammering while the queue drains.
|
||||||
|
const CLAUDE_QUEUE_RETRY_AFTER = parseInt(process.env.CLAUDE_QUEUE_RETRY_AFTER || "5", 10);
|
||||||
const BREAKER_THRESHOLD = parseInt(process.env.CLAUDE_BREAKER_THRESHOLD || "6", 10);
|
const BREAKER_THRESHOLD = parseInt(process.env.CLAUDE_BREAKER_THRESHOLD || "6", 10);
|
||||||
const BREAKER_COOLDOWN = parseInt(process.env.CLAUDE_BREAKER_COOLDOWN || "120000", 10);
|
const BREAKER_COOLDOWN = parseInt(process.env.CLAUDE_BREAKER_COOLDOWN || "120000", 10);
|
||||||
const BREAKER_WINDOW = parseInt(process.env.CLAUDE_BREAKER_WINDOW || "300000", 10);
|
const BREAKER_WINDOW = parseInt(process.env.CLAUDE_BREAKER_WINDOW || "300000", 10);
|
||||||
@@ -279,6 +291,12 @@ const BREAKER_HALF_OPEN_MAX = parseInt(process.env.CLAUDE_BREAKER_HALF_OPEN_MAX
|
|||||||
const HEARTBEAT_INTERVAL = parseInt(process.env.CLAUDE_HEARTBEAT_INTERVAL || "0", 10);
|
const HEARTBEAT_INTERVAL = parseInt(process.env.CLAUDE_HEARTBEAT_INTERVAL || "0", 10);
|
||||||
const BIND_ADDRESS = process.env.CLAUDE_BIND || "127.0.0.1";
|
const BIND_ADDRESS = process.env.CLAUDE_BIND || "127.0.0.1";
|
||||||
const NO_CONTEXT = process.env.CLAUDE_NO_CONTEXT === "true";
|
const NO_CONTEXT = process.env.CLAUDE_NO_CONTEXT === "true";
|
||||||
|
// Kill-switch for the FIX-③ default-path spawn-home isolation (see resolveSpawnHome /
|
||||||
|
// spawnHomeMode below). When "1", the -p/stream-json spawn always runs in the operator's
|
||||||
|
// real HOME with no cwd override — byte-for-byte the pre-isolation behaviour — even if an
|
||||||
|
// OAuth token is resolvable. Provided as an escape hatch in case a host depends on the real
|
||||||
|
// HOME's claude config for the spawned process.
|
||||||
|
const SPAWN_REAL_HOME = process.env.OCP_SPAWN_REAL_HOME === "1";
|
||||||
const AUTH_MODE = process.env.CLAUDE_AUTH_MODE || (PROXY_API_KEY ? "shared" : "none");
|
const AUTH_MODE = process.env.CLAUDE_AUTH_MODE || (PROXY_API_KEY ? "shared" : "none");
|
||||||
const ADMIN_KEY = process.env.OCP_ADMIN_KEY || "";
|
const ADMIN_KEY = process.env.OCP_ADMIN_KEY || "";
|
||||||
const PROXY_ANONYMOUS_KEY = process.env.PROXY_ANONYMOUS_KEY || "";
|
const PROXY_ANONYMOUS_KEY = process.env.PROXY_ANONYMOUS_KEY || "";
|
||||||
@@ -332,6 +350,116 @@ const tuiStats = {
|
|||||||
entrypointMismatches: 0, // count of cli-expected-but-got-other turns
|
entrypointMismatches: 0, // count of cli-expected-but-got-other turns
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// ── FIX ③ (latency): default-path (-p / stream-json) spawn-home isolation ──────────────
|
||||||
|
// PROBLEM (measured, not theoretical): OCP's default spawn inherits the operator's real HOME
|
||||||
|
// (loading the global ~/.claude — plugins, skills, hooks) and runs with cwd=~/ocp (loading the
|
||||||
|
// project CLAUDE.md / skills) on EVERY request. Pure Anthropic API floor for haiku "hi" ≈ 1–2s;
|
||||||
|
// the same claude CLI spawned in the operator's real HOME/cwd ≈ 10–28s; a clean minimal HOME +
|
||||||
|
// CLAUDE_CODE_OAUTH_TOKEN ≈ 3–7s and authenticates fine. So the heavy global config is pure
|
||||||
|
// per-request latency tax with no proxy benefit (a proxy must NOT leak the host's context into
|
||||||
|
// the proxied turn — same rationale as NO_CONTEXT / the TUI path's CLAUDE_MDS suppression).
|
||||||
|
//
|
||||||
|
// FIX: when an OAuth token is resolvable, run the default spawn under a CREDENTIAL-FREE minimal
|
||||||
|
// scratch HOME (`<realHome>/.ocp/spawn-home`) with cwd = that same neutral dir, and pass the
|
||||||
|
// resolved token via CLAUDE_CODE_OAUTH_TOKEN so the env token is authoritative. This MIRRORS the
|
||||||
|
// TUI path's resolveTuiHome() env-token mode (lib/tui/session.mjs): for `-p`, the env token wins
|
||||||
|
// over a credentials.json (the opposite of interactive claude), so credential isolation is not
|
||||||
|
// even strictly required for auth here, but a credential-FREE home is still the right shape —
|
||||||
|
// nothing to refresh, nothing to corrupt, no heavy config to load.
|
||||||
|
//
|
||||||
|
// SAFETY: if NO token is resolvable → fall back to the real HOME with no cwd override (zero
|
||||||
|
// regression). OCP_SPAWN_REAL_HOME=1 forces that legacy behaviour even when a token exists.
|
||||||
|
// The scratch home holds NO .credentials.json / NO settings.json / NO plugins — it is created
|
||||||
|
// minimal and (re)cleaned of any settings.json on prepare.
|
||||||
|
const SPAWN_HOME_DIR = `${process.env.HOME}/.ocp/spawn-home`;
|
||||||
|
|
||||||
|
// Idempotently prepare the minimal scratch HOME. Creates the dir if missing and removes any
|
||||||
|
// settings.json that might have crept in, so the spawned claude loads no host settings/plugins.
|
||||||
|
// Best-effort: a failure here degrades toward "dir may be missing", which spawn() tolerates by
|
||||||
|
// erroring loudly — never a silent auth/credential corruption (there are no credentials here).
|
||||||
|
function prepareSpawnHome(dir = SPAWN_HOME_DIR) {
|
||||||
|
try {
|
||||||
|
mkdirSync(`${dir}/.claude`, { recursive: true });
|
||||||
|
// Belt-and-braces: ensure no settings.json/plugins leak in (this home is fully ours).
|
||||||
|
for (const f of [`${dir}/.claude/settings.json`, `${dir}/.claude/settings.local.json`]) {
|
||||||
|
try { if (existsSync(f)) rmSync(f, { force: true }); } catch { /* best effort */ }
|
||||||
|
}
|
||||||
|
} catch { /* best effort — spawn will surface a hard error if the dir is truly unusable */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
// Resolve the default-spawn HOME-isolation decision ONCE, lazily + memoized (so it runs after
|
||||||
|
// getOAuthCredentials is defined regardless of source order, and the token probe happens at most
|
||||||
|
// once). Returns { isolated, home, token } where:
|
||||||
|
// - isolated:true → spawn under SPAWN_HOME_DIR with cwd=SPAWN_HOME_DIR + the env token.
|
||||||
|
// - isolated:false → legacy real-HOME spawn, no cwd override (no token, or kill-switch on).
|
||||||
|
// NEVER logs/returns the token verbatim to any caller that logs; spawnClaudeProcess uses it only
|
||||||
|
// to populate the spawn env. token is null when isolation is off.
|
||||||
|
let _spawnHomeMode = null;
|
||||||
|
function getSpawnHomeMode() {
|
||||||
|
if (_spawnHomeMode) return _spawnHomeMode;
|
||||||
|
if (SPAWN_REAL_HOME) {
|
||||||
|
_spawnHomeMode = { isolated: false, home: null, token: null, reason: "kill-switch (OCP_SPAWN_REAL_HOME=1)" };
|
||||||
|
return _spawnHomeMode;
|
||||||
|
}
|
||||||
|
let token = null;
|
||||||
|
try { token = getOAuthCredentials()?.accessToken || null; } catch { token = null; }
|
||||||
|
if (token) {
|
||||||
|
prepareSpawnHome(SPAWN_HOME_DIR);
|
||||||
|
_spawnHomeMode = { isolated: true, home: SPAWN_HOME_DIR, token, reason: "oauth token resolved" };
|
||||||
|
} else {
|
||||||
|
_spawnHomeMode = { isolated: false, home: null, token: null, reason: "no oauth token resolvable" };
|
||||||
|
}
|
||||||
|
return _spawnHomeMode;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── FIX ⑥ (concurrency): bounded wait-queue for the -p / stream-json path ──────────────
|
||||||
|
// PROBLEM (proven): spawnClaudeProcess used `if (activeRequests >= MAX_CONCURRENT) throw` →
|
||||||
|
// the client got an opaque 500 AND the rejection was NOT counted in stats (a 15-concurrent
|
||||||
|
// stress run returned 7×500 while /health stats.errors stayed 0). The TUI path already has a
|
||||||
|
// bounded-queue semaphore (TuiSemaphore); the -p path did not.
|
||||||
|
//
|
||||||
|
// FIX: requests beyond MAX_CONCURRENT WAIT on this semaphore (up to CLAUDE_MAX_QUEUE) instead of
|
||||||
|
// being rejected. Only when the queue is ALSO full do we reject — with HTTP 429 + Retry-After
|
||||||
|
// (deterministic backpressure), a distinct `concurrency_queue_full` log, and a stats.queueRejections
|
||||||
|
// counter that shows up on /health. The slot is released on EVERY exit path via the existing
|
||||||
|
// idempotent cleanup() (proc exit/close/error/timeout) — the #37/#40 slot-leak guard.
|
||||||
|
const claudeSemaphore = new TuiSemaphore(MAX_CONCURRENT, { maxQueue: CLAUDE_MAX_QUEUE });
|
||||||
|
|
||||||
|
// Tagged error so callers can map this single overflow case to HTTP 429 (every OTHER throw stays
|
||||||
|
// a 500). Carries retryAfter for the Retry-After header.
|
||||||
|
class ConcurrencyOverflowError extends Error {
|
||||||
|
constructor(message) { super(message); this.name = "ConcurrencyOverflowError"; this.httpStatus = 429; this.retryAfter = CLAUDE_QUEUE_RETRY_AFTER; }
|
||||||
|
}
|
||||||
|
|
||||||
|
// Acquire a -p concurrency slot, queuing if all are busy (up to CLAUDE_MAX_QUEUE). Resolves to a
|
||||||
|
// release() fn that MUST be called exactly once on every exit path (wired into ctx.cleanup()).
|
||||||
|
// Rejects with ConcurrencyOverflowError when the wait-queue is full. Increments stats.queued while
|
||||||
|
// waiting (decremented on acquire) and stats.queueRejections on overflow.
|
||||||
|
async function acquireClaudeSlot() {
|
||||||
|
stats.queued = claudeSemaphore.queued + 1; // reflect this waiter before we (maybe) block
|
||||||
|
try {
|
||||||
|
await claudeSemaphore.acquire();
|
||||||
|
} catch (e) {
|
||||||
|
stats.queued = claudeSemaphore.queued;
|
||||||
|
stats.queueRejections++;
|
||||||
|
logEvent("warn", "concurrency_queue_full", {
|
||||||
|
limit: claudeSemaphore.limit, maxQueue: claudeSemaphore.maxQueue,
|
||||||
|
inflight: claudeSemaphore.inflight, queued: claudeSemaphore.queued,
|
||||||
|
});
|
||||||
|
throw new ConcurrencyOverflowError(
|
||||||
|
`backpressure: concurrency limit (${claudeSemaphore.limit}) reached and wait queue ` +
|
||||||
|
`(${claudeSemaphore.maxQueue}) is full — retry shortly`);
|
||||||
|
}
|
||||||
|
stats.queued = claudeSemaphore.queued;
|
||||||
|
let released = false;
|
||||||
|
return function releaseClaudeSlot() {
|
||||||
|
if (released) return; // idempotent — cleanup() may be reached via multiple proc events
|
||||||
|
released = true;
|
||||||
|
claudeSemaphore.release();
|
||||||
|
stats.queued = claudeSemaphore.queued;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
// SECURITY fail-loud: TUI-mode is incompatible with any configuration that allows
|
// SECURITY fail-loud: TUI-mode is incompatible with any configuration that allows
|
||||||
// non-operator prompts to reach the interactive claude session. Three cases:
|
// non-operator prompts to reach the interactive claude session. Three cases:
|
||||||
// 1. AUTH_MODE=multi — guest/anonymous keys can submit prompts.
|
// 1. AUTH_MODE=multi — guest/anonymous keys can submit prompts.
|
||||||
@@ -542,6 +670,8 @@ const stats = {
|
|||||||
sessionHits: 0,
|
sessionHits: 0,
|
||||||
sessionMisses: 0,
|
sessionMisses: 0,
|
||||||
oneOffRequests: 0,
|
oneOffRequests: 0,
|
||||||
|
queued: 0, // current requests waiting for a -p concurrency slot (FIX ⑥)
|
||||||
|
queueRejections: 0, // total requests rejected with HTTP 429 because the wait-queue was full (FIX ⑥)
|
||||||
};
|
};
|
||||||
const recentErrors = []; // last 20 errors
|
const recentErrors = []; // last 20 errors
|
||||||
|
|
||||||
@@ -769,11 +899,14 @@ function getModelTier(cliModel) {
|
|||||||
// (messagesToPrompt), so multi-turn correctness is preserved without sessions.
|
// (messagesToPrompt), so multi-turn correctness is preserved without sessions.
|
||||||
// The sessions Map is retained for stats/logging but no longer drives --resume.
|
// The sessions Map is retained for stats/logging but no longer drives --resume.
|
||||||
// Reference: OLP ADR 0009 Amendment 1 + commit 97e7d16.
|
// Reference: OLP ADR 0009 Amendment 1 + commit 97e7d16.
|
||||||
function spawnClaudeProcess(model, messages, conversationId, keyName) {
|
// FIX ⑥: concurrency is now bounded by the claudeSemaphore via acquireClaudeSlot(), which the
|
||||||
if (stats.activeRequests >= MAX_CONCURRENT) {
|
// caller MUST await before calling this, passing the resulting release fn as `releaseSlot`. The
|
||||||
throw new Error(`concurrency limit reached (${stats.activeRequests}/${MAX_CONCURRENT})`);
|
// old `if (activeRequests >= MAX_CONCURRENT) throw` gate (→ opaque 500, uncounted) is GONE: at
|
||||||
}
|
// most MAX_CONCURRENT callers hold a slot when they reach here, so this spawn is always within
|
||||||
|
// budget. releaseSlot is wired into the idempotent cleanup() so the slot is freed on EVERY exit
|
||||||
|
// path (close/error/timeout/abort). Back-compat: releaseSlot defaults to a no-op so any future
|
||||||
|
// internal caller that does its own gating still works.
|
||||||
|
function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlot = () => {}) {
|
||||||
const cliModel = MODEL_MAP[model] || model;
|
const cliModel = MODEL_MAP[model] || model;
|
||||||
|
|
||||||
// Circuit breaker: disabled (see comment at top of breaker section)
|
// Circuit breaker: disabled (see comment at top of breaker section)
|
||||||
@@ -810,7 +943,25 @@ function spawnClaudeProcess(model, messages, conversationId, keyName) {
|
|||||||
env.CLAUDE_CODE_DISABLE_AUTO_MEMORY = "1";
|
env.CLAUDE_CODE_DISABLE_AUTO_MEMORY = "1";
|
||||||
}
|
}
|
||||||
|
|
||||||
const proc = spawn(CLAUDE, cliArgs, { env, stdio: ["pipe", "pipe", "pipe"] });
|
// FIX ③ (latency): default-path spawn-home isolation. When a token is resolvable (and the
|
||||||
|
// OCP_SPAWN_REAL_HOME kill-switch is off), run claude under a credential-free minimal HOME
|
||||||
|
// with cwd = that same neutral dir, so it loads NONE of the operator's global ~/.claude
|
||||||
|
// (plugins/skills/hooks) or the ~/ocp project CLAUDE.md/skills — the measured 10–28s → 3–7s
|
||||||
|
// latency win. The env token is authoritative for `-p` (unlike interactive claude). When no
|
||||||
|
// token is resolvable, falls back to real HOME + inherited cwd (zero regression). See
|
||||||
|
// getSpawnHomeMode() / prepareSpawnHome() above. The DISABLE_CLAUDE_MDS / AUTO_MEMORY flags
|
||||||
|
// are set unconditionally in isolated mode (belt-and-braces; mirrors the TUI path).
|
||||||
|
const spawnHome = getSpawnHomeMode();
|
||||||
|
const spawnOpts = { env, stdio: ["pipe", "pipe", "pipe"] };
|
||||||
|
if (spawnHome.isolated) {
|
||||||
|
env.HOME = spawnHome.home;
|
||||||
|
env.CLAUDE_CODE_OAUTH_TOKEN = spawnHome.token; // env token is authoritative for -p
|
||||||
|
env.CLAUDE_CODE_DISABLE_CLAUDE_MDS = "1";
|
||||||
|
env.CLAUDE_CODE_DISABLE_AUTO_MEMORY = "1";
|
||||||
|
spawnOpts.cwd = spawnHome.home; // neutral cwd: no project CLAUDE.md/skills
|
||||||
|
}
|
||||||
|
|
||||||
|
const proc = spawn(CLAUDE, cliArgs, spawnOpts);
|
||||||
activeProcesses.add(proc);
|
activeProcesses.add(proc);
|
||||||
|
|
||||||
const t0 = Date.now();
|
const t0 = Date.now();
|
||||||
@@ -822,6 +973,10 @@ function spawnClaudeProcess(model, messages, conversationId, keyName) {
|
|||||||
cleaned = true;
|
cleaned = true;
|
||||||
clearTimeout(overallTimer);
|
clearTimeout(overallTimer);
|
||||||
stats.activeRequests--;
|
stats.activeRequests--;
|
||||||
|
// FIX ⑥: free the concurrency slot for a queued waiter. releaseSlot is itself idempotent,
|
||||||
|
// and cleanup() is guarded by `cleaned`, so the slot is released exactly once on the first
|
||||||
|
// exit path reached (proc 'exit' fires before 'close'; 'error' covers spawn failure).
|
||||||
|
try { releaseSlot(); } catch { /* never let release throw out of cleanup */ }
|
||||||
}
|
}
|
||||||
|
|
||||||
// Guarantee slot release on ANY exit path (normal close, error, timeout kill,
|
// Guarantee slot release on ANY exit path (normal close, error, timeout kill,
|
||||||
@@ -891,12 +1046,18 @@ function spawnClaudeProcess(model, messages, conversationId, keyName) {
|
|||||||
// We accumulate full text across all content_block_delta events plus the
|
// We accumulate full text across all content_block_delta events plus the
|
||||||
// assistant-aggregate fallback, then resolve with the assembled string.
|
// assistant-aggregate fallback, then resolve with the assembled string.
|
||||||
// Reference: OLP ADR 0009 Amendment 1 + commit 97e7d16.
|
// Reference: OLP ADR 0009 Amendment 1 + commit 97e7d16.
|
||||||
function callClaude(model, messages, conversationId, keyName) {
|
async function callClaude(model, messages, conversationId, keyName) {
|
||||||
|
// FIX ⑥: acquire a concurrency slot first (queues up to CLAUDE_MAX_QUEUE; rejects with a
|
||||||
|
// ConcurrencyOverflowError → 429 when the queue is full). The release fn is passed into the
|
||||||
|
// spawn so the idempotent cleanup() frees it on every exit path. If the spawn itself throws
|
||||||
|
// synchronously (before cleanup is wired), release here so the slot never leaks.
|
||||||
|
const releaseSlot = await acquireClaudeSlot();
|
||||||
return new Promise((resolve, reject) => {
|
return new Promise((resolve, reject) => {
|
||||||
let ctx;
|
let ctx;
|
||||||
try {
|
try {
|
||||||
ctx = spawnClaudeProcess(model, messages, conversationId, keyName);
|
ctx = spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlot);
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
|
releaseSlot();
|
||||||
return reject(err);
|
return reject(err);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1063,14 +1224,28 @@ function startHeartbeat(res, intervalMs, sessionId) {
|
|||||||
// We parse line-by-line and forward content_block_delta text events as SSE.
|
// We parse line-by-line and forward content_block_delta text events as SSE.
|
||||||
// The result event triggers the stop/[DONE] sequence.
|
// The result event triggers the stop/[DONE] sequence.
|
||||||
// Reference: OLP ADR 0009 Amendment 1 + commits 97e7d16, 65f945c.
|
// Reference: OLP ADR 0009 Amendment 1 + commits 97e7d16, 65f945c.
|
||||||
function callClaudeStreaming(model, messages, conversationId, res, authInfo = {}) {
|
async function callClaudeStreaming(model, messages, conversationId, res, authInfo = {}) {
|
||||||
const id = `chatcmpl-${randomUUID()}`;
|
const id = `chatcmpl-${randomUUID()}`;
|
||||||
const created = Math.floor(Date.now() / 1000);
|
const created = Math.floor(Date.now() / 1000);
|
||||||
|
|
||||||
|
// FIX ⑥: acquire a concurrency slot first (queues up to CLAUDE_MAX_QUEUE). On overflow, surface
|
||||||
|
// HTTP 429 + Retry-After (NOT 500). Release is wired into cleanup() for every exit path; if the
|
||||||
|
// spawn throws synchronously before cleanup is wired, release here.
|
||||||
|
let releaseSlot;
|
||||||
|
try {
|
||||||
|
releaseSlot = await acquireClaudeSlot();
|
||||||
|
} catch (err) {
|
||||||
|
if (err instanceof ConcurrencyOverflowError) {
|
||||||
|
return jsonResponse(res, 429, { error: { message: sanitizeError(err.message), type: "rate_limit_error" } }, { "Retry-After": String(err.retryAfter) });
|
||||||
|
}
|
||||||
|
return jsonResponse(res, 500, { error: { message: sanitizeError(err.message), type: "proxy_error" } });
|
||||||
|
}
|
||||||
|
|
||||||
let ctx;
|
let ctx;
|
||||||
try {
|
try {
|
||||||
ctx = spawnClaudeProcess(model, messages, conversationId, authInfo.keyName);
|
ctx = spawnClaudeProcess(model, messages, conversationId, authInfo.keyName, releaseSlot);
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
|
releaseSlot();
|
||||||
return jsonResponse(res, 500, { error: { message: sanitizeError(err.message), type: "proxy_error" } });
|
return jsonResponse(res, 500, { error: { message: sanitizeError(err.message), type: "proxy_error" } });
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1263,12 +1438,23 @@ function sanitizeError(msg) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ── Response helpers ────────────────────────────────────────────────────
|
// ── Response helpers ────────────────────────────────────────────────────
|
||||||
function jsonResponse(res, status, data) {
|
function jsonResponse(res, status, data, extraHeaders = null) {
|
||||||
if (res.headersSent || res.writableEnded || res.destroyed) return;
|
if (res.headersSent || res.writableEnded || res.destroyed) return;
|
||||||
res.writeHead(status, { "Content-Type": "application/json" });
|
// extraHeaders is optional + additive (e.g. Retry-After on a 429); Content-Type always wins.
|
||||||
|
res.writeHead(status, { ...(extraHeaders || {}), "Content-Type": "application/json" });
|
||||||
res.end(JSON.stringify(data));
|
res.end(JSON.stringify(data));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// FIX ⑥: map an upstream error to the right HTTP response. A ConcurrencyOverflowError (the
|
||||||
|
// wait-queue was full) becomes HTTP 429 + Retry-After + rate_limit_error; every other error
|
||||||
|
// stays a 500 proxy_error (byte-for-byte the pre-fix behaviour for non-overflow errors).
|
||||||
|
function respondUpstreamError(res, err) {
|
||||||
|
if (err instanceof ConcurrencyOverflowError) {
|
||||||
|
return jsonResponse(res, 429, { error: { message: sanitizeError(err.message), type: "rate_limit_error" } }, { "Retry-After": String(err.retryAfter) });
|
||||||
|
}
|
||||||
|
return jsonResponse(res, 500, { error: { message: sanitizeError(err.message), type: "proxy_error" } });
|
||||||
|
}
|
||||||
|
|
||||||
function sendSSE(res, data, hb) {
|
function sendSSE(res, data, hb) {
|
||||||
hb?.reset();
|
hb?.reset();
|
||||||
res.write(`data: ${JSON.stringify(data)}\n\n`);
|
res.write(`data: ${JSON.stringify(data)}\n\n`);
|
||||||
@@ -1688,7 +1874,9 @@ function applySettingUpdate(key, value) {
|
|||||||
|
|
||||||
switch (key) {
|
switch (key) {
|
||||||
case "timeout": TIMEOUT = value; break;
|
case "timeout": TIMEOUT = value; break;
|
||||||
case "maxConcurrent": MAX_CONCURRENT = value; break;
|
// FIX ⑥: keep the -p wait-queue semaphore's limit in sync with the runtime MAX_CONCURRENT
|
||||||
|
// so a /settings change to maxConcurrent actually changes how many claude procs run at once.
|
||||||
|
case "maxConcurrent": MAX_CONCURRENT = value; claudeSemaphore.limit = Math.max(1, value); break;
|
||||||
case "sessionTTL": SESSION_TTL = value; break;
|
case "sessionTTL": SESSION_TTL = value; break;
|
||||||
case "maxPromptChars": MAX_PROMPT_CHARS = value; break;
|
case "maxPromptChars": MAX_PROMPT_CHARS = value; break;
|
||||||
case "cacheTTL": CACHE_TTL = value; break;
|
case "cacheTTL": CACHE_TTL = value; break;
|
||||||
@@ -1897,7 +2085,7 @@ async function handleChatCompletions(req, res) {
|
|||||||
try { res.end(); } catch {}
|
try { res.end(); } catch {}
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
return jsonResponse(res, 500, { error: { message: sanitizeError(err.message), type: "proxy_error" } });
|
return respondUpstreamError(res, err);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1914,8 +2102,9 @@ async function handleChatCompletions(req, res) {
|
|||||||
try { res.end(); } catch {}
|
try { res.end(); } catch {}
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
// Sanitize error: strip internal file paths before sending to client
|
// Sanitize error: strip internal file paths before sending to client.
|
||||||
jsonResponse(res, 500, { error: { message: sanitizeError(err.message), type: "proxy_error" } });
|
// FIX ⑥: ConcurrencyOverflowError → 429 + Retry-After; all other errors → 500 (unchanged).
|
||||||
|
respondUpstreamError(res, err);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -2076,6 +2265,31 @@ const server = createServer(async (req, res) => {
|
|||||||
circuitBreaker: "disabled",
|
circuitBreaker: "disabled",
|
||||||
sessions: sessionList,
|
sessions: sessionList,
|
||||||
recentErrors: recentErrors.slice(-5),
|
recentErrors: recentErrors.slice(-5),
|
||||||
|
// ── FIX ③ spawn-home isolation surface — ADDITIVE (default -p/stream-json path) ──
|
||||||
|
// Lets the operator confirm the latency-fix isolation is active without inspecting logs.
|
||||||
|
// NEVER includes the token. mode: "isolated-scratch-home" | "real-home". home is the
|
||||||
|
// scratch HOME path when isolated (null otherwise). For TUI_MODE the -p path is unused,
|
||||||
|
// so report it as disabled.
|
||||||
|
spawn: (() => {
|
||||||
|
if (TUI_MODE) return { mode: "tui (default -p path unused)", isolated: false, home: null };
|
||||||
|
const shm = getSpawnHomeMode();
|
||||||
|
return {
|
||||||
|
mode: shm.isolated ? "isolated-scratch-home" : "real-home",
|
||||||
|
isolated: shm.isolated,
|
||||||
|
home: shm.isolated ? shm.home : null,
|
||||||
|
reason: shm.reason,
|
||||||
|
};
|
||||||
|
})(),
|
||||||
|
// ── FIX ⑥ -p concurrency wait-queue surface — ADDITIVE ──
|
||||||
|
// inflight/queued are live; queueRejections is cumulative (also in stats.queueRejections).
|
||||||
|
// Lets the operator see backpressure instead of guessing from opaque 500s.
|
||||||
|
concurrency: {
|
||||||
|
maxConcurrent: MAX_CONCURRENT,
|
||||||
|
maxQueue: claudeSemaphore.maxQueue,
|
||||||
|
inflight: claudeSemaphore.inflight,
|
||||||
|
queued: claudeSemaphore.queued,
|
||||||
|
queueRejections: stats.queueRejections,
|
||||||
|
},
|
||||||
// ── TUI observability (audit C-5) — ADDITIVE block (ADR 0007 PR-B amendment) ──
|
// ── TUI observability (audit C-5) — ADDITIVE block (ADR 0007 PR-B amendment) ──
|
||||||
// /health is a grandfathered B.2 endpoint (ADR 0006). This block is NEW fields only;
|
// /health is a grandfathered B.2 endpoint (ADR 0006). This block is NEW fields only;
|
||||||
// every existing field above is byte-identical → behaviour-preserving for existing
|
// every existing field above is byte-identical → behaviour-preserving for existing
|
||||||
@@ -2364,7 +2578,7 @@ server.listen(PORT, BIND_ADDRESS, () => {
|
|||||||
console.log(`Architecture: on-demand spawning (no pool)`);
|
console.log(`Architecture: on-demand spawning (no pool)`);
|
||||||
console.log(`Models: ${MODELS.map((m) => m.id).join(", ")}`);
|
console.log(`Models: ${MODELS.map((m) => m.id).join(", ")}`);
|
||||||
console.log(`Claude binary: ${CLAUDE}`);
|
console.log(`Claude binary: ${CLAUDE}`);
|
||||||
console.log(`Timeout: ${TIMEOUT / 1000}s | Max concurrent: ${MAX_CONCURRENT}`);
|
console.log(`Timeout: ${TIMEOUT / 1000}s | Max concurrent: ${MAX_CONCURRENT} | Queue: ${CLAUDE_MAX_QUEUE} (429 on overflow)`);
|
||||||
console.log(`Circuit breaker: disabled`);
|
console.log(`Circuit breaker: disabled`);
|
||||||
console.log(`Tools: ${SKIP_PERMISSIONS ? "all (skip-permissions)" : ALLOWED_TOOLS.join(", ")}`);
|
console.log(`Tools: ${SKIP_PERMISSIONS ? "all (skip-permissions)" : ALLOWED_TOOLS.join(", ")}`);
|
||||||
console.log(`Sessions: TTL=${SESSION_TTL / 1000}s`);
|
console.log(`Sessions: TTL=${SESSION_TTL / 1000}s`);
|
||||||
@@ -2376,6 +2590,15 @@ server.listen(PORT, BIND_ADDRESS, () => {
|
|||||||
if (NO_CONTEXT) console.log(`Context: suppressed (CLAUDE_NO_CONTEXT=true — no CLAUDE.md, no auto-memory)`);
|
if (NO_CONTEXT) console.log(`Context: suppressed (CLAUDE_NO_CONTEXT=true — no CLAUDE.md, no auto-memory)`);
|
||||||
if (CACHE_TTL > 0) console.log(`Cache: enabled (TTL=${CACHE_TTL / 1000}s)`);
|
if (CACHE_TTL > 0) console.log(`Cache: enabled (TTL=${CACHE_TTL / 1000}s)`);
|
||||||
else console.log(`Cache: disabled (set CLAUDE_CACHE_TTL to enable)`);
|
else console.log(`Cache: disabled (set CLAUDE_CACHE_TTL to enable)`);
|
||||||
|
// FIX ③: announce default-path (-p/stream-json) spawn-home isolation mode (never logs the token).
|
||||||
|
if (!TUI_MODE) {
|
||||||
|
const shm = getSpawnHomeMode();
|
||||||
|
if (shm.isolated) {
|
||||||
|
console.log(`Spawn home: isolated-scratch-home (${shm.home}, cwd-neutral, env-token auth) — fast path`);
|
||||||
|
} else {
|
||||||
|
console.log(`Spawn home: real-home (${shm.reason}) — set CLAUDE_CODE_OAUTH_TOKEN for the isolated fast path`);
|
||||||
|
}
|
||||||
|
}
|
||||||
if (TUI_MODE) {
|
if (TUI_MODE) {
|
||||||
console.warn(`⚠️ TUI-mode ON — single-user only; do NOT enable on a multi-user OCP (guest prompts would run claude with operator filesystem access). See ADR 0007.`);
|
console.warn(`⚠️ TUI-mode ON — single-user only; do NOT enable on a multi-user OCP (guest prompts would run claude with operator filesystem access). See ADR 0007.`);
|
||||||
const tuiAuth = process.env.CLAUDE_CODE_OAUTH_TOKEN
|
const tuiAuth = process.env.CLAUDE_CODE_OAUTH_TOKEN
|
||||||
|
|||||||
@@ -2046,6 +2046,45 @@ await asyncTest("wait queue is bounded — run() rejects with tui_queue_full whe
|
|||||||
assert.equal(sem.inflight, 0);
|
assert.equal(sem.inflight, 0);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
console.log("\n-p concurrency wait-queue (FIX ⑥ — same TuiSemaphore reused for the -p path):");
|
||||||
|
|
||||||
|
// server.mjs reuses TuiSemaphore as `claudeSemaphore = new TuiSemaphore(MAX_CONCURRENT,
|
||||||
|
// { maxQueue: CLAUDE_MAX_QUEUE })` and wraps acquire()/release() in acquireClaudeSlot(). These
|
||||||
|
// tests assert the contract that the 429-mapping depends on: requests beyond the limit QUEUE
|
||||||
|
// (not reject), only an overflow past the queue rejects (→ HTTP 429 in server.mjs), and a
|
||||||
|
// released slot is reusable (the #37/#40 slot-leak guard — no leak on normal completion).
|
||||||
|
await asyncTest("FIX ⑥: requests beyond MAX_CONCURRENT queue, not reject (limit=1, queue=1)", async () => {
|
||||||
|
const sem = new TuiSemaphore(1, { maxQueue: 1 }); // mirrors CLAUDE_MAX_CONCURRENT=1, CLAUDE_MAX_QUEUE=1
|
||||||
|
const g1 = deferred();
|
||||||
|
const inflightP = sem.run(async () => { await g1.p; }); // request 1 — holds the only slot
|
||||||
|
await new Promise((r) => setImmediate(r));
|
||||||
|
assert.equal(sem.inflight, 1, "req1 inflight");
|
||||||
|
const queuedP = sem.run(async () => {}); // request 2 — WAITS (queued), does NOT reject
|
||||||
|
await new Promise((r) => setImmediate(r));
|
||||||
|
assert.equal(sem.queued, 1, "req2 queued (waits), not rejected → would be served, not 429");
|
||||||
|
// request 3 — queue full → reject (server.mjs maps this single case to 429 + Retry-After)
|
||||||
|
await assert.rejects(sem.run(async () => {}), /tui_queue_full|queue/, "req3 overflows → reject (→429)");
|
||||||
|
g1.resolve();
|
||||||
|
await inflightP; await queuedP;
|
||||||
|
assert.equal(sem.inflight, 0, "all slots released after drain (no leak)");
|
||||||
|
assert.equal(sem.queued, 0, "queue fully drained");
|
||||||
|
});
|
||||||
|
|
||||||
|
await asyncTest("FIX ⑥: slot released on normal completion is immediately reusable (no #37/#40 leak)", async () => {
|
||||||
|
const sem = new TuiSemaphore(1, { maxQueue: 16 }); // mirrors default CLAUDE_MAX_QUEUE=16
|
||||||
|
for (let i = 0; i < 5; i++) {
|
||||||
|
await sem.run(async () => { /* a normal, completing turn */ });
|
||||||
|
assert.equal(sem.inflight, 0, `slot released after turn ${i}`);
|
||||||
|
}
|
||||||
|
// Prove the limit still binds after many acquire/release cycles.
|
||||||
|
const g = deferred();
|
||||||
|
const held = sem.run(async () => { await g.p; });
|
||||||
|
await new Promise((r) => setImmediate(r));
|
||||||
|
assert.equal(sem.inflight, 1, "limit still enforced after reuse cycles");
|
||||||
|
g.resolve(); await held;
|
||||||
|
assert.equal(sem.inflight, 0);
|
||||||
|
});
|
||||||
|
|
||||||
console.log("\nTUI drift observability (C-5):");
|
console.log("\nTUI drift observability (C-5):");
|
||||||
|
|
||||||
test("recordTuiEntrypoint: observed 'cli' is NOT a mismatch and sets lastEntrypoint", () => {
|
test("recordTuiEntrypoint: observed 'cli' is NOT a mismatch and sets lastEntrypoint", () => {
|
||||||
|
|||||||
Reference in New Issue
Block a user