mirror of
https://github.com/dtzp555-max/ocp.git
synced 2026-07-21 21:15:09 +00:00
Backlog item #3 of docs/plans/2026-07-13-tui-latency/README.md. Every TUI request currently cold-boots a tmux+claude pane. This adds an OPT-IN pool of pre-booted panes. Recorded as ADR 0008 (docs/adr/0008-tui-warm-pane-pool.md), which extends ADR 0007. MEASURED (this host, Sonnet 4.6, --effort low, through a real OCP instance; a sample counts only if HTTP 200 AND the body carries the demanded marker): pool off (main code) n= 6 p50 10.17s [9164 9499 9760 10572 10774 11281] pool on, warm hits n=12 p50 6.00s [5286 5289 5520 5584 5621 5969 6040 6098 6280 7846 8036 11053] pool on, warm hits (post- n= 6 p50 5.62s [4729 4753 5236 6004 7548 9548] review-fix re-run) -> -4.17s / -41%. 12 hits / 1 miss / 0 bootFailures over 13 requests (and 6/1/0 on the post-fix re-run). Robust to counting the miss: n=13 p50 -> -40.6%. The plan doc predicted only -1.0s (the boot). It is ~4.2s because the cold path also pays ~2.9s INSIDE the first turn beyond claude's own reported turn_duration — post- input-bar init that an idle pane has already finished. Phase decomposition of the cold path (n=6 medians): prep 2ms | tmux spawn 27ms | boot->input-ready 1232ms | paste 8ms | paste-verify 426ms | submit->terminal 8458ms | teardown 8ms = 10162ms total, vs native turn_duration 5539ms => 4490ms of OCP-side overhead, of which the pool recovers ~1.26s of boot and ~2.9s of in-claude cold start. (The 426ms paste-verify is one 400ms poll tick; a real paste lands in ~80ms. Not addressed here — separate item.) DESIGN - SINGLE-USE panes. A pooled pane serves exactly ONE turn, then is killed and replaced in the background. Each carries its OWN fresh --session-id fixed at boot, so one session still holds one exchange. This is what keeps transcript.mjs's extractLatestAssistantText correct; its warning about a future warm pool reusing a session is answered in-place (comment updated) and left standing for anyone who later wants a second turn on a pane — that would be a cross-request TEXT LEAK and needs user-line scoping in the transcript reader first. - Pool keyed by model; --model is fixed at spawn. A miss falls back to the cold path with zero behaviour change. The pool warms the most recently requested model, so the first request after start (and after a model switch) is always a cold miss. - REAPER COEXISTENCE (the crux). An idle warm pane IS ours, and the periodic sweep runs precisely when we are idle. reapStaleTuiSessions() takes a `spare` set of EXACT live session names, and server.mjs DRAINS the pool immediately before the sweep: 1. a live pooled pane is never reaped — INCLUDING one still BOOTING (see below); 2. an orphaned pooled pane IS still reaped — membership is by exact name from a live in-memory registry, never by name shape, so a pane from a dead process generation has nothing claiming it. Omitting `spare` reaps MORE, never less (fail-safe); 3. kill-server is suppressed while any pane is spared — hence the drain, so the sweep still flushes <defunct> claude zombies (the only mechanism that can). - THE POOL TRACKS ITS IN-FLIGHT BOOT BY NAME, NOT AS A COUNT. bootTuiPane creates the tmux session SYNCHRONOUSLY and only then waits up to POOL_BOOT_MS (20s) for the input bar, so a pooled session can be LIVE for ~20s before its boot resolves. Tracking boots as a count meant the pool could not name that session, which caused two real bugs (found in review, reproduced, fixed, and now regression-tested): * the reap sweep KILLED the booting pane (it could not be spared), left the pool empty with nothing scheduled, and logged the exact tui_pool_boot_failed WARN operators are told to alert on — for a completely healthy drain; * graceful shutdown ORPHANED a live authenticated idle `claude`: gracefulShutdown calls process.exit(0) in the SAME TICK as the drain (TUI panes are tmux children, so activeProcesses is empty and the wait-for-children path exits immediately), so cleanup deferred to a .then() never ran. Fix: the pool mints each pane's identity up front ({sessionId, name}) and holds it in _bootingPane. liveNames() includes it; drain() kills it SYNCHRONOUSLY. A generation counter distinguishes "cancelled by us" from "genuinely failed", so a drain never inflates bootFailures and resume() reliably starts a fresh boot. Deriving the name from the session-id also makes `tmux ls` correlate to the transcript file. - SLOT ACCOUNTING. Refill boots take NO TuiSemaphore slot (those bound real turns and would be starved); they cannot leak one either, since they never hold one. Refills are SERIALIZED, one boot at a time — live at size=2, two cold boots racing an in-flight turn overran the readiness cap and a refill was discarded. A genuinely failed boot does not re-kick the chain (backoff; a broken claude must not respawn forever). Background boots get a more generous readiness cap (POOL_BOOT_MS = 5x BOOT_MS): BOOT_MS is tight because a client is blocked on it, which is not true of a pre-boot. - BOUNDED COST. A warm pane is a LIVE idle claude process held whether or not a request arrives. Peak processes = pool size + OCP_TUI_MAX_CONCURRENT + 1 booting replacement. Size clamped to POOL_MAX_SIZE=4; garbage values disable rather than guess. Panes have a 10-min TTL and a health check at hand-out (dead/degraded pane => miss, never a hang). Missing collaborators throw at CONSTRUCTION, not on a live request (refill() is called synchronously from the request path). DEFAULT OFF (OCP_TUI_POOL_SIZE=0). This is a stable production path and the pool holds standing processes, so the operator opts in. With the pool off, runTuiTurn takes the IDENTICAL code path as before (the `pool ? pool.acquire() : null` branch yields null, and tuiPool is null so no observer is attached and no new log line is emitted) — that is what establishes the default path is unchanged. A pool-off control run (n=6, p50 9.40s) is consistent with the 10.17s baseline but had 2/6 samples >12s, so it is corroboration, NOT proof: n=6 cannot establish "unregressed" on its own. The code-path equivalence can. BANNER: NO SPAWN ARGUMENT CHANGED. buildTuiCmd is byte-identical to main (verified by extracting the function body from both revisions and comparing). Live banner captured from two real POOLED panes anyway: "Sonnet 4.6 with low effort · Claude Max" — the subscription pool, never "API Usage Billing". /health: `tui.pool` added (null when off), incl. `cancelled` (boots WE killed — not a fault; do not alert on it). The tui block is ADR-0007-owned and post-dates ADR 0006's v3.16.4 grandfather snapshot; the addition is purely additive — every pre-existing key keeps a byte-identical value. Authorization recorded in ADR 0008. ALIGNMENT: Class B / ADR 0007 + ADR 0008 (OCP-owned TUI spawn machinery). cli.js does NOT perform this operation — there is no cli.js citation and none is required: this is not an Anthropic API surface, it is OCP's own process management around the claude CLI, exactly as the existing tmux session lifecycle and reaper already are (ALIGNMENT.md Rule 2). TESTS: 294 passed / 0 failed (was 267). +27 covering acquire/hit/miss, single-use (a pane is never handed out twice), bounded + serialized refill, TTL + health-check drops, model retarget, drain/resume, boot-failure backoff, identity linkage, all three reaper invariants incl. post-drain kill-server restoration, and — the coverage gap that let both bugs ship — FIVE mid-boot tests: the booting pane is nameable/spareable, the sweep's drain kills it and resume starts a fresh boot with no bogus WARN, shutdown kills it synchronously (asserted WITHOUT awaiting, since process.exit runs in the same tick), a stale settle cannot clear a newer boot's slot, and a model switch cancels an in-flight boot for the old model. Live verification (temporary 20s reap interval, reverted): sweep drained both panes -> reaped -> refilled with NEW panes; a foreign tmux session survived untouched; with no foreign session kill-server fired and the pool still recovered and served the next request. Both review bugs reproduced against a PRIVATE tmux server (-L pr3repro, so the reaper's internal kill-server could not touch the host) before and after the fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
159 lines
8.2 KiB
JavaScript
159 lines
8.2 KiB
JavaScript
// TUI-path concurrency limiter (audit finding C-4).
|
||
//
|
||
// WHY THIS EXISTS, SEPARATE FROM server.mjs's MAX_CONCURRENT:
|
||
// The global MAX_CONCURRENT gate lives in spawnClaudeProcess() (the -p / stream-json
|
||
// path). callClaudeTui() NEVER calls spawnClaudeProcess — it calls runTuiTurn(), which
|
||
// boots a full interactive `claude` inside a fresh tmux session. So nothing bounded the
|
||
// TUI path: N concurrent TUI requests spawned N simultaneous cold-boot tmux+claude
|
||
// processes. On a small host (a Pi 4 serving a family) a burst of ~5 is an OOM risk, and
|
||
// it also multiplies subscription rate-limit pressure. This is an INDEPENDENT limiter for
|
||
// the TUI path that mirrors MAX_CONCURRENT's intent without coupling to it (the two pools
|
||
// are different shapes: a stream-json spawn is cheap and fast; a TUI turn is a heavy
|
||
// cold-boot + up to 120s wallclock).
|
||
//
|
||
// QUEUE vs REJECT: we QUEUE (await a slot), mirroring the spirit of MAX_CONCURRENT's
|
||
// intent not to drop requests, rather than rejecting immediately. To avoid unbounded
|
||
// memory growth from a runaway client, the wait queue itself is bounded by maxQueue
|
||
// (default: a generous multiple of the concurrency limit). When the queue is full, run()
|
||
// rejects with a tui_queue_full error (the caller surfaces it as a 503) — a deterministic
|
||
// backpressure signal rather than silent OOM.
|
||
//
|
||
// Pure + importable so test-features.mjs can assert the bound directly (no server boot).
|
||
|
||
// Thrown by acquire() when the caller-supplied AbortSignal fires before a slot was granted
|
||
// (audit finding F2 — a client that disconnects while queued must never receive a slot; the
|
||
// queue entry is spliced out, not just flagged, so `queued` accounting stays exact). Distinct
|
||
// `name` lets callers (server.mjs acquireClaudeSlot) tell "client went away" apart from
|
||
// "queue is full" without string-matching the message.
|
||
export class SemaphoreAbortError extends Error {
|
||
constructor(message) { super(message); this.name = "SemaphoreAbortError"; }
|
||
}
|
||
|
||
export class TuiSemaphore {
|
||
// limit: max concurrent slots. maxQueue: max waiters before run() rejects with backpressure.
|
||
constructor(limit, { maxQueue } = {}) {
|
||
this.limit = Math.max(1, parseInt(limit, 10) || 1);
|
||
// Default queue cap: 32× the limit. Large enough that real family-burst traffic never
|
||
// hits it, small enough that a pathological flood can't grow the queue without bound.
|
||
this.maxQueue = Number.isFinite(maxQueue) ? maxQueue : this.limit * 32;
|
||
this._inflight = 0;
|
||
this._waiters = []; // FIFO queue of resolve callbacks waiting for a slot
|
||
}
|
||
|
||
get inflight() { return this._inflight; }
|
||
get queued() { return this._waiters.length; }
|
||
|
||
// Runtime-adjust the concurrency limit (audit finding F1 — a PATCH /settings maxConcurrent
|
||
// change must actually take effect, not just be ignored until every currently-inflight task
|
||
// happens to finish). Lowering the limit is handled lazily by release() (see below) — it
|
||
// simply stops re-granting until inflight drains under the new, lower limit. Raising the
|
||
// limit has immediate headroom, so we wake as many queued waiters as now fit.
|
||
setLimit(limit) {
|
||
this.limit = Math.max(1, parseInt(limit, 10) || 1);
|
||
while (this._inflight < this.limit && this._waiters.length > 0) {
|
||
const next = this._waiters.shift();
|
||
this._inflight++;
|
||
next();
|
||
}
|
||
}
|
||
|
||
// Acquire a slot. Resolves once a slot is free (immediately if under the limit, otherwise
|
||
// when an in-flight task releases). Rejects synchronously-ish if the wait queue is full.
|
||
// `signal` (optional AbortSignal, F2) lets the caller cancel a QUEUED wait — e.g. wired to
|
||
// a client's socket "close" event so a request that disconnects before a slot is granted
|
||
// is removed from the queue instead of eventually being handed a slot for a dead socket.
|
||
// If `signal` is already aborted, reject immediately without ever touching the queue.
|
||
acquire(signal) {
|
||
if (signal?.aborted) {
|
||
return Promise.reject(new SemaphoreAbortError("acquire aborted before requesting a slot"));
|
||
}
|
||
if (this._inflight < this.limit) {
|
||
this._inflight++;
|
||
return Promise.resolve();
|
||
}
|
||
if (this._waiters.length >= this.maxQueue) {
|
||
return Promise.reject(new Error(
|
||
`tui_queue_full: TUI concurrency limit (${this.limit}) reached and wait queue ` +
|
||
`(${this.maxQueue}) is full`));
|
||
}
|
||
return new Promise((resolve, reject) => {
|
||
let waiter; // the FIFO entry — captured so onAbort can find + splice exactly this one
|
||
const onAbort = () => {
|
||
const idx = this._waiters.indexOf(waiter);
|
||
if (idx === -1) return; // already granted a slot (shifted out by release()/setLimit) — too late to cancel
|
||
this._waiters.splice(idx, 1); // remove, not just flag — keeps `queued` accounting exact
|
||
reject(new SemaphoreAbortError("acquire aborted while queued"));
|
||
};
|
||
waiter = () => {
|
||
signal?.removeEventListener("abort", onAbort);
|
||
resolve();
|
||
};
|
||
signal?.addEventListener("abort", onAbort, { once: true });
|
||
this._waiters.push(waiter);
|
||
});
|
||
}
|
||
|
||
// Release a slot. Always frees the caller's own slot first, then re-grants it to the next
|
||
// waiter ONLY if the (post-decrement) inflight count is still under the current limit (F1
|
||
// fix). This is what makes a runtime-lowered limit actually bite: if the limit was lowered
|
||
// while over-subscribed, releases stop re-granting and inflight drains toward the new limit
|
||
// instead of a freed slot being handed straight back out at the old, higher occupancy.
|
||
release() {
|
||
if (this._inflight > 0) this._inflight--;
|
||
if (this._inflight < this.limit) {
|
||
const next = this._waiters.shift();
|
||
if (next) {
|
||
this._inflight++;
|
||
next();
|
||
}
|
||
}
|
||
}
|
||
|
||
// Run fn() under one slot. Releases in a finally so a throw (PR-A's honesty gates,
|
||
// wallclock truncation, paste-not-landed, tmux spawn failure) NEVER leaks a slot.
|
||
// `signal` (optional, F2) is forwarded to acquire() so a queued run() can be cancelled.
|
||
async run(fn, signal) {
|
||
await this.acquire(signal);
|
||
try {
|
||
return await fn();
|
||
} finally {
|
||
this.release();
|
||
}
|
||
}
|
||
}
|
||
|
||
// ── TUI drift observability (audit C-5) — pure helpers, importable for testing ──
|
||
|
||
// Record an observed cc_entrypoint into the (mutable) tuiStats counter. Sets lastEntrypoint
|
||
// unconditionally and increments entrypointMismatches when the spawn was supposed to be
|
||
// subscription-pool ("cli") but the transcript reported something else (a silent drift to
|
||
// the metered Agent SDK pool — the audit's top risk after the 6/15 billing flip).
|
||
// Returns true iff this observation was a mismatch (so the caller can also emit a log).
|
||
export function recordTuiEntrypoint(tuiStats, observed, expectedMode = "cli") {
|
||
tuiStats.lastEntrypoint = observed ?? null;
|
||
const mismatch = expectedMode === "cli" && observed !== "cli";
|
||
if (mismatch) tuiStats.entrypointMismatches++;
|
||
return mismatch;
|
||
}
|
||
|
||
// Build the additive /health `tui` block (ADR 0007 PR-B amendment). Pure: given the
|
||
// config + live counters, returns the exact object embedded in /health. New fields only —
|
||
// behaviour-preserving for existing /health consumers (grandfathered B.2 under ADR 0006).
|
||
//
|
||
// `pool` (optional, warm pane pool — lib/tui/pool.mjs): a TuiPanePool, or null/undefined
|
||
// when the pool is off (the default). Reported as `pool: null` when off so the block's
|
||
// shape stays stable, and as the pool's stats (size / warm / hits / misses / …) when on —
|
||
// the operator's window onto both the hit rate and the standing idle-process cost.
|
||
export function buildTuiHealthBlock({ enabled, entrypointMode, maxConcurrent }, tuiStats, semaphore, pool = null) {
|
||
return {
|
||
enabled,
|
||
entrypointMode, // cli | auto | off
|
||
lastEntrypoint: tuiStats.lastEntrypoint, // last observed cc_entrypoint, or null
|
||
entrypointMismatches: tuiStats.entrypointMismatches,
|
||
inflight: semaphore.inflight, // current concurrent TUI turns
|
||
queued: semaphore.queued, // turns waiting for a slot
|
||
maxConcurrent,
|
||
pool: pool ? pool.stats() : null, // warm pane pool, or null when disabled
|
||
};
|
||
}
|