diff --git a/README.md b/README.md index d182d78..1acd23c 100644 --- a/README.md +++ b/README.md @@ -960,6 +960,7 @@ See [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) and ADR 0007 PR- | `OCP_TUI_ENTRYPOINT` | `cli` | (TUI-mode) Billing-classifier labeling: `cli` (default) pins `cc_entrypoint=cli` deterministically; `auto` lets claude self-classify via TTY detection; `off` leaves the inherited env untouched. Honest only when the spawn is a genuine interactive PTY — see ADR 0007. | | `OCP_TUI_EFFORT` | `low` | (TUI-mode) Effort level passed to the interactive `claude` as an explicit `--effort` flag: `low` (default), `medium`, `high`, `xhigh`, `max`, or `inherit` to omit the flag (the pre-flag behaviour: the pane inherits a HOME-dependent effort — the operator's `~/.claude/settings.json` `effortLevel` in real-home mode, claude's built-in default in env-token scratch mode). Explicit `low` cuts measured TTFT p50 by ~40% and collapses run-to-run variance ~15× versus an inherited `xhigh` (see `docs/plans/2026-07-13-tui-latency/`); proxied requests rarely benefit from extended thinking. Banner-verified to stay on the subscription pool (`· Claude Max`). An invalid value logs a warning and falls back to `low`. | | `OCP_TUI_MAX_CONCURRENT` | `2` | (TUI-mode) Max concurrent interactive TUI turns. **Independent** of `CLAUDE_MAX_CONCURRENT` (which bounds the `-p`/stream-json path; TUI never uses it). A TUI turn is heavy (per-request cold-boot of tmux+claude + up to `CLAUDE_TUI_WALLCLOCK_MS` wallclock), so the default is low to keep small hosts (e.g. a Pi 4) alive under a burst. Excess turns **queue** (bounded); a full queue yields a 503. See ADR 0007 PR-B amendment. | +| `OCP_TUI_POOL_SIZE` | `0` (off) | (TUI-mode) Number of **pre-booted warm `claude` panes** kept ready, so a request does not pay the cold boot. `0` disables the pool entirely — the request path is then exactly the cold-boot path. Max `4`; an unparseable value disables it rather than guessing. **Measured on a Mac mini (Sonnet 4.6, `--effort low`): end-to-end p50 `10.17s` (n=6, pool off) → `6.00s` (n=12 warm hits) — −4.2 s / −41%** — the pool recovers both the ~1.2 s boot *and* ~2.9 s of post-input-bar init that a pane which has been idle a moment has already finished. **Cost:** each warm pane is a *live idle `claude` process* held whether or not a request ever arrives (peak processes ≈ pool size + `OCP_TUI_MAX_CONCURRENT` + 1 booting replacement) — which is why it is opt-in. Panes are **single-use**: one turn, then killed and replaced in the background. The **first request after start (and after any model switch) is always a cold miss** — the pool warms the most recently requested model, since OCP cannot know which model the next caller wants. See `docs/plans/2026-07-13-tui-latency/`. | | `OCP_SKIP_AUTH_TEST` | *(unset)* | When `=1`, skip the `claude -p` auth probe during `setup.mjs`. After 2026-06-15 this probe draws from the Agent SDK credit pool; set this to avoid burning a metered credit on re-installs or `ocp update` runs. Auth is validated at the first real request. | | `OCP_TUI_FULL_TOOLS` | *(unset)* | (TUI-mode, **single-user only**) When `=1`, grant the interactive session the **same tool surface as the `-p` path** — `--allowedTools` (+ optional `--mcp-config`, read from `CLAUDE_ALLOWED_TOOLS` / `CLAUDE_MCP_CONFIG`) — instead of the default MCP-walled, built-in-tools-only set. Lets a trusted single-operator TUI deployment run a **tool-using / MCP agent** (e.g. an OpenClaw assistant) on the subscription pool. Safe because TUI **refuses to boot under `AUTH_MODE=multi`** (hard exit) — no guest key can ever reach the TUI path, so this gate cannot expose tools to an untrusted caller. (Under `AUTH_MODE=shared` + `OCP_TUI_ALLOW_LAN=1`, anyone holding the single shared key reaches it — that is the existing TUI trust model, unchanged.) Note: `--dangerously-skip-permissions` / `CLAUDE_SKIP_PERMISSIONS` is **not** supported for TUI — claude v2.1.x shows an interactive bypass-acceptance screen in headless tmux that cannot be answered, bricking the pane. Use scratch-home `settings.json` `additionalDirectories` instead. See [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) and ADR 0007. | @@ -1040,6 +1041,7 @@ Then restart OCP. At boot you will see (with the env token set, isolated home au - **Stale tmux sessions are reaped.** The pane's `claude` is a child of the tmux server (not OCP), so OCP cannot reap it directly; `claude` zombies can otherwise accumulate as `` over a long-running host. OCP reaps them at boot and on a 15-min idle sweep by issuing `tmux kill-server` — but **only when no foreign tmux session remains** (it never disrupts a co-hosted `olp-tui-*` instance). See ADR 0007 PR-C amendment. - **Default path unchanged.** Unset `CLAUDE_TUI_MODE` and restart → `callClaude` / `callClaudeStreaming` are used again, byte-for-byte identical to today. - **Concurrency is bounded separately.** TUI turns are heavy (per-request cold-boot + long wallclock), so the TUI path has its own limiter — `OCP_TUI_MAX_CONCURRENT` (default `2`), independent of `CLAUDE_MAX_CONCURRENT`. Excess turns queue; a full queue returns a 503. Tune it up only on a host that can run more interactive `claude` sessions at once. +- **Optional warm pane pool (`OCP_TUI_POOL_SIZE`, default off).** Pre-boots panes so a request skips the cold boot — measured p50 `10.17s` → `6.00s` (−41%). Pooled panes are **single-use** (one turn, then killed and replaced in the background), each carrying its own fresh `--session-id`, so one session still means one exchange and no earlier-turn text can leak into a later answer. They are named `ocp-tui--p` and coexist with the reaper by design: the sweep **drains the pool first**, then reaps (so `kill-server` still flushes `` zombies), then the pool refills in the background. Drain→reap→resume is synchronous, so no request can land mid-sweep; a request arriving while the pool is still re-booting simply misses it and cold-boots. A live pooled pane is never reaped — **including one that is still booting**, whose tmux session already exists — while an *orphaned* one (left by a previous process generation) still is. ### ⚠️ Latency: TUI mode has a ~6-second floor, and it is immovable @@ -1082,12 +1084,26 @@ a sub-5-second budget. Full measurements and methodology: "entrypointMismatches": 0, // count of cli-expected-but-got-other turns — ALERT if this climbs "inflight": 1, // TUI turns running right now "queued": 0, // TUI turns waiting for a concurrency slot - "maxConcurrent": 2 // OCP_TUI_MAX_CONCURRENT + "maxConcurrent": 2, // OCP_TUI_MAX_CONCURRENT + "pool": { // warm pane pool — null when OCP_TUI_POOL_SIZE=0 (the default) + "size": 2, // target warm panes (OCP_TUI_POOL_SIZE) + "warm": 2, // panes ready right now — each is a LIVE idle claude process + "booting": 0, // replacement panes currently pre-booting + "model": "claude-sonnet-4-6", // the model being warmed (the most recently requested one) + "hits": 12, // requests served by a warm pane + "misses": 1, // requests that fell back to the cold boot (the 1st is always one) + "boots": 14, // panes successfully pre-booted + "bootFailures": 0, // pre-boots that genuinely never reached the input bar — WATCH this + "cancelled": 4, // in-flight boots OCP killed on purpose (drain / model switch) — not faults + "dropped": 8 // panes discarded unused (drain sweep / expired / unhealthy) + } } ``` Alert on `entrypointMismatches > 0` (or `lastEntrypoint !== "cli"`): it means a turn drew from the metered Agent SDK pool instead of the subscription. `inflight` / `queued` show how close the TUI path is to its concurrency cap. +With the pool on, `hits` / `misses` is the hit rate (a steady single-model consumer should sit near 100% after the first request), and `warm` is your standing idle-process cost. A climbing `bootFailures` means panes are not reaching their input bar — the pool then degrades safely to the cold path, but latency reverts to the un-pooled numbers. `cancelled` counts boots OCP killed *on purpose* (a drain, a model switch) and is **not** a fault signal — do not alert on it. A steadily climbing `dropped` is likewise normal: the 15-min reap sweep drains and re-boots the pool on every tick so `kill-server` can still flush `` zombies. + ### Kill-switch ```bash diff --git a/docs/adr/0008-tui-warm-pane-pool.md b/docs/adr/0008-tui-warm-pane-pool.md new file mode 100644 index 0000000..3fa7430 --- /dev/null +++ b/docs/adr/0008-tui-warm-pane-pool.md @@ -0,0 +1,208 @@ +# ADR 0008 — TUI Warm Pane Pool + +**Date:** 2026-07-13 +**Status:** Proposed +**Extends:** [ADR 0007](0007-tui-interactive-mode.md) (TUI interactive mode). This ADR does not +change ADR 0007's billing-pool argument, security posture, or kill-switch — it adds a latency +optimization *inside* the TUI spawn machinery ADR 0007 owns. + +--- + +## Context + +TUI mode (ADR 0007) serves every request by cold-booting a fresh `tmux` session running an +interactive `claude`, submitting one prompt, reading the native transcript, and killing the +session. That cold boot is paid on **every** request. + +[`docs/plans/2026-07-13-tui-latency/`](../plans/2026-07-13-tui-latency/README.md) measured the +TUI path and listed a warm pane pool as backlog item #3, costed at "**~1.0 s**" (the observed +boot-to-input-bar time). Instrumenting the real request path showed that estimate is **~4× +too low**. Phase decomposition of the cold path (n=6 medians, Sonnet 4.6, `--effort low`, +through a real OCP instance): + +| Phase | Median | +|---|---| +| prep (trust cwd, write prompt file) | 2 ms | +| `tmux new-session` | 27 ms | +| **boot → input bar ready** | **1232 ms** | +| paste (`load-buffer` + `paste-buffer`) | 8 ms | +| paste-verify poll | 426 ms | +| **submit → transcript terminal** | **8458 ms** | +| teardown | 8 ms | +| **total** | **10162 ms** | +| *claude's own reported `turn_duration`* | *5539 ms* | +| **OCP-side overhead** | **4490 ms** | + +The `submit → terminal` phase exceeds claude's own `turn_duration` by **~2.9 s**. That gap is +**post-input-bar initialization inside `claude`** — work that a pane which has merely *sat idle +for a few seconds* has already completed. A direct spike confirmed it: an identical pane, idle +12 s before receiving the same prompt, completed its turn in a median 5537 ms versus 7980 ms +cold. + +So a warm pane recovers **~1.26 s of boot *and* ~2.9 s of in-`claude` cold start** — not the +~1.0 s the plan predicted. + +The reason this was worth a pool rather than a "keep one session and reuse it" cache is a +hazard already flagged in the code. `lib/tui/transcript.mjs` returns the **last text-bearing +assistant entry in the whole transcript file**, which is correct *only* under OCP's +one-session-per-request model, and it says so: + +> *"If a future warm-pool ever reuses a session WITHOUT a fresh session-id / clear, earlier-turn +> text could leak — that author must add user-line scoping here."* + +Reusing a pane for a second turn puts two exchanges in one transcript and would leak the earlier +turn's text into the later turn's answer — a **cross-request data leak**, not merely a bug. + +--- + +## Decision + +Add an **opt-in pool of pre-booted, single-use `claude` panes**, `OCP_TUI_POOL_SIZE` (default +`0` = off, max `4`). Implementation: `lib/tui/pool.mjs`. + +### 1. Panes are SINGLE-USE. This is the load-bearing rule. + +A pooled pane serves **exactly one turn**, then is killed and replaced in the background. Each +pane is booted with its **own fresh `--session-id`**, fixed at spawn, and the turn locates its +transcript by that id. + +This preserves one-session-per-request exactly, so the `transcript.mjs` hazard above **does not +arise** and no user-line scoping was needed. The warning in `transcript.mjs` is deliberately +left standing, now annotated: it still binds anyone who later wants a pane to serve a second +turn, or to reset a session with `/clear` and reuse it. **Neither is permitted without first +adding user-line scoping to the transcript reader.** + +Rejected alternative — *reuse a pane for N turns, `/clear` between* — is strictly cheaper +(no re-boot per request) and was rejected on exactly this basis. The latency win is not worth a +cross-request text-leak surface guarded only by a `/clear` that we cannot verify landed. + +### 2. The pool is keyed by model, and a MISS is always safe. + +`--model` is fixed at spawn, so a pane can only serve the model it booted with. A pool miss +falls back to the existing cold-boot path with **zero behavioural difference**. There is no +boot-time pre-warm and no configured model: OCP cannot know which model the next caller wants, +so the pool warms the **most recently requested** model. Consequence, stated plainly: **the +first request after start, and the first after any model switch, is always a cold miss.** + +### 3. The pool and the session reaper coexist by an explicit invariant. + +This is the subtle part. `reapStaleTuiSessions()` kills every session matching this instance's +`ocp-tui--` prefix, and issues `tmux kill-server` when no foreign session remains (the +only mechanism that can reap `` `claude` zombies — the pane's `claude` is a child of +the tmux *server*, not of node). A warm pooled pane **is** one of our own sessions, alive and +idle **by design** — and the periodic sweep runs precisely **when the instance is idle**, i.e. +exactly when the pool is full. + +The invariant, stated in a comment above `reapStaleTuiSessions` and pinned by tests: + +1. **A live pooled pane is never reaped — including one that is still BOOTING.** The reaper + takes a `spare` set of **exact session names** supplied by the pool's live registry. +2. **An orphaned pooled pane IS still reaped.** Membership is by **exact name from a live + in-memory registry, never by name shape**. A pane the pool no longer owns — handed out, + dropped, cancelled, or left behind by a previous process generation (whose registry died with + it) — is absent from `spare` and is killed like any other stale session. **Fail-safe: + omitting `spare` reaps *more*, never less.** Pool panes are named `ocp-tui--p` + purely for operator legibility; that shape is *not* the exemption mechanism. +3. **`kill-server` is suppressed while any pane is spared** (it would kill a live child of the + tmux server). Therefore **the pool is DRAINED immediately before every sweep**, so `spare` is + empty on the normal tick and `kill-server` still fires. Without the drain, a permanently-full + pool would **permanently disable zombie reaping** — the pool would silently break the thing + the sweep exists to do. The drain costs one pane re-boot per tick (15 min). + +The `spare` mechanism is belt-and-braces given the drain: it makes it impossible for a reap call +site that *forgets* to drain to kill a live pane. + +### 4. The pool tracks its in-flight boot BY NAME, not as a count. + +`bootTuiPane` creates the tmux session **synchronously** and only *then* waits (up to +`POOL_BOOT_MS`, 20 s) for the input bar. So **a pooled tmux session can be live for ~20 s before +its boot resolves.** A pool that tracked in-flight boots as a *count* could not name that +session, and this produced two real bugs (both caught in review, both now regression-tested): + +- the periodic sweep **killed the booting pane** (it could not be spared), then left the pool + empty with nothing scheduled, and logged the exact `tui_pool_boot_failed` warning operators are + told to alert on — for a completely healthy drain; +- graceful shutdown **orphaned a live, authenticated, idle `claude`**: `gracefulShutdown` calls + `process.exit(0)` in the same tick as the drain (TUI panes are tmux children, so node's + `activeProcesses` set is empty and the "wait for children" path exits immediately), so any + cleanup deferred to a `.then()` never ran. + +The pool therefore **mints each pane's identity up front** (`{sessionId, name}`, name derived +from the session-id so `tmux ls` correlates to the transcript file) and holds it in +`_bootingPane`. `liveNames()` includes it; `drain()` kills it **synchronously**. A generation +counter distinguishes *"cancelled by us"* from *"genuinely failed"*, so a drain never inflates +`bootFailures` and `resume()` reliably starts a fresh boot. + +### 5. Refills take no concurrency slot, and are serialized. + +A refill boot deliberately does **not** take a `TuiSemaphore` slot: those slots bound concurrent +*turns* and belong to real requests, and charging a background pre-boot against them would let +the pool starve the traffic it exists to speed up. It cannot leak a slot either, since it never +holds one. Boots are **serialized** (one at a time): two cold boots racing an in-flight turn were +observed to overrun even the generous pool readiness cap. A genuinely failed boot does **not** +re-kick the chain (backoff — a broken `claude` must not respawn forever). + +Background boots get a more generous readiness cap (`POOL_BOOT_MS` = 5 × `BOOT_MS`): `BOOT_MS` is +tight because a *client* is blocked on it, which is not true of a pre-boot. Slow ≠ broken. + +--- + +## Consequences + +### Cost — standing processes, paid whether or not a request arrives + +**A warm pane is a live idle `claude` process.** Peak process count is +`OCP_TUI_POOL_SIZE` + `OCP_TUI_MAX_CONCURRENT` + 1 (booting replacement). This is the whole +reason the pool is **default-off**: an operator must opt into holding processes for traffic that +may never come. Size is clamped to `POOL_MAX_SIZE` = 4; an unparseable value **disables** the +pool rather than guessing. + +Panes carry a 10-minute TTL and are health-checked at hand-out; a dead or degraded pane becomes +a **miss** (cold path), never a hung turn. + +### Benefit + +Measured end-to-end through a real OCP instance (Sonnet 4.6, `--effort low`): +**p50 10.17 s (n=6, pool off) → 6.00 s (n=12 warm hits) — −4.2 s / −41%.** + +### The floor is unchanged + +The pool does not touch the **~6 s TTFT floor** documented in the latency plan (claude always +prefills the full Claude Code system prompt). TUI mode remains unsuitable for interactive / +real-time consumers; it is for batch and background work. This ADR does not change that +conclusion. + +### Observability + +`/health`'s `tui` block gains a `pool` sub-object (`null` when off): `size`, `warm`, `booting`, +`model`, `hits`, `misses`, `boots`, `bootFailures`, `cancelled`, `dropped`. A climbing +`bootFailures` means panes are not reaching their input bar — the pool then degrades safely to +the cold path, but latency reverts to the un-pooled numbers. A steadily climbing `dropped` is +**normal** (the 15-min sweep drains and re-boots the pool on every tick, by design — see +Decision 3). + +### ALIGNMENT authorization + +- **Class B / OCP-owned.** The warm pool is process management around the `claude` CLI — the + same category as the existing tmux session lifecycle and the defunct-session reaper it extends. + **`cli.js` does not perform this operation, and no `cli.js` citation applies**; the authority + is ADR 0007 (which owns the TUI spawn machinery) plus this ADR. This is `ALIGNMENT.md` Rule 2's + Class B citation requirement, discharged explicitly rather than by silence. +- **The `/health` extension** adds sub-fields to the `tui` block. That block is **owned by ADR + 0007** and post-dates ADR 0006's v3.16.4 grandfather snapshot, so it is not part of the frozen + B.2 inventory. The change is additive — every pre-existing `/health` field keeps a + byte-identical value, and `pool` is `null` unless the operator opts in — which is the + behaviour-preserving bar ADR 0006 sets. This ADR records that authorization. +- **No spawn argument changed.** `buildTuiCmd` is byte-identical; the pool calls it with the same + arguments. Banner-verified on live pooled panes: `· Claude Max`, never `API Usage Billing` + (the `--bare` trap documented in the latency plan). + +### What a future contributor must not undo + +- **Do not let a pane serve a second turn** (or `/clear`-and-reuse one) without first adding + user-line scoping to `lib/tui/transcript.mjs`. That is a cross-request text leak, not a perf + tweak. See Decision 1. +- **Do not remove the drain-before-sweep.** It is what keeps `kill-server` zombie reaping alive. + See Decision 3. +- **Do not go back to counting in-flight boots.** The pool must be able to *name* a session that + exists but has not finished booting. See Decision 4. diff --git a/docs/adr/README.md b/docs/adr/README.md index 1423ef5..c68ba8d 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -23,6 +23,8 @@ New ADRs increment from the highest existing number. Filenames are | [0004](0004-openclaw-auto-sync.md) | OpenClaw Auto-Sync | Why `scripts/sync-openclaw.mjs` runs on `ocp update`, what its scope boundary is (writes only `models.providers["claude-local"].models` and `agents.defaults.models["claude-local/*"]`), and the idempotency contract. | | [0005](0005-no-multi-provider.md) | No Multi-Provider | Why OCP stays single-provider (Anthropic-via-cli.js) and does not extend to OpenAI / Gemini / OpenRouter. Cost estimate: ~7 weeks for a v1 that buys neither moat nor commercial readiness. Separate commercial work starts in a separate repo. | | [0006](0006-openai-shim-scope.md) | OpenAI Shim Scope | The Class A / Class B taxonomy. Class A endpoints (`cli.js`-mirror) keep Rules 1–5 verbatim; Class B endpoints (OCP-owned compatibility surface — `/v1/chat/completions`, `/v1/models`, admin endpoints) are anchored to OpenAI's spec (B.1) or to an authorizing ADR (B.2). Triggered by PR #99 (external `response_format` honoring). Grandfathers the existing B.2 inventory at v3.16.4. | +| [0007](0007-tui-interactive-mode.md) | TUI Interactive Mode | Why TUI-mode spawns an interactive `claude` in a tmux pane (no `-p`) to reach the **subscription** billing pool (`cc_entrypoint=cli`) rather than the metered Agent SDK pool. Owns the TUI spawn machinery: entrypoint labeling, credential-isolated home, MCP hard-disable, session namespace + defunct-session reaping, the independent concurrency bound, and the `/health` `tui` block. **Single-user only** — hard FATAL on multi-user configs. | +| [0008](0008-tui-warm-pane-pool.md) | TUI Warm Pane Pool | Why `OCP_TUI_POOL_SIZE` pre-boots **single-use** `claude` panes (one turn each, own `--session-id`) — and why reuse is forbidden (`transcript.mjs` returns the last assistant entry in the file, so a reused session leaks the earlier turn's text). Measured −41% end-to-end. Defines the pool↔reaper invariant (exemption by exact name from a live registry; drain before every sweep so `kill-server` zombie reaping survives) and the standing idle-process cost. Extends ADR 0007. | ## When to write a new ADR diff --git a/lib/tui/pool.mjs b/lib/tui/pool.mjs new file mode 100644 index 0000000..f05cdbc --- /dev/null +++ b/lib/tui/pool.mjs @@ -0,0 +1,295 @@ +// TUI warm pane pool (docs/plans/2026-07-13-tui-latency backlog #3). +// +// WHAT IT IS: a small set of PRE-BOOTED `claude` panes, each already sitting at its +// input bar, so a request does not pay the cold boot. Opt-in: OCP_TUI_POOL_SIZE=0 +// (default) disables it entirely and the request path is byte-for-byte today's. +// +// ── SINGLE-USE IS THE LOAD-BEARING RULE ───────────────────────────────────── +// A pooled pane serves EXACTLY ONE turn and is then killed and replaced in the +// background. Each pane carries its OWN fresh `--session-id`, fixed at boot, and the +// turn locates its transcript by that id. So OCP's one-session-per-request model is +// preserved: a session's transcript still holds exactly one logical exchange. +// That is what keeps lib/tui/transcript.mjs's extractLatestAssistantText (which returns +// the LAST text-bearing assistant entry in the whole file, not "text since the matching +// user line") correct — see the scoping note there. A pane MUST NEVER serve a second +// turn, and a session MUST NEVER be reset with /clear and reused: either would put two +// exchanges in one transcript and leak the earlier turn's text into the later turn's +// answer. Nothing here reuses a pane; keep it that way. +// +// ── WHY IT'S WORTH MORE THAN THE BOOT TIME ────────────────────────────────── +// Measured on this host (n=6 through OCP, Sonnet 4.6, --effort low): the cold path +// spends ~1.23 s reaching the input bar, but ALSO ~2.9 s inside the first turn beyond +// what claude itself reports as the turn duration — post-input-bar init that a pane +// which has been idle for a few seconds has already finished. A warm pane recovers both. +// +// ── COST (bounded, and paid whether or not a request arrives) ─────────────── +// Each warm pane is a LIVE `claude` process (plus its tmux pane) sitting idle. Peak +// process count is (pool size) + (OCP_TUI_MAX_CONCURRENT in-flight turns) + (panes +// currently booting as replacements). Pool size is clamped to POOL_MAX_SIZE. +// +// Pure + injectable (bootPane / killPane / paneHealthy / now) so test-features.mjs can +// assert acquire / miss / refill / TTL / reaper-exemption with no tmux and no claude. + +// Hard cap on OCP_TUI_POOL_SIZE. Each pane is an idle claude process; 4 is already a +// lot of resident memory on a small host (a Pi serving a family) for zero in-flight work. +export const POOL_MAX_SIZE = 4; + +// A warm pane older than this is dropped on acquire rather than handed out. The periodic +// reap tick (server.mjs) drains the pool every 15 min anyway, so this only bites when +// that tick kept getting skipped because the TUI path was never idle. Guards against +// handing out a pane whose `claude` has been sitting so long it may have drifted +// (auto-compaction prompts, an idle-disconnect banner, an expired in-pane token). +export const POOL_MAX_AGE_MS = 10 * 60 * 1000; + +// Clamp the operator-supplied size into [0, POOL_MAX_SIZE]. A garbage value disables the +// pool rather than guessing — an unparseable size must never silently boot 4 processes. +export function resolvePoolSize(raw) { + const n = parseInt(raw, 10); + if (!Number.isFinite(n) || n <= 0) return 0; + return Math.min(n, POOL_MAX_SIZE); +} + +export class TuiPanePool { + // size: target number of warm panes (0 = disabled). + // maxAgeMs: per-pane TTL (see POOL_MAX_AGE_MS). + // mintPane: () => ({ sessionId, name }) — mints the identity of the NEXT pane. The POOL, + // not the boot function, owns this: the tmux session springs into existence the + // instant bootPane starts, so the pool must already know its NAME (see + // _bootingPane below). Deriving the name from the sessionId also makes `tmux ls` + // correlate to the transcript file. + // bootPane: async (model, {sessionId, name}) => { name, sessionId, model, bootedAt } — + // boots ONE pane under exactly that identity and resolves only once it is + // input-ready; throws if it never becomes ready. + // killPane: (name) => void — tmux kill-session. MUST be synchronous (see drain). + // paneHealthy:(name) => bool — pane still exists AND is still at its input bar. + constructor({ size, maxAgeMs = POOL_MAX_AGE_MS, mintPane, bootPane, killPane, paneHealthy, now = Date.now, log = () => {} }) { + this.size = Math.max(0, Math.min(parseInt(size, 10) || 0, POOL_MAX_SIZE)); + // Fail fast at CONSTRUCTION, not at request time. refill() is called synchronously from + // the request path (runTuiTurn), so a missing collaborator would otherwise surface as a + // 500 on a live request instead of a loud error at boot. + if (this.size > 0) { + for (const [k, fn] of [["mintPane", mintPane], ["bootPane", bootPane], ["killPane", killPane], ["paneHealthy", paneHealthy]]) { + if (typeof fn !== "function") throw new TypeError(`TuiPanePool: ${k} must be a function`); + } + } + this.maxAgeMs = maxAgeMs; + this._mintPane = mintPane; + this._bootPane = bootPane; + this._killPane = killPane; + this._paneHealthy = paneHealthy; + this._now = now; + this._log = log; + + this._panes = []; // warm, available panes: { name, sessionId, model, bootedAt } + // The pane currently BOOTING, BY NAME ({sessionId, name, model}) — or null. + // + // WHY A NAME AND NOT A COUNT (this is a fixed bug, don't regress it): bootTuiPane creates + // the tmux session SYNCHRONOUSLY and only THEN waits up to POOL_BOOT_MS (20 s) for the + // input bar. So for up to 20 s there is a LIVE pooled tmux session. When the pool tracked + // only a count, it could not NAME that session, so: + // - liveNames() could not spare it and the periodic reap sweep KILLED it (and + // kill-server'd on top), leaving the pool empty with nothing scheduled and firing the + // very tui_pool_boot_failed WARN operators are told to alert on; and + // - drain() could not kill it, so on shutdown it ORPHANED a live authenticated `claude` + // (the boot's .then that was supposed to clean up never runs — gracefulShutdown calls + // process.exit in the same tick). + // Both are fixed by holding the identity here, before the session exists. + this._bootingPane = null; + // Generation counter. Bumped whenever an in-flight boot is CANCELLED (drain / model + // switch). A boot compares the generation it started under against the current one: + // if they differ, its pane was already killed by us and its settle is inert — in + // particular a rejection is a CANCELLATION, not an operator-visible boot failure. + this._gen = 0; + this._paused = false; // true while drained; refill() is a no-op until resume() + this.warmModel = null; // the model the pool currently warms — learned from traffic (see acquire) + + this.hits = 0; // requests served by a warm pane + this.misses = 0; // requests that fell back to the cold path + this.boots = 0; // panes successfully pre-booted + this.bootFailures = 0; // pre-boots that genuinely never reached the input bar + this.cancelled = 0; // in-flight boots WE killed (drain / model switch) — not failures + this.dropped = 0; // panes discarded unused (unhealthy / expired / wrong model / drained) + } + + get enabled() { return this.size > 0; } + get warm() { return this._panes.length; } + get booting() { return this._bootingPane ? 1 : 0; } + + // The reaper's spare set: the EXACT names of every pane the pool currently owns and has NOT + // handed out — the warm ones AND the one currently booting (whose tmux session is already + // live; see _bootingPane). See the POOL/REAPER INVARIANT in lib/tui/session.mjs. + // Fail-safe by construction: a pane leaves this set the instant it is acquired, dropped, or + // cancelled, and if the pool is empty (or the process restarted) the set is empty — so an + // orphaned pooled pane looks exactly like any other stale session and IS reaped. + liveNames() { + const names = new Set(this._panes.map((p) => p.name)); + if (this._bootingPane) names.add(this._bootingPane.name); + return names; + } + + // Take a warm pane for `model`, or null (caller must fall back to the cold path — a MISS + // is always safe, never an error). Synchronous: paneHealthy is a cheap tmux capture. + // + // The pool warms the MOST RECENTLY REQUESTED model (`warmModel`). There is no boot-time + // pre-warm and no configured model: OCP cannot know which model the next caller wants, and + // pre-booting a process for a model nobody asks for is pure waste. Consequence, stated + // plainly: the FIRST request after start (and the first after a model switch) is always a + // MISS. The pool pays off for the steady repeat traffic it exists to serve. + acquire(model) { + if (!this.enabled) return null; + + // Retarget on a model switch: --model is fixed at spawn, so panes for another model are + // useless. Drop them now (they are replaced by the next refill) rather than holding + // processes for a model that is no longer being asked for. This includes any pane + // currently BOOTING for the old model — its tmux session already exists, so leaving it to + // die on resolve would both hold a useless process and block the next refill (one boot at + // a time) for up to POOL_BOOT_MS. + if (model !== this.warmModel) { + for (const p of this._panes) { this._drop(p, "model_switch"); } + this._panes = []; + this._cancelBooting("model_switch"); + this.warmModel = model; + } + + while (this._panes.length) { + const p = this._panes.shift(); + if (this._now() - p.bootedAt > this.maxAgeMs) { this._drop(p, "expired"); continue; } + if (!this._paneHealthy(p.name)) { this._drop(p, "unhealthy"); continue; } + this.hits++; + return p; // caller OWNS it now: it is out of the registry (so out of the spare set), + // and the caller's finally MUST kill it. Single-use — never returned here. + } + this.misses++; + return null; + } + + // Bring the pool back up to `size` warm panes for `warmModel`. Fire-and-forget: never + // awaited on the request path and never throws into it. + // + // SLOT ACCOUNTING: a refill boot deliberately does NOT take a TuiSemaphore slot. Those + // slots bound concurrent *turns* (each up to the 120 s wallclock) and belong to real + // requests; charging a background pre-boot against them would let the pool starve the + // traffic it exists to speed up. It cannot leak a slot either, because it never holds one. + // + // SERIALIZED, ONE BOOT AT A TIME (and re-kicked on success until the pool is at target). + // An earlier version launched all `want` boots at once; live at size=2 that put two cold + // `claude` boots plus an in-flight turn on the CPU together, and a refill overran even the + // generous pool readiness cap (tui_pool_boot_failed). Booting sequentially keeps each boot + // near its uncontended ~1.2 s, bounds the CPU burst the pool can cause, and still has the + // replacement pane warm long before the next request arrives. + // + // A genuinely FAILED boot deliberately does NOT re-kick the chain — that is the backoff. A + // persistently failing boot (bad claude binary, no auth) would otherwise spin, respawning + // forever. The next natural trigger (the following request's refill, or the reap tick's + // resume) retries it. A CANCELLED boot is different: we killed it on purpose, nothing is + // wrong, and resume() is expected to start a fresh one immediately. + refill() { + if (!this.enabled || this._paused || !this.warmModel) return; + if (this._bootingPane) return; // one boot in flight at a time + if (this._panes.length >= this.size) return; // already at target + + const model = this.warmModel; + const gen = this._gen; + // Mint the identity BEFORE booting: bootPane creates the tmux session synchronously, so + // the pool must be able to name (and therefore spare, and kill) it from this moment on. + const ident = this._mintPane(); + this._bootingPane = { ...ident, model }; + let enlisted = false; + Promise.resolve() + .then(() => this._bootPane(model, ident)) + .then((pane) => { + // The world may have moved while we booted. If our generation was cancelled, our pane + // was ALREADY killed by _cancelBooting — do not touch it, do not enlist it. + if (gen !== this._gen) return; + // Otherwise: still possible the pool filled or retargeted without a cancellation. + if (this._paused || model !== this.warmModel || this._panes.length >= this.size) { + this._drop(pane, "stale_boot"); + return; + } + this._panes.push(pane); + this.boots++; + enlisted = true; + }) + .catch((e) => { + // A rejection from a CANCELLED generation is not a fault: it is almost always + // "tui_pane_not_ready", thrown because WE killed the pane out from under the boot. + // Counting it as a bootFailure would fire the exact WARN operators are told to alert + // on, for a completely healthy drain. Stay silent — _cancelBooting already counted + // this as a cancellation, so do NOT count it again here. + if (gen !== this._gen) return; + this.bootFailures++; + this._log("warn", "tui_pool_boot_failed", { model, error: e && e.message }); + }) + .finally(() => { + // ONLY the current generation's boot owns the booting slot. A stale settle must not + // clear a slot that a newer boot (started by resume()) already holds. + if (gen === this._gen) this._bootingPane = null; + if (enlisted) this.refill(); // continue toward target, still one at a time + }); + } + + // Kill the in-flight boot's pane, SYNCHRONOUSLY, and invalidate its generation. Returns 1 + // if there was one, else 0. The tmux session already exists (bootPane created it before it + // started waiting for readiness), so this is a real kill, not a cancellation flag. + _cancelBooting(reason) { + if (!this._bootingPane) return 0; + this._gen++; // the in-flight boot's settle is now inert + this._drop(this._bootingPane, reason); // synchronous kill-session + this._bootingPane = null; + this.cancelled++; + return 1; + } + + // Kill every pane the pool owns — warm AND currently booting — and stop refilling. Returns + // how many were killed. + // + // Called (a) before the periodic reap sweep — reapStaleTuiSessions can only reap defunct + // `claude` zombies via kill-server, and kill-server is suppressed while any live pooled pane + // exists (including a booting one), so without this drain the pool would permanently disable + // zombie reaping; and (b) on graceful shutdown, so no pane outlives the process as an orphan. + // + // EVERY KILL HERE IS SYNCHRONOUS, and that is load-bearing. It is NOT safe to leave the + // booting pane to clean itself up on resolve: gracefulShutdown calls process.exit() in the + // same tick as this drain (TUI panes are children of the tmux SERVER, not of node, so + // node's activeProcesses set is empty on a TUI host and the "wait for children" path exits + // immediately). A .then()/.catch() scheduled here would never run, and the pane would + // survive as an orphaned, authenticated, idle `claude`. + drain() { + this._paused = true; + let n = this._panes.length; + for (const p of this._panes) this._drop(p, "drain"); + this._panes = []; + n += this._cancelBooting("drain_booting"); + return n; + } + + // Undo drain() and start refilling again. Because drain() CANCELLED the in-flight boot + // (rather than leaving it pending), the booting slot is free and this really does start a + // fresh boot — the pool is never left empty with nothing scheduled. + resume() { + this._paused = false; + this.refill(); + } + + // /health surface (additive). + stats() { + return { + size: this.size, + warm: this._panes.length, + booting: this.booting, + model: this.warmModel, + hits: this.hits, + misses: this.misses, + boots: this.boots, + bootFailures: this.bootFailures, + cancelled: this.cancelled, + dropped: this.dropped, + }; + } + + _drop(pane, reason) { + this.dropped++; + try { this._killPane(pane.name); } catch { /* already gone */ } + this._log("info", "tui_pool_pane_dropped", { name: pane.name, reason }); + } +} diff --git a/lib/tui/semaphore.mjs b/lib/tui/semaphore.mjs index c84b3cc..1e2b614 100644 --- a/lib/tui/semaphore.mjs +++ b/lib/tui/semaphore.mjs @@ -139,7 +139,12 @@ export function recordTuiEntrypoint(tuiStats, observed, expectedMode = "cli") { // Build the additive /health `tui` block (ADR 0007 PR-B amendment). Pure: given the // config + live counters, returns the exact object embedded in /health. New fields only — // behaviour-preserving for existing /health consumers (grandfathered B.2 under ADR 0006). -export function buildTuiHealthBlock({ enabled, entrypointMode, maxConcurrent }, tuiStats, semaphore) { +// +// `pool` (optional, warm pane pool — lib/tui/pool.mjs): a TuiPanePool, or null/undefined +// when the pool is off (the default). Reported as `pool: null` when off so the block's +// shape stays stable, and as the pool's stats (size / warm / hits / misses / …) when on — +// the operator's window onto both the hit rate and the standing idle-process cost. +export function buildTuiHealthBlock({ enabled, entrypointMode, maxConcurrent }, tuiStats, semaphore, pool = null) { return { enabled, entrypointMode, // cli | auto | off @@ -148,5 +153,6 @@ export function buildTuiHealthBlock({ enabled, entrypointMode, maxConcurrent }, inflight: semaphore.inflight, // current concurrent TUI turns queued: semaphore.queued, // turns waiting for a slot maxConcurrent, + pool: pool ? pool.stats() : null, // warm pane pool, or null when disabled }; } diff --git a/lib/tui/session.mjs b/lib/tui/session.mjs index 1f6ade6..97c874e 100644 --- a/lib/tui/session.mjs +++ b/lib/tui/session.mjs @@ -73,6 +73,36 @@ const defaultTmux = (args, opts = {}) => // `port` (required) is this instance's own listen port (server.mjs's PORT / lib/constants.mjs // DEFAULT_PORT resolution) — the SPOT for "which sessions are ours." // +// ── POOL/REAPER INVARIANT (warm pane pool — lib/tui/pool.mjs) ─────────────────────────── +// A warm pooled pane is one of OUR OWN `ocp-tui--*` sessions that is ALIVE AND IDLE +// BY DESIGN — and the periodic sweep runs precisely when the instance is idle, i.e. exactly +// when the pool is full. Without an exemption the sweep would kill every warm pane on every +// tick (and kill-server on top). The exemption is `spare`: a set of EXACT session names the +// caller declares live. Three properties, all load-bearing: +// +// 1. A LIVE POOLED PANE IS NEVER REAPED — INCLUDING ONE THAT IS STILL BOOTING. It is in +// `spare` (the pool's live registry), so it is skipped by name. The booting case is not +// a footnote, it is the one that bit us: bootTuiPane creates the tmux session +// SYNCHRONOUSLY and only then waits up to POOL_BOOT_MS for the input bar, so a pooled +// session can be live for ~20 s before its boot resolves. The pool therefore mints the +// pane's NAME up front and holds it in `_bootingPane`, so liveNames() can name — and +// spare — a session whose boot has not finished. (An earlier version tracked only a +// COUNT of in-flight boots; the sweep could not name that session and killed it.) +// 2. A LEAKED/ORPHANED POOLED PANE IS STILL REAPED. Membership is by EXACT NAME from a +// live in-memory registry — NOT by "looks pooled" (name shape). A pane the pool no +// longer owns (handed out, dropped, cancelled, or left behind by a previous process +// generation — whose registry died with it) is absent from `spare` and is killed like +// any other stale session. Fail-safe: forgetting to pass `spare` reaps MORE, never less. +// 3. KILL-SERVER NEVER KILLS A LIVE POOL PANE. A spared session suppresses kill-server +// exactly as a foreign session does (it is a live child of the tmux server). The +// consequence — that a permanently-full pool would permanently disable the defunct- +// zombie reaping that ONLY kill-server can do — is resolved in server.mjs by DRAINING +// the pool immediately before the sweep, so `spare` is empty on the normal tick and +// kill-server still fires. `spare` is the belt-and-braces: a reap call site that +// forgets to drain still cannot kill a live pane. +// +// `spare` (default: none) — iterable of session names, or a Set. Ignored when the pool is off. +// // `includeLegacy` (default false): when true, sessions matching the exact OLD bare-prefix // shape (LEGACY_SESSION_NAME_RE) are ALSO treated as ours for kill-session purposes. This is // the boot-time legacy migration: an operator upgrading past this fix could otherwise be left @@ -88,14 +118,20 @@ const defaultTmux = (args, opts = {}) => // same class of residual risk the audit finding itself accepts ("no live instance of the new // version creates them"); this PR does not regress that scenario, it only removes the far // more common same-version collision (the actual F7 finding). -export function reapStaleTuiSessions({ tmux = defaultTmux, port, includeLegacy = false } = {}) { +export function reapStaleTuiSessions({ tmux = defaultTmux, port, includeLegacy = false, spare = null } = {}) { const r = tmux(["list-sessions", "-F", "#{session_name}"]); if (!r || r.status !== 0) return 0; // no tmux server / no sessions const names = String(r.stdout || "").split("\n").map((s) => s.trim()).filter(Boolean); const ownPrefix = sessionPrefixForPort(port); + const spared = spare instanceof Set ? spare : new Set(spare || []); let killed = 0; let othersRemain = false; + let sparedLive = 0; for (const name of names) { + // Property 1+2: exemption is by EXACT NAME from the pool's live registry. A pooled- + // LOOKING name that is not in the registry is an orphan and falls through to the + // normal kill path below. + if (spared.has(name)) { sparedLive++; continue; } const isOwn = name.startsWith(ownPrefix); const isLegacyOwn = includeLegacy && LEGACY_SESSION_NAME_RE.test(name); if (isOwn || isLegacyOwn) { @@ -109,7 +145,11 @@ export function reapStaleTuiSessions({ tmux = defaultTmux, port, includeLegacy = // Reap defunct `claude` zombies: safe ONLY when the server is now ours-only/empty. // kill-server is what actually reaps (server exit reparents survivors to init); a // per-session kill cannot, since node is not the zombies' parent. - if (!othersRemain) { + // + // Property 3: a SPARED session is a live child of this tmux server, so kill-server would + // kill it — it therefore suppresses kill-server exactly as a foreign session does. On the + // normal sweep the pool is drained first, so sparedLive is 0 and kill-server still fires. + if (!othersRemain && sparedLive === 0) { tmux(["kill-server"]); } return killed; @@ -119,6 +159,12 @@ export function reapStaleTuiSessions({ tmux = defaultTmux, port, includeLegacy = // Boot + paste-settle timing. Conservative defaults validated on PI231; env-tunable. const BOOT_MS = parseInt(process.env.OCP_TUI_BOOT_MS || "4000", 10); // max wait for input-ready +// Readiness cap for a POOL pre-boot. Deliberately far more generous than BOOT_MS: BOOT_MS is +// tight because a client is blocked on it, whereas a warm-pane boot happens in the background +// with nobody waiting. Observed live at size=2: a refill booting alongside an in-flight turn +// exceeded 4000 ms and was discarded (tui_pool_boot_failed), quietly costing hit rate for a +// pane that was merely slow, not broken. Scales with OCP_TUI_BOOT_MS if an operator raises it. +export const POOL_BOOT_MS = BOOT_MS * 5; const READY_POLL_MS = parseInt(process.env.OCP_TUI_READY_POLL_MS || "400", 10); // readiness / paste-verify poll interval const PASTE_VERIFY_MS = parseInt(process.env.OCP_TUI_PASTE_VERIFY_MS || "5000", 10); // max wait for pasted prompt to render @@ -412,38 +458,60 @@ export function buildTuiCmd(claudeBin, model, sessionId, ehome, entrypointMode) ].join(" "); } -// Full per-request TUI lifecycle: -// 1. Pre-trust the scratch cwd (no trust dialog will appear). -// 2. Write prompt to a 0600 temp file (no shell injection from prompt content). -// 3. Boot an interactive `claude` in a fresh tmux session in the scratch cwd; poll -// capture-pane until the `? for shortcuts` input bar appears (readiness-poll -// replaces the old blind boot sleep). BOOT_MS is the max wait, not a fixed delay. -// 4. Paste the prompt via tmux load-buffer + paste-buffer -p (bracketed paste) — -// reliable for large multi-line prompts where send-keys -l is not (issue #130). -// Poll-verify the prompt landed in the input (placeholder gone / [Pasted text]); -// fast-fail with tui_paste_not_landed if it never lands (prevents the 120s -// wallclock "stuck typing" hang). Then submit with a SEPARATE Enter key event. -// 5. Block on the native JSONL transcript (located by session-id) until terminal -// marker or wall-clock cap. -// 6. Always teardown: kill session + rm temp dir (even on throw). -// Returns { text, entrypoint } from readTuiTranscript (entrypoint is the billing-pool -// classifier, e.g. "cli", or null if the transcript did not include a turn_duration). -export async function runTuiTurn({ - prompt, - model, - claudeBin, - home, - realHome, - cwd, - port, - wallclockMs = 120000, - entrypointMode = "cli", - tmux = defaultTmux, +// Is a pane alive AND still sitting at its input bar? Used by the warm pool to decide, +// at hand-out time, whether a pre-booted pane is still usable (a dead/degraded pane must +// become a MISS → cold path, never a hung turn). capture-pane exits non-zero when the +// session no longer exists, so this covers "pane gone" and "pane not ready" in one call. +export function tuiPaneHealthy(tmux, tmuxName) { + const r = tmux(["capture-pane", "-p", "-t", tmuxName]); + if (!r || r.status !== 0 || typeof r.stdout !== "string") return false; + return tuiInputReady(r.stdout); +} + +// Pool pane names carry a "p" marker after the port-scoped prefix: +// turn pane: ocp-tui--<8hex> (unchanged) +// pool pane: ocp-tui--p<8hex> +// Purely for operator legibility (`tmux ls` shows which panes are warm). It is NOT the +// reaper's exemption mechanism — that is the exact-name spare set (see the POOL/REAPER +// INVARIANT above), so a pooled-LOOKING orphan is still reaped. Both shapes start with +// sessionPrefixForPort(port), so both remain reapable as "ours", and neither can match +// LEGACY_SESSION_NAME_RE. +export function poolPaneName(port, sessionId) { + return sessionPrefixForPort(port) + "p" + sessionId.slice(0, 8); +} + +// Boot ONE interactive `claude` pane and wait for its input bar. Shared by the cold +// request path (runTuiTurn) and the warm pool (lib/tui/pool.mjs) so a pooled pane is +// spawned with byte-for-byte the same argv, HOME, cwd and trust preparation as a +// cold-booted one — the pool must not become a second, drifting spawn path. +// +// Each pane gets its OWN fresh randomUUID() --session-id, fixed at boot. That is what +// keeps a pooled pane single-use-safe: its transcript holds exactly one exchange. +// +// requireReady: the cold path tolerates a readiness timeout (it falls through and lets +// the paste-verify decide — pre-existing behaviour, unchanged). The POOL sets it, because +// a pane that never reached its input bar is worthless as a warm pane and must not be +// enlisted: throw, let the pool count a bootFailure, and leave the request path to +// cold-boot as usual. +// bootMs: max wait for the input bar. Defaults to BOOT_MS (the REQUEST path's cap, which is +// deliberately tight — a client is blocked on it). The POOL passes POOL_BOOT_MS instead: a +// background pre-boot has nobody waiting on it, and capping it at the request-path's 4 s +// made real refills fail (observed live: a refill booting alongside an in-flight turn took +// >4 s and was discarded, silently lowering the hit rate). Slow != broken for a pre-boot. +// `sessionId` / `name` (both optional): the caller may supply the pane's identity instead of +// letting bootTuiPane mint it. The POOL does, because it must know the tmux session's NAME +// before this function runs — the session is created synchronously below, well before the +// readiness wait returns, so a pool that only learned the name on resolve could neither spare +// the session from the reaper nor kill it on shutdown. Supplying BOTH also keeps the name's +// hex suffix equal to the session-id's, so `tmux ls` correlates to the transcript file. +export async function bootTuiPane({ + model, claudeBin, home, realHome, cwd, port, entrypointMode = "cli", + tmux = defaultTmux, sessionId = null, name = null, requireReady = false, bootMs = BOOT_MS, }) { - const sessionId = randomUUID(); + const sid = sessionId || randomUUID(); // Port-scoped session name (F7 fix) — see sessionPrefixForPort / reapStaleTuiSessions // for why this instance's own listen port is the namespace discriminator. - const tmuxName = sessionPrefixForPort(port) + sessionId.slice(0, 8); + const tmuxName = name || (sessionPrefixForPort(port) + sid.slice(0, 8)); const ehome = home || process.env.HOME; // HOME claude runs under (scratch or real) const rhome = realHome || process.env.HOME; // real home (OAuth + onboarded config source) @@ -460,42 +528,97 @@ export async function runTuiTurn({ if (!existsSync(cwd)) mkdirSync(cwd, { recursive: true }); prepareTuiHome(rhome, ehome, cwd, { envTokenMode }); - // Write prompt to a temp file (mode 0600) so the content never touches argv. - const tmpDir = mkdtempSync(`${tmpdir()}/ocp-tui-`); - const promptFile = `${tmpDir}/prompt.txt`; - writeFileSync(promptFile, prompt, { mode: 0o600 }); - // Minimal env for spawnSync (tmux itself). The pane's claude env comes exclusively // from the `env` prefix string built inside buildTuiCmd — tmux does NOT forward the // spawning process's env to the pane, so the {env} here is intentionally minimal. const env = { ...process.env }; env.HOME = ehome; // tmux needs HOME; all claude-specific vars go via buildTuiCmd prefix + // Boot the interactive session inside tmux, rooted at the scratch cwd. + // Capture the result: if tmux new-session fails (status !== 0) there is no PTY, no + // interactive spawn — abort BEFORE the boot wait rather than paste into a non-existent + // session or issue a billing request without a verified interactive context. + const spawnResult = tmux( + ["new-session", "-d", "-s", tmuxName, "-x", "220", "-y", "50", "-c", cwd, + buildTuiCmd(claudeBin, model, sid, ehome, entrypointMode)], + { env }, + ); + if (!spawnResult || spawnResult.status !== 0) { + throw new Error("tui_spawn_failed: tmux session not created"); + } + + // Wait until claude's input bar is actually ready (not a blind sleep). + // bootMs is the MAX readiness wait, not a fixed delay. + const ready = await pollUntil(() => tuiInputReady(tuiCapturePane(tmux, tmuxName)), + { timeoutMs: bootMs, intervalMs: READY_POLL_MS }); + if (!ready) { + if (requireReady) { + try { tmux(["kill-session", "-t", tmuxName]); } catch { /* already gone */ } + throw new Error("tui_pane_not_ready: input bar did not appear within " + bootMs + "ms"); + } + // Cold path (pre-existing behaviour): readiness timed out; rely on paste-verify. + console.error("[tui] input_not_ready", tmuxName); + } + return { name: tmuxName, sessionId: sid, model, ehome, bootedAt: Date.now() }; +} + +// Full per-request TUI lifecycle: +// 1. Take a WARM pane from the pool if one is available for this model (opt-in; +// OCP_TUI_POOL_SIZE=0 => always null => steps 2-3 below are exactly today's path). +// A pooled pane is SINGLE-USE: it already carries its own fresh --session-id, it +// serves this one turn, and it is killed in the finally like any other pane. +// 2. On a MISS: pre-trust the scratch cwd, boot an interactive `claude` in a fresh tmux +// session in the scratch cwd, poll capture-pane until the `? for shortcuts` input bar +// appears (bootTuiPane). BOOT_MS is the max wait, not a fixed delay. +// 3. Write prompt to a 0600 temp file (no shell injection from prompt content). +// 4. Paste the prompt via tmux load-buffer + paste-buffer -p (bracketed paste) — +// reliable for large multi-line prompts where send-keys -l is not (issue #130). +// Poll-verify the prompt landed in the input (placeholder gone / [Pasted text]); +// fast-fail with tui_paste_not_landed if it never lands (prevents the 120s +// wallclock "stuck typing" hang). Then submit with a SEPARATE Enter key event. +// 5. Block on the native JSONL transcript (located by THIS pane's session-id) until +// terminal marker or wall-clock cap. +// 6. Always teardown: kill session + rm temp dir (even on throw), and kick a background +// pool refill so the next request finds a warm pane. +// Returns { text, entrypoint } from readTuiTranscript (entrypoint is the billing-pool +// classifier, e.g. "cli", or null if the transcript did not include a turn_duration). +export async function runTuiTurn({ + prompt, + model, + claudeBin, + home, + realHome, + cwd, + port, + wallclockMs = 120000, + entrypointMode = "cli", + tmux = defaultTmux, + pool = null, // TuiPanePool | null — null (default) === today's cold-boot-only path + onPane = null, // optional observer: ({ warm }) => void, for logging/metrics +}) { + // 1. Warm pane, or cold boot. A MISS is never an error — it is exactly today's path. + let pane = pool ? pool.acquire(model) : null; + const warm = !!pane; + // Kick the refill IMMEDIATELY (not after the turn): the replacement pane then boots + // CONCURRENTLY with this turn and is warm by the time the next request arrives. Also + // runs on a MISS — acquire() has just retargeted the pool to this model, so the miss + // that cold-boots today warms the pool for the next caller. Fire-and-forget; it takes + // no TuiSemaphore slot (see pool.refill's SLOT ACCOUNTING note). + if (pool) pool.refill(); + if (onPane) { try { onPane({ warm }); } catch { /* observer must never break a turn */ } } + if (!pane) { + pane = await bootTuiPane({ model, claudeBin, home, realHome, cwd, port, entrypointMode, tmux }); + } + const tmuxName = pane.name; + const sessionId = pane.sessionId; // THIS pane's own session-id — one session, one turn + const ehome = pane.ehome || home || process.env.HOME; + + // Write prompt to a temp file (mode 0600) so the content never touches argv. + const tmpDir = mkdtempSync(`${tmpdir()}/ocp-tui-`); + const promptFile = `${tmpDir}/prompt.txt`; + writeFileSync(promptFile, prompt, { mode: 0o600 }); + try { - // 1. Boot the interactive session inside tmux, rooted at the scratch cwd. - // Capture the result: if tmux new-session fails (status !== 0) there is no - // PTY, no interactive spawn — abort BEFORE the boot sleep rather than paste - // into a non-existent session or issue a billing request without a verified - // interactive context. The finally teardown is still harmless (kill-session - // is a no-op when the session never existed). - const spawnResult = tmux( - ["new-session", "-d", "-s", tmuxName, "-x", "220", "-y", "50", "-c", cwd, - buildTuiCmd(claudeBin, model, sessionId, ehome, entrypointMode)], - { env }, - ); - if (!spawnResult || spawnResult.status !== 0) { - throw new Error("tui_spawn_failed: tmux session not created"); - } - - // 2. Wait until claude's input bar is actually ready (was: blind sleep(BOOT_MS)). - // BOOT_MS is now the MAX readiness wait, not a fixed delay. - const ready = await pollUntil(() => tuiInputReady(tuiCapturePane(tmux, tmuxName)), - { timeoutMs: BOOT_MS, intervalMs: READY_POLL_MS }); - if (!ready) { - // (readiness timed out; relying on paste-verify) - console.error("[tui] input_not_ready", tmuxName); - } - // 3. Paste the prompt via a tmux PASTE BUFFER with bracketed paste (-p), NOT // `send-keys -l`. send-keys of a large multi-line prompt is unreliable: the // embedded newlines arrive as separate key events (effectively repeated Enter), @@ -521,11 +644,12 @@ export async function runTuiTurn({ // Submit (separate Enter key event). tmux(["send-keys", "-t", tmuxName, "Enter"]); - // 4. Block on the native transcript (resolved by session-id) until terminal. + // 5. Block on the native transcript (resolved by THIS pane's session-id) until terminal. // Returns { text, entrypoint } from readTuiTranscript. return await readTuiTranscript({ home: ehome, sessionId, wallclockMs }); } finally { - // 5. Teardown — always, even on throw. + // 6. Teardown — always, even on throw. A pooled pane is torn down here exactly like a + // cold-booted one: SINGLE-USE, never returned to the pool (see pool.mjs). try { tmux(["kill-session", "-t", tmuxName]); } catch { /* already gone */ } try { rmSync(tmpDir, { recursive: true, force: true }); } catch { /* best effort */ } } diff --git a/lib/tui/transcript.mjs b/lib/tui/transcript.mjs index e37ae23..bde66f2 100644 --- a/lib/tui/transcript.mjs +++ b/lib/tui/transcript.mjs @@ -73,6 +73,17 @@ export function isTerminalLine(obj) { // transcript holding one logical exchange). If a future warm-pool ever reuses a // session WITHOUT a fresh session-id / clear, earlier-turn text could leak — that // author must add user-line scoping here. See spec §7.2. +// +// STATUS (warm pool, lib/tui/pool.mjs — the "future warm-pool" this note anticipated): +// the pool does NOT reuse sessions, so the precondition above still holds and no +// user-line scoping was added. Each pooled pane is booted with its OWN fresh +// randomUUID() --session-id (bootTuiPane) and is SINGLE-USE: it serves exactly one turn +// and is then killed and replaced. One session still means one logical exchange, so the +// last assistant entry is still that request's answer. +// The warning therefore stands UNCHANGED for anyone who later wants a pane to serve a +// SECOND turn (or to reset one with /clear and reuse it): that is a leak, and it needs +// user-line scoping HERE before it can be safe. Do not relax pool.mjs's single-use rule +// without doing that work first. export function extractLatestAssistantText(events) { let text = ""; for (const ev of events) { diff --git a/server.mjs b/server.mjs index 9f3c7bd..2f0f833 100644 --- a/server.mjs +++ b/server.mjs @@ -22,6 +22,8 @@ * CLAUDE_MAX_CONCURRENT — max concurrent claude processes, -p/stream-json path (default: 8) * CLAUDE_MAX_QUEUE — max requests waiting for a -p slot before HTTP 429 (default: 16) * OCP_TUI_MAX_CONCURRENT — max concurrent interactive TUI turns, TUI-mode path (default: 2) + * OCP_TUI_POOL_SIZE — pre-booted warm `claude` panes held for TUI-mode (default: 0 = off; + * max 4). Each is a live idle process; cuts ~3-4s per request. * OCP_SPAWN_REAL_HOME — "1" forces the -p spawn to use the real HOME (disables the * latency spawn-home isolation; default: isolated when a token exists) * CLAUDE_BREAKER_THRESHOLD — failures in window before circuit opens (default: 6) @@ -32,7 +34,7 @@ * CLAUDE_HEARTBEAT_INTERVAL — SSE heartbeat interval in ms on streaming path (default: 0 = disabled) */ import { createServer } from "node:http"; -import { spawn, execFileSync } from "node:child_process"; +import { spawn, execFileSync, spawnSync } from "node:child_process"; import { randomUUID, timingSafeEqual } from "node:crypto"; import { readFileSync, readdirSync, accessSync, existsSync, constants, chmodSync, statSync, mkdirSync, writeFileSync, rmSync } from "node:fs"; import { fileURLToPath } from "node:url"; @@ -41,9 +43,10 @@ import { homedir } from "node:os"; import { validateKey, recordUsage, getUsageByKey, getUsageTimeline, getRecentUsage, createKey, listKeys, revokeKey, closeDb, checkQuota, updateKeyQuota, getKeyQuota, findKey, cacheHash, getCachedResponse, setCachedResponse, clearCache, getCacheStats, hasCacheControl, singleflight, getInflightStats } from "./keys.mjs"; import { DEFAULT_PORT } from "./lib/constants.mjs"; import { isLoopbackBind } from "./lib/net.mjs"; -import { runTuiTurn, reapStaleTuiSessions, resolveTuiHome } from "./lib/tui/session.mjs"; +import { runTuiTurn, reapStaleTuiSessions, resolveTuiHome, bootTuiPane, tuiPaneHealthy, poolPaneName, POOL_BOOT_MS } from "./lib/tui/session.mjs"; import { detectTuiUpstreamError } from "./lib/tui/transcript.mjs"; import { TuiSemaphore, SemaphoreAbortError, recordTuiEntrypoint, buildTuiHealthBlock } from "./lib/tui/semaphore.mjs"; +import { TuiPanePool, resolvePoolSize, POOL_MAX_SIZE } from "./lib/tui/pool.mjs"; import { createSerialMutex, createTtlCache, isTokenExpiring, orderLabelsLastGoodFirst } from "./lib/spawn-auth.mjs"; const __dirname = dirname(fileURLToPath(import.meta.url)); @@ -351,6 +354,48 @@ const tuiStats = { entrypointMismatches: 0, // count of cli-expected-but-got-other turns }; +// ── Warm pane pool (docs/plans/2026-07-13-tui-latency #3) — opt-in; default OFF ───────── +// OCP_TUI_POOL_SIZE=0 (default) => tuiPool is null => runTuiTurn's cold-boot path is +// byte-for-byte unchanged. Set it to N (clamped to POOL_MAX_SIZE) to keep N pre-booted +// `claude` panes warm, each SINGLE-USE (see lib/tui/pool.mjs for why single-use is the +// load-bearing rule, and lib/tui/session.mjs for the POOL/REAPER INVARIANT). +// +// Default-off is deliberate on a stable production path: a warm pane is a LIVE idle +// `claude` process held whether or not a request ever arrives, so the operator must opt +// in to that standing cost. Measured saving when on (this host, Sonnet 4.6, --effort low): +// end-to-end p50 10.17 s (n=6, pool off) -> 6.00 s (n=12 warm hits), i.e. -41%. +// cli.js does NOT perform this operation (Class B, OCP-owned TUI spawn) — see ADR 0007. +const TUI_POOL_SIZE = TUI_MODE ? resolvePoolSize(process.env.OCP_TUI_POOL_SIZE) : 0; +const tuiPool = TUI_POOL_SIZE > 0 + ? new TuiPanePool({ + size: TUI_POOL_SIZE, + // The POOL mints the pane's identity, not bootTuiPane: the tmux session exists the + // instant the boot starts, so the pool must be able to name (hence spare, hence kill) + // it before then. Name is derived from the session-id, so `tmux ls` correlates to the + // transcript file /.claude/projects/*/.jsonl. + mintPane: () => { + const sessionId = randomUUID(); + return { sessionId, name: poolPaneName(PORT, sessionId) }; + }, + bootPane: (model, ident) => bootTuiPane({ + model, + claudeBin: CLAUDE, + home: TUI_HOME, + realHome: process.env.HOME, + cwd: TUI_CWD, + port: PORT, + entrypointMode: TUI_ENTRYPOINT, + sessionId: ident.sessionId, + name: ident.name, + requireReady: true, // a pane that never reached its input bar must not be enlisted + bootMs: POOL_BOOT_MS, // background pre-boot — no client is blocked, so be patient + }), + killPane: (name) => { try { spawnSync(process.env.OCP_TUI_TMUX_BIN || "tmux", ["kill-session", "-t", name]); } catch { /* already gone */ } }, + paneHealthy: (name) => tuiPaneHealthy((args) => spawnSync(process.env.OCP_TUI_TMUX_BIN || "tmux", args, { encoding: "utf8" }), name), + log: (level, event, data) => logEvent(level, event, data), + }) + : null; + // ── FIX ③ (latency): default-path (-p / stream-json) spawn-home isolation ────────────── // PROBLEM (measured, not theoretical): OCP's default spawn inherits the operator's real HOME // (loading the global ~/.claude — plugins, skills, hooks) and runs with cwd=~/ocp (loading the @@ -770,19 +815,45 @@ const cacheCleanupInterval = setInterval(() => { // mechanism and the 15-min cadence makes the window negligible). // Gated on TUI_MODE — zero effect (no kill-server, no list-sessions) when TUI is off. // cli.js does NOT perform this operation (Class B, OCP-owned TUI spawn) — see ADR 0007. +// +// WARM POOL INTERACTION (the crux — see the POOL/REAPER INVARIANT in lib/tui/session.mjs). +// A warm pooled pane is one of OUR OWN ocp-tui--* sessions that is alive and idle BY +// DESIGN, and this sweep fires precisely when the instance is idle — i.e. exactly when the +// pool is full. Two things are therefore required, and both are done here: +// (a) DRAIN the pool BEFORE the sweep. Zombie reaping is possible ONLY via kill-server, +// and a live pooled pane suppresses kill-server (it is a live child of the tmux +// server). A permanently-full pool would otherwise permanently disable the very +// thing this tick exists to do. Draining costs one pane re-boot per tick (~1.2 s of +// background work every 15 min) and is invisible to callers: a request landing in the +// drain→refill gap simply MISSES the pool and takes today's cold path. +// (b) Pass the pool's live registry as `spare` anyway. After (a) it is empty, so this is +// belt-and-braces — it makes it impossible for THIS call site (or a future one) to +// kill a live pooled pane even if the drain were ever removed or reordered. +// RESIDUAL (unchanged in kind from the pre-pool code, and explicitly accepted there): a +// request arriving in the narrow window between the idle-check and kill-server has its pane +// torn down and fails cleanly via runTuiTurn's honesty gates. The drain widens that window +// by the cost of N kill-session calls (single-digit ms), not materially. const TUI_REAP_INTERVAL_MS = 15 * 60 * 1000; const tuiReapInterval = TUI_MODE ? setInterval(() => { if (tuiSemaphore.inflight > 0 || tuiSemaphore.queued > 0) return; // a turn is live — defer try { + const drained = tuiPool ? tuiPool.drain() : 0; // F7 fix: scope to THIS instance's own port; a sibling ocp-tui--* session // (a second OCP instance on the same host) is treated as foreign, same as olp-tui-*. // includeLegacy is NOT set here — see reapStaleTuiSessions' comment: the periodic sweep // conservatively treats any lingering bare-prefix legacy session as foreign so it can // never trigger kill-server on a steady-state tick; only the one-time boot reap below // claims legacy-shaped zombies. - const n = reapStaleTuiSessions({ port: PORT }); - if (n) logEvent("info", "tui_reaped_stale_sessions", { count: n, trigger: "periodic" }); + const n = reapStaleTuiSessions({ port: PORT, spare: tuiPool ? tuiPool.liveNames() : null }); + if (n || drained) { + logEvent("info", "tui_reaped_stale_sessions", { count: n, poolDrained: drained, trigger: "periodic" }); + } } catch (e) { logEvent("error", "tui_periodic_reap_failed", { error: e.message }); } + finally { + // Refill in the background regardless of how the sweep went — a throw mid-sweep must not + // leave the pool permanently paused (it would silently degrade to the cold path forever). + if (tuiPool) { try { tuiPool.resume(); } catch { /* best effort */ } } + } }, TUI_REAP_INTERVAL_MS) : null; if (tuiReapInterval && typeof tuiReapInterval.unref === "function") tuiReapInterval.unref(); @@ -1323,6 +1394,15 @@ async function callClaudeTui(model, messages, _conversationId, _keyName, res) { // different port never collides with this instance's reap/kill-server logic. wallclockMs: TUI_WALLCLOCK_MS, entrypointMode: TUI_ENTRYPOINT, + // Warm pane pool (null unless OCP_TUI_POOL_SIZE > 0 → today's cold path exactly). + // A pooled pane is single-use: runTuiTurn kills it in its finally like any other. + pool: tuiPool, + // Only observe when the pool is ON — with it off (the default) no new log line is + // emitted, so the disabled path stays byte-for-byte today's, logs included. + onPane: tuiPool + ? ({ warm }) => logEvent("info", warm ? "tui_pool_hit" : "tui_pool_miss", + { model: cliModel, warmRemaining: tuiPool.warm }) + : null, }); // ── Honesty gates (issue #133) ─ run BEFORE recordModelSuccess / cache write-back. // A throw here propagates to the catch below (recordModelError + reject), so the @@ -2558,9 +2638,12 @@ const server = createServer(async (req, res) => { // still appears with enabled:false (cheap, harmless) so the shape is stable. // entrypointMismatches/lastEntrypoint exist so an operator can poll /health to catch a // silent metered-pool drift (the audit's top risk after the 6/15 billing flip). + // `pool` is a NEW nested field inside the (already additive) tui block: null when the + // warm pool is off (the default), so the disabled shape is unchanged apart from one + // explicit null. Lets the operator confirm hit rate + standing process cost. tui: buildTuiHealthBlock( { enabled: TUI_MODE, entrypointMode: TUI_ENTRYPOINT, maxConcurrent: TUI_MAX_CONCURRENT }, - tuiStats, tuiSemaphore, + tuiStats, tuiSemaphore, tuiPool, ), }); } @@ -2797,6 +2880,27 @@ function gracefulShutdown(signal) { if (tuiReapInterval) clearInterval(tuiReapInterval); closeDb(); + // 2b. Drain the warm pane pool. A pooled `claude` is a child of the tmux SERVER, not of + // this node process, so it is NOT in activeProcesses and step 3 below cannot reach it — + // without this explicit drain every warm pane would outlive OCP as an orphan (and the + // pool's in-memory registry dies with the process, so nothing would remember it owned them). + // + // drain() kills the pane that is currently BOOTING too, and it does so SYNCHRONOUSLY. That + // is required, not incidental: step 4 below calls process.exit(0) in THIS SAME TICK whenever + // activeProcesses is empty — which on a TUI host it always is — so any cleanup a boot + // deferred to a .then()/.catch() would simply never run. (That was a real bug: the pool used + // to track in-flight boots as a count, could not name the booting session, and orphaned a + // live authenticated `claude` on every shutdown that landed mid-boot.) + // + // Orphans that survive anyway (SIGKILL, power loss) are still caught by the next instance's + // boot reap — this makes the graceful path clean, it is not the only safety net. + if (tuiPool) { + try { + const drained = tuiPool.drain(); + if (drained) logEvent("info", "tui_pool_drained", { count: drained, trigger: "shutdown" }); + } catch (e) { logEvent("error", "tui_pool_drain_failed", { error: e.message }); } + } + // 3. Kill all active child processes for (const proc of activeProcesses) { try { proc.kill("SIGTERM"); } catch {} @@ -2866,11 +2970,19 @@ server.listen(PORT, BIND_ADDRESS, () => { ? (TUI_HOME === process.env.HOME ? "env-token (real home — unset OCP_TUI_HOME for credential isolation)" : "env-token (credential-isolated home — no credentials.json)") : "credentials.json (no CLAUDE_CODE_OAUTH_TOKEN — see Troubleshooting #401)"; console.log(` TUI-mode: ON home=${TUI_HOME} cwd=${TUI_CWD} auth=${tuiAuth} wallclock=${TUI_WALLCLOCK_MS}ms maxConcurrent=${TUI_MAX_CONCURRENT}`); + console.log(TUI_POOL_SIZE > 0 + ? ` TUI warm pool: ON size=${TUI_POOL_SIZE} — ${TUI_POOL_SIZE} idle \`claude\` process(es) held warm; first request per model is still a cold MISS` + : ` TUI warm pool: OFF (set OCP_TUI_POOL_SIZE=1..${POOL_MAX_SIZE} to pre-boot panes and cut ~3-4s per request)`); try { // F7 fix: scope to THIS instance's own port (see reapStaleTuiSessions). includeLegacy: // true ONLY here — the one-time boot reap is the designated point to claim orphaned // bare-prefix ("ocp-tui-") zombie sessions left by a PRE-fix process generation // of this same instance (no live post-fix instance ever creates that shape again). + // No `spare`: the warm pool is EMPTY at boot (there is no boot-time pre-warm — the pool + // learns its model from the first request), so this reap has no live pane to protect and + // it is exactly what SHOULD claim any ocp-tui--p* pool orphans left by a previous + // process generation of this instance (POOL/REAPER INVARIANT property 2). If a future + // change ever pre-warms at boot, this call MUST start passing tuiPool.liveNames(). const n = reapStaleTuiSessions({ port: PORT, includeLegacy: true }); if (n) logEvent("info", "tui_reaped_stale_sessions", { count: n }); } catch {} diff --git a/test-features.mjs b/test-features.mjs index dbec3cd..4e055f7 100644 --- a/test-features.mjs +++ b/test-features.mjs @@ -2027,6 +2027,457 @@ test("reaper with includeLegacy=true still spares a sibling instance's port-scop assert.ok(!calls.includes("kill-server"), "sibling instance's live session still blocks kill-server"); }); +// ── TUI warm pane pool (docs/plans/2026-07-13-tui-latency #3) ──────────── +import { TuiPanePool, resolvePoolSize, POOL_MAX_SIZE, POOL_MAX_AGE_MS } from "./lib/tui/pool.mjs"; +import { poolPaneName as poolName } from "./lib/tui/session.mjs"; + +// A pool wired to fakes: no tmux, no claude. bootPane resolves on the microtask queue; use +// `await settle()` after a refill() to let the SERIALIZED boot chain run to target. +// `live` models the real tmux server: bootTuiPane creates the session SYNCHRONOUSLY and only +// THEN waits (up to POOL_BOOT_MS) for the input bar, so the fake boot registers the session +// immediately and only afterwards resolves. `opts.hold` keeps a boot in that mid-flight window +// so tests can act on a pane that is live-but-not-yet-warm — the state that hid two bugs. +function makeFakePool(opts = {}) { + const killed = []; + const booted = []; + const live = new Set(); // "tmux sessions" that currently exist + let seq = 0; + let clock = 1_000_000; + const healthy = new Set(); + // FIFO gate queue — one entry per in-flight held boot. A single `release` slot would be + // OVERWRITTEN by a later boot, so releasing "the first boot" would silently release the + // second instead (and mask the stale-settle bug this harness exists to test). + const gates = []; + const pool = new TuiPanePool({ + size: opts.size ?? 2, + maxAgeMs: opts.maxAgeMs ?? POOL_MAX_AGE_MS, + now: () => clock, + mintPane: () => { + const n = ++seq; + return { sessionId: `sid-${n}`, name: `ocp-tui-3456-p${String(n).padStart(8, "0")}` }; + }, + bootPane: async (model, { sessionId, name }) => { + live.add(name); // session exists NOW + booted.push({ name, model }); + if (opts.hold) { + await new Promise((r) => gates.push(r)); // ...stuck waiting for readiness + } + if (opts.bootThrows) { live.delete(name); throw new Error("boom"); } + // A pane whose session was killed while booting can never become ready — exactly what + // the real bootTuiPane does (it throws tui_pane_not_ready). + if (!live.has(name)) throw new Error("tui_pane_not_ready"); + healthy.add(name); + return { name, sessionId, model, bootedAt: clock }; + }, + killPane: (name) => { killed.push(name); healthy.delete(name); live.delete(name); }, + paneHealthy: (name) => healthy.has(name), + }); + return { + pool, killed, booted, healthy, live, + releaseBoot: () => { const r = gates.shift(); if (r) r(); }, // release the OLDEST held boot + advance: (ms) => { clock += ms; }, at: () => clock, + }; +} +const tick = () => new Promise((r) => setImmediate(r)); +// Refills are SERIALIZED (one boot at a time, re-kicked on success), so settling the pool +// takes a chain of microtask turns, not one. 40 is far more than POOL_MAX_SIZE needs. +const settle = async () => { for (let i = 0; i < 40; i++) await tick(); }; + +console.log("\nTUI warm pane pool (acquire / miss / refill / TTL / reaper exemption):"); + +test("resolvePoolSize: default/garbage/negative disable the pool; size is clamped to POOL_MAX_SIZE", () => { + assert.equal(resolvePoolSize(undefined), 0, "unset => off (byte-for-byte today's cold path)"); + assert.equal(resolvePoolSize("0"), 0); + assert.equal(resolvePoolSize("-3"), 0); + assert.equal(resolvePoolSize("banana"), 0, "garbage disables rather than guessing a size"); + assert.equal(resolvePoolSize("2"), 2); + assert.equal(resolvePoolSize("99"), POOL_MAX_SIZE, "clamped — never boot an unbounded number of idle claudes"); +}); + +test("pool size 0 is inert: acquire always misses and refill never boots", async () => { + const { pool, booted } = makeFakePool({ size: 0 }); + assert.equal(pool.enabled, false); + assert.equal(pool.acquire("m1"), null, "disabled pool always MISSES → caller cold-boots"); + pool.refill(); + await settle(); + assert.equal(booted.length, 0, "a disabled pool must never spawn a process"); +}); + +test("acquire MISSES on an empty pool, and the miss refills for the requested model", async () => { + const { pool, booted } = makeFakePool({ size: 2 }); + assert.equal(pool.acquire("sonnet"), null, "first request is always a MISS (no boot-time pre-warm)"); + assert.equal(pool.misses, 1); + pool.refill(); + await settle(); + assert.equal(pool.warm, 2, "refilled to target"); + assert.deepEqual(booted.map((b) => b.model), ["sonnet", "sonnet"], "warmed for the model that missed"); +}); + +test("acquire HITS a warm pane, hands it out ONCE, and never returns it (single-use)", async () => { + const { pool } = makeFakePool({ size: 2 }); + pool.acquire("sonnet"); pool.refill(); await settle(); + assert.equal(pool.warm, 2); + + const a = pool.acquire("sonnet"); + assert.ok(a && a.name && a.sessionId, "warm pane handed out"); + assert.equal(pool.hits, 1); + assert.equal(pool.warm, 1, "the pane LEAVES the registry when acquired"); + + const b = pool.acquire("sonnet"); + assert.notEqual(b.name, a.name, "a pane is NEVER handed out twice — single-use"); + assert.notEqual(b.sessionId, a.sessionId, "each pane carries its OWN fresh session-id (transcript.mjs scoping)"); + assert.equal(pool.warm, 0); + assert.equal(pool.acquire("sonnet"), null, "exhausted pool MISSES rather than reusing a pane"); +}); + +test("refill is bounded: never more than `size` panes, and concurrent refills do not overshoot", async () => { + const { pool, booted } = makeFakePool({ size: 2 }); + pool.acquire("sonnet"); + pool.refill(); pool.refill(); pool.refill(); // hammer it + await settle(); + assert.equal(pool.warm, 2, "still exactly `size` warm panes"); + assert.equal(booted.length, 2, "the _booting guard prevented duplicate boots"); +}); + +// Live finding at size=2: two cold `claude` boots racing an in-flight turn made a refill +// overrun even the generous pool readiness cap. Boots are therefore SERIALIZED. +test("refill boots panes ONE AT A TIME (never two claude cold-boots racing each other)", async () => { + let concurrent = 0, peak = 0; + let seq = 0; + const pool = new TuiPanePool({ + size: 3, + mintPane: () => { const n = ++seq; return { sessionId: `s${n}`, name: `p${n}` }; }, + bootPane: async (model, { sessionId, name }) => { + concurrent++; peak = Math.max(peak, concurrent); + await new Promise((r) => setImmediate(r)); // simulate boot latency + concurrent--; + return { name, sessionId, model, bootedAt: Date.now() }; + }, + killPane: () => {}, + paneHealthy: () => true, + }); + pool.acquire("sonnet"); + pool.refill(); + await settle(); + assert.equal(pool.warm, 3, "chain still reaches the target size"); + assert.equal(peak, 1, "at most ONE boot in flight at any moment"); +}); + +test("a FAILED boot does not re-kick the chain (backoff — a broken claude must not spin)", async () => { + const { pool, booted } = makeFakePool({ size: 3, bootThrows: true }); + pool.acquire("sonnet"); + pool.refill(); + await settle(); + assert.equal(pool.bootFailures, 1, "counted as a genuine failure (nobody cancelled it)"); + assert.equal(booted.length, 1, "exactly ONE attempt — a failure stops the chain, it does not respawn forever"); + assert.equal(pool.warm, 0); + assert.equal(pool.booting, 0, "and the booting slot is released, so the next trigger can retry"); +}); + +test("acquire drops an UNHEALTHY warm pane (kills it) and falls through to a MISS", async () => { + const { pool, killed, healthy, booted } = makeFakePool({ size: 1 }); + pool.acquire("sonnet"); pool.refill(); await settle(); + const dead = booted[0].name; + healthy.delete(dead); // pane died / stopped being input-ready while idle + + assert.equal(pool.acquire("sonnet"), null, "a dead pane must MISS, never hang a turn"); + assert.ok(killed.includes(dead), "the dead pane is killed, not leaked"); + assert.equal(pool.misses, 2); +}); + +test("acquire drops an EXPIRED warm pane (older than maxAgeMs)", async () => { + const { pool, killed, booted, advance } = makeFakePool({ size: 1, maxAgeMs: 60_000 }); + pool.acquire("sonnet"); pool.refill(); await settle(); + advance(60_001); + assert.equal(pool.acquire("sonnet"), null, "a pane past its TTL is not handed out"); + assert.ok(killed.includes(booted[0].name), "expired pane is killed"); +}); + +test("a model switch drops the wrong-model panes and retargets the pool (--model is fixed at spawn)", async () => { + const { pool, killed, booted } = makeFakePool({ size: 2 }); + pool.acquire("sonnet"); pool.refill(); await settle(); + const sonnetPanes = booted.map((b) => b.name); + + assert.equal(pool.acquire("opus"), null, "different model => MISS (a sonnet pane cannot serve opus)"); + assert.equal(pool.warm, 0, "sonnet panes dropped"); + for (const p of sonnetPanes) assert.ok(killed.includes(p), "wrong-model pane killed, not leaked"); + assert.equal(pool.warmModel, "opus", "pool retargeted to the model actually being asked for"); + + pool.refill(); await settle(); + assert.deepEqual(booted.slice(2).map((b) => b.model), ["opus", "opus"], "refilled for the NEW model"); +}); + +test("a boot that resolves AFTER a drain kills its own pane instead of enlisting it", async () => { + const { pool, killed } = makeFakePool({ size: 1 }); + pool.acquire("sonnet"); + pool.refill(); // boot is in flight... + pool.drain(); // ...pool drained before it resolves (shutdown / reap sweep) + await settle(); + assert.equal(pool.warm, 0, "the late pane must NOT be enlisted into a drained pool"); + assert.equal(killed.length, 1, "it kills itself — no orphan process left behind"); +}); + +test("bootPane failure is counted, never thrown into the request path, and does not wedge refill", async () => { + const { pool } = makeFakePool({ size: 1, bootThrows: true }); + pool.acquire("sonnet"); + pool.refill(); + await settle(); + assert.equal(pool.warm, 0); + assert.equal(pool.bootFailures, 1); + assert.equal(pool.booting, 0, "the _booting counter is released on failure (else refill wedges forever)"); + assert.equal(pool.acquire("sonnet"), null, "and the caller just MISSES → cold path"); +}); + +test("drain kills every warm pane and pauses refills; resume restarts them", async () => { + const { pool, killed } = makeFakePool({ size: 2 }); + pool.acquire("sonnet"); pool.refill(); await settle(); + assert.equal(pool.warm, 2); + + assert.equal(pool.drain(), 2, "drain reports how many it killed"); + assert.equal(pool.warm, 0); + assert.equal(killed.length, 2, "both panes killed — none outlive the drain"); + + pool.refill(); await settle(); + assert.equal(pool.warm, 0, "refill is a NO-OP while drained (paused)"); + + pool.resume(); await settle(); + assert.equal(pool.warm, 2, "resume refills"); +}); + +// ── The crux: pool ↔ reaper coexistence (POOL/REAPER INVARIANT, lib/tui/session.mjs) ── +console.log("\nTUI warm pool ↔ session reaper coexistence:"); + +test("INVARIANT 1: a LIVE pooled pane is NEVER reaped (it is in the spare set)", () => { + const killed = []; + const live = "ocp-tui-3456-pdeadbeef"; + const fakeTmux = (args) => { + if (args[0] === "list-sessions") return { status: 0, stdout: `${live}\nocp-tui-3456-aaaa\n` }; + if (args[0] === "kill-session") { killed.push(args[args.indexOf("-t") + 1]); return { status: 0 }; } + return { status: 0, stdout: "" }; + }; + const n = reapStaleTuiSessions({ tmux: fakeTmux, port: 3456, spare: new Set([live]) }); + assert.equal(n, 1, "only the stale turn session was reaped"); + assert.ok(!killed.includes(live), "the live warm pane must survive the sweep"); + assert.ok(killed.includes("ocp-tui-3456-aaaa"), "a genuinely stale own session is still reaped"); +}); + +test("INVARIANT 2: an ORPHANED pooled pane (pool-shaped but NOT in the spare set) IS reaped", () => { + const killed = []; + // ocp-tui-3456-porphan01 LOOKS pooled but the live registry does not claim it — e.g. left + // behind by a previous process generation, whose in-memory registry died with it. + const fakeTmux = (args) => { + if (args[0] === "list-sessions") return { status: 0, stdout: "ocp-tui-3456-porphan01\nocp-tui-3456-plive0001\n" }; + if (args[0] === "kill-session") { killed.push(args[args.indexOf("-t") + 1]); return { status: 0 }; } + return { status: 0, stdout: "" }; + }; + const n = reapStaleTuiSessions({ tmux: fakeTmux, port: 3456, spare: new Set(["ocp-tui-3456-plive0001"]) }); + assert.equal(n, 1); + assert.deepEqual(killed, ["ocp-tui-3456-porphan01"], "exemption is by EXACT NAME, never by name shape"); +}); + +test("INVARIANT 2b: with NO spare set (the pre-pool call shape) pool-shaped panes are reaped — fail-safe", () => { + const killed = []; + const fakeTmux = (args) => { + if (args[0] === "list-sessions") return { status: 0, stdout: "ocp-tui-3456-pdeadbeef\n" }; + if (args[0] === "kill-session") { killed.push(args[args.indexOf("-t") + 1]); return { status: 0 }; } + return { status: 0, stdout: "" }; + }; + const n = reapStaleTuiSessions({ tmux: fakeTmux, port: 3456 }); + assert.equal(n, 1, "omitting `spare` reaps MORE, never less — forgetting it can't leak panes"); + assert.deepEqual(killed, ["ocp-tui-3456-pdeadbeef"]); +}); + +test("INVARIANT 3: kill-server is SUPPRESSED while a live pooled pane is spared", () => { + const calls = []; + const live = "ocp-tui-3456-plive0001"; + const fakeTmux = (args) => { + calls.push(args.join(" ")); + if (args[0] === "list-sessions") return { status: 0, stdout: `${live}\nocp-tui-3456-aaaa\n` }; + return { status: 0, stdout: "" }; + }; + reapStaleTuiSessions({ tmux: fakeTmux, port: 3456, spare: new Set([live]) }); + assert.ok(!calls.includes("kill-server"), "kill-server would kill the live pane (a child of the tmux server)"); +}); + +test("INVARIANT 3b: after a DRAIN the spare set is empty, so kill-server fires again (zombie reaping preserved)", async () => { + // This is the whole reason server.mjs drains BEFORE the periodic sweep: a permanently-full + // pool would otherwise permanently suppress the only mechanism that reaps defunct claudes. + const { pool } = makeFakePool({ size: 2 }); + pool.acquire("sonnet"); pool.refill(); await settle(); + assert.equal(pool.liveNames().size, 2, "pool is full → the sweep would be suppressed"); + + pool.drain(); + assert.equal(pool.liveNames().size, 0, "drain empties the live registry"); + + const calls = []; + const fakeTmux = (args) => { + calls.push(args.join(" ")); + if (args[0] === "list-sessions") return { status: 0, stdout: "ocp-tui-3456-aaaa\n" }; + return { status: 0, stdout: "" }; + }; + reapStaleTuiSessions({ tmux: fakeTmux, port: 3456, spare: pool.liveNames() }); + assert.ok(calls.includes("kill-server"), "kill-server fires post-drain — defunct zombies still get reaped"); +}); + +// ── MID-BOOT: the state that hid M1a + M1b ──────────────────────────────────────────── +// bootTuiPane creates the tmux session SYNCHRONOUSLY, then waits up to POOL_BOOT_MS (20s) +// for the input bar. So a pooled session can be LIVE for ~20s before its boot resolves. +// Every reaper test above uses a pool that is either full or drained — never mid-boot. +// That gap is exactly why both bugs shipped past the first round of tests. + +test("M1a: a reap tick during an IN-FLIGHT BOOT must not orphan-kill the booting pane", async () => { + const { pool, live } = makeFakePool({ size: 1, hold: true }); + pool.acquire("sonnet"); + pool.refill(); + await tick(); // boot started; session live; NOT yet warm + assert.equal(pool.warm, 0, "not warm yet"); + assert.equal(pool.booting, 1, "a boot is in flight"); + assert.equal(live.size, 1, "...and its tmux session ALREADY EXISTS"); + + const bootingName = [...live][0]; + assert.ok(pool.liveNames().has(bootingName), + "REGRESSION GUARD: the booting pane MUST be nameable, or the sweep cannot spare it " + + "(the pool used to track in-flight boots as a COUNT and this was empty)"); +}); + +test("M1a: the reap tick's drain kills the booting pane, and resume() starts a FRESH boot", async () => { + const warns = []; + const { pool, live, releaseBoot } = makeFakePool({ size: 1, hold: true }); + pool._log = (lvl, ev) => { if (lvl === "warn") warns.push(ev); }; + pool.acquire("sonnet"); + pool.refill(); + await tick(); + const first = [...live][0]; + + // The reap tick, as server.mjs runs it: drain -> reap -> resume. + const drained = pool.drain(); + assert.equal(drained, 1, "drain accounts for the booting pane"); + assert.equal(live.size, 0, "its tmux session is killed — kill-server can now flush zombies"); + assert.equal(pool.liveNames().size, 0, "nothing left to spare, so kill-server is not suppressed"); + + pool.resume(); + await tick(); + assert.equal(pool.booting, 1, "resume() started a FRESH boot — the pool is not left empty with nothing scheduled"); + assert.notEqual([...live][0], first, "and it is a NEW pane, not the killed one"); + + // Now let the ORIGINAL (cancelled) boot settle. It rejects with tui_pane_not_ready because + // we killed its session — but that is OUR doing, not a fault. + releaseBoot(); + await settle(); + assert.equal(pool.bootFailures, 0, + "a cancelled boot must NOT be counted as a bootFailure — that is the WARN operators alert on"); + assert.deepEqual(warns, [], "and it must not log tui_pool_boot_failed for a healthy drain"); + assert.equal(pool.cancelled, 1, "it is counted as a cancellation instead (counted exactly once)"); +}); + +test("M1b: shutdown drain kills the booting pane SYNCHRONOUSLY — no orphaned claude", async () => { + const { pool, live } = makeFakePool({ size: 1, hold: true }); + pool.acquire("sonnet"); + pool.refill(); + await tick(); + assert.equal(live.size, 1, "a live pooled session exists"); + + // gracefulShutdown: drain() then process.exit(0) IN THE SAME TICK (TUI panes are tmux + // children, not node children, so activeProcesses is empty and the exit is immediate). + // Nothing scheduled on the microtask queue can run. So we assert WITHOUT awaiting. + pool.drain(); + assert.equal(live.size, 0, + "REGRESSION GUARD: the pane must be dead BEFORE any await. A .then()-based cleanup would " + + "never run before process.exit and would orphan a live authenticated `claude`."); +}); + +test("a stale boot settling after drain+resume must not clear the NEW boot's slot", async () => { + const { pool, releaseBoot } = makeFakePool({ size: 1, hold: true }); + pool.acquire("sonnet"); + pool.refill(); + await tick(); + pool.drain(); // cancels boot #1 (generation bumped) + pool.resume(); + await tick(); + assert.equal(pool.booting, 1, "boot #2 owns the slot"); + releaseBoot(); // boot #1 finally settles (rejects) + await settle(); + assert.equal(pool.booting, 1, "boot #2 STILL owns the slot — a stale settle must not free it"); +}); + +test("a model switch cancels an in-flight boot for the OLD model (kills it, frees the slot)", async () => { + const { pool, live, killed } = makeFakePool({ size: 1, hold: true }); + pool.acquire("sonnet"); + pool.refill(); + await tick(); + const sonnetPane = [...live][0]; + + pool.acquire("opus"); // retarget mid-boot + assert.ok(killed.includes(sonnetPane), "the old model's booting pane is killed, not left to linger"); + assert.equal(pool.booting, 0, "and its slot is freed immediately, so the new model can boot now"); + assert.equal(pool.warmModel, "opus"); + + pool.refill(); + await tick(); + assert.equal(pool.booting, 1, "a boot for the NEW model starts without waiting out the old one"); +}); + +test("a pane handed out for a turn leaves the spare set immediately (so its teardown is authoritative)", async () => { + const { pool } = makeFakePool({ size: 2 }); + pool.acquire("sonnet"); pool.refill(); await settle(); + const taken = pool.acquire("sonnet"); + assert.ok(!pool.liveNames().has(taken.name), + "an acquired pane is the CALLER's — the pool must not also claim it live, or a crashed turn's pane would be spared forever"); +}); + +test("N1: the pool mints ONE identity — the tmux name's hex is the transcript session-id's hex", async () => { + // Without this, `tmux ls` shows a pane whose name has no relation to any transcript file, + // so a live pane cannot be correlated to /.claude/projects/*/.jsonl. + const seen = []; + const pool = new TuiPanePool({ + size: 1, + mintPane: () => { + const sessionId = "deadbeef-1111-2222-3333-444444444444"; + return { sessionId, name: poolName(3456, sessionId) }; + }, + bootPane: async (model, ident) => { + seen.push(ident); + return { ...ident, model, bootedAt: Date.now() }; + }, + killPane: () => {}, + paneHealthy: () => true, + }); + pool.acquire("sonnet"); + pool.refill(); + await settle(); + assert.equal(seen.length, 1, "bootPane received the pool-minted identity"); + assert.equal(seen[0].name, "ocp-tui-3456-pdeadbeef"); + assert.ok(seen[0].name.endsWith(seen[0].sessionId.slice(0, 8)), + "the tmux session name carries the session-id's own hex — `tmux ls` correlates to the transcript"); + const pane = pool.acquire("sonnet"); + assert.equal(pane.sessionId, seen[0].sessionId, + "and the turn reads the transcript under THAT session-id — one identity end to end"); +}); + +test("pool pane names are port-scoped (reapable as ours) and never match the legacy shape", () => { + const name = poolName(3456, "deadbeef-1111-2222-3333-444444444444"); + assert.ok(name.startsWith(sessionPrefixForPort(3456)), "pool panes are OURS → reapable when orphaned"); + assert.equal(name, "ocp-tui-3456-pdeadbeef"); + assert.ok(!LEGACY_SESSION_NAME_RE.test(name), "must never be mistaken for a legacy bare-prefix session"); + assert.ok(!poolName(9999, "aaaaaaaa-0000-0000-0000-000000000000").startsWith(sessionPrefixForPort(3456)), + "a sibling instance's pool pane is foreign to us"); +}); + +test("buildTuiHealthBlock reports pool:null when off, and the pool's stats when on", async () => { + const st = { lastEntrypoint: "cli", entrypointMismatches: 0 }; + const sem = { inflight: 0, queued: 0 }; + const off = buildTuiHealthBlock({ enabled: true, entrypointMode: "cli", maxConcurrent: 2 }, st, sem, null); + assert.equal(off.pool, null, "pool disabled → explicit null (stable /health shape)"); + + const { pool } = makeFakePool({ size: 2 }); + pool.acquire("sonnet"); pool.refill(); await settle(); + const on = buildTuiHealthBlock({ enabled: true, entrypointMode: "cli", maxConcurrent: 2 }, st, sem, pool); + assert.equal(on.pool.size, 2); + assert.equal(on.pool.warm, 2); + assert.equal(on.pool.misses, 1); + assert.equal(on.pool.model, "sonnet"); +}); + // ── TUI home preparation (scratch vs real) ─────────────────────────────── import { prepareTuiHome, ensureTuiCwdTrusted } from "./lib/tui/session.mjs"; import { mkdtempSync as hMkdtemp, mkdirSync as hMkdir, writeFileSync as hWrite, readFileSync as hRead, existsSync as hExists, readlinkSync as hReadlink } from "node:fs"; @@ -2474,8 +2925,13 @@ test("buildTuiHealthBlock: shape + live counters (the additive /health tui block const ts = { lastEntrypoint: "cli", entrypointMismatches: 3 }; const block = buildTuiHealthBlock( { enabled: true, entrypointMode: "cli", maxConcurrent: 2 }, ts, sem); + // `pool` joined this key set with the warm pane pool. The tui block is ADR-0007-owned (it + // did not exist at v3.16.4, so it is outside ADR 0006's grandfather freeze), and the + // addition is purely additive: every pre-existing key below still carries a byte-identical + // value, and `pool` is null unless the operator opts in via OCP_TUI_POOL_SIZE. assert.deepEqual(Object.keys(block).sort(), - ["enabled", "entrypointMismatches", "entrypointMode", "inflight", "lastEntrypoint", "maxConcurrent", "queued"]); + ["enabled", "entrypointMismatches", "entrypointMode", "inflight", "lastEntrypoint", "maxConcurrent", "pool", "queued"]); + assert.equal(block.pool, null, "no pool passed → null (the default, pool disabled)"); assert.equal(block.enabled, true); assert.equal(block.entrypointMode, "cli"); assert.equal(block.lastEntrypoint, "cli");