mirror of
https://github.com/dtzp555-max/ocp.git
synced 2026-07-21 21:15:09 +00:00
feat(tui): warm pane pool — single-use pre-booted panes, opt-in via OCP_TUI_POOL_SIZE (−41%) (#158)
* feat(tui): warm pane pool — single-use pre-booted panes, opt-in via OCP_TUI_POOL_SIZE Backlog item #3 of docs/plans/2026-07-13-tui-latency/README.md. Every TUI request currently cold-boots a tmux+claude pane. This adds an OPT-IN pool of pre-booted panes. Recorded as ADR 0008 (docs/adr/0008-tui-warm-pane-pool.md), which extends ADR 0007. MEASURED (this host, Sonnet 4.6, --effort low, through a real OCP instance; a sample counts only if HTTP 200 AND the body carries the demanded marker): pool off (main code) n= 6 p50 10.17s [9164 9499 9760 10572 10774 11281] pool on, warm hits n=12 p50 6.00s [5286 5289 5520 5584 5621 5969 6040 6098 6280 7846 8036 11053] pool on, warm hits (post- n= 6 p50 5.62s [4729 4753 5236 6004 7548 9548] review-fix re-run) -> -4.17s / -41%. 12 hits / 1 miss / 0 bootFailures over 13 requests (and 6/1/0 on the post-fix re-run). Robust to counting the miss: n=13 p50 -> -40.6%. The plan doc predicted only -1.0s (the boot). It is ~4.2s because the cold path also pays ~2.9s INSIDE the first turn beyond claude's own reported turn_duration — post- input-bar init that an idle pane has already finished. Phase decomposition of the cold path (n=6 medians): prep 2ms | tmux spawn 27ms | boot->input-ready 1232ms | paste 8ms | paste-verify 426ms | submit->terminal 8458ms | teardown 8ms = 10162ms total, vs native turn_duration 5539ms => 4490ms of OCP-side overhead, of which the pool recovers ~1.26s of boot and ~2.9s of in-claude cold start. (The 426ms paste-verify is one 400ms poll tick; a real paste lands in ~80ms. Not addressed here — separate item.) DESIGN - SINGLE-USE panes. A pooled pane serves exactly ONE turn, then is killed and replaced in the background. Each carries its OWN fresh --session-id fixed at boot, so one session still holds one exchange. This is what keeps transcript.mjs's extractLatestAssistantText correct; its warning about a future warm pool reusing a session is answered in-place (comment updated) and left standing for anyone who later wants a second turn on a pane — that would be a cross-request TEXT LEAK and needs user-line scoping in the transcript reader first. - Pool keyed by model; --model is fixed at spawn. A miss falls back to the cold path with zero behaviour change. The pool warms the most recently requested model, so the first request after start (and after a model switch) is always a cold miss. - REAPER COEXISTENCE (the crux). An idle warm pane IS ours, and the periodic sweep runs precisely when we are idle. reapStaleTuiSessions() takes a `spare` set of EXACT live session names, and server.mjs DRAINS the pool immediately before the sweep: 1. a live pooled pane is never reaped — INCLUDING one still BOOTING (see below); 2. an orphaned pooled pane IS still reaped — membership is by exact name from a live in-memory registry, never by name shape, so a pane from a dead process generation has nothing claiming it. Omitting `spare` reaps MORE, never less (fail-safe); 3. kill-server is suppressed while any pane is spared — hence the drain, so the sweep still flushes <defunct> claude zombies (the only mechanism that can). - THE POOL TRACKS ITS IN-FLIGHT BOOT BY NAME, NOT AS A COUNT. bootTuiPane creates the tmux session SYNCHRONOUSLY and only then waits up to POOL_BOOT_MS (20s) for the input bar, so a pooled session can be LIVE for ~20s before its boot resolves. Tracking boots as a count meant the pool could not name that session, which caused two real bugs (found in review, reproduced, fixed, and now regression-tested): * the reap sweep KILLED the booting pane (it could not be spared), left the pool empty with nothing scheduled, and logged the exact tui_pool_boot_failed WARN operators are told to alert on — for a completely healthy drain; * graceful shutdown ORPHANED a live authenticated idle `claude`: gracefulShutdown calls process.exit(0) in the SAME TICK as the drain (TUI panes are tmux children, so activeProcesses is empty and the wait-for-children path exits immediately), so cleanup deferred to a .then() never ran. Fix: the pool mints each pane's identity up front ({sessionId, name}) and holds it in _bootingPane. liveNames() includes it; drain() kills it SYNCHRONOUSLY. A generation counter distinguishes "cancelled by us" from "genuinely failed", so a drain never inflates bootFailures and resume() reliably starts a fresh boot. Deriving the name from the session-id also makes `tmux ls` correlate to the transcript file. - SLOT ACCOUNTING. Refill boots take NO TuiSemaphore slot (those bound real turns and would be starved); they cannot leak one either, since they never hold one. Refills are SERIALIZED, one boot at a time — live at size=2, two cold boots racing an in-flight turn overran the readiness cap and a refill was discarded. A genuinely failed boot does not re-kick the chain (backoff; a broken claude must not respawn forever). Background boots get a more generous readiness cap (POOL_BOOT_MS = 5x BOOT_MS): BOOT_MS is tight because a client is blocked on it, which is not true of a pre-boot. - BOUNDED COST. A warm pane is a LIVE idle claude process held whether or not a request arrives. Peak processes = pool size + OCP_TUI_MAX_CONCURRENT + 1 booting replacement. Size clamped to POOL_MAX_SIZE=4; garbage values disable rather than guess. Panes have a 10-min TTL and a health check at hand-out (dead/degraded pane => miss, never a hang). Missing collaborators throw at CONSTRUCTION, not on a live request (refill() is called synchronously from the request path). DEFAULT OFF (OCP_TUI_POOL_SIZE=0). This is a stable production path and the pool holds standing processes, so the operator opts in. With the pool off, runTuiTurn takes the IDENTICAL code path as before (the `pool ? pool.acquire() : null` branch yields null, and tuiPool is null so no observer is attached and no new log line is emitted) — that is what establishes the default path is unchanged. A pool-off control run (n=6, p50 9.40s) is consistent with the 10.17s baseline but had 2/6 samples >12s, so it is corroboration, NOT proof: n=6 cannot establish "unregressed" on its own. The code-path equivalence can. BANNER: NO SPAWN ARGUMENT CHANGED. buildTuiCmd is byte-identical to main (verified by extracting the function body from both revisions and comparing). Live banner captured from two real POOLED panes anyway: "Sonnet 4.6 with low effort · Claude Max" — the subscription pool, never "API Usage Billing". /health: `tui.pool` added (null when off), incl. `cancelled` (boots WE killed — not a fault; do not alert on it). The tui block is ADR-0007-owned and post-dates ADR 0006's v3.16.4 grandfather snapshot; the addition is purely additive — every pre-existing key keeps a byte-identical value. Authorization recorded in ADR 0008. ALIGNMENT: Class B / ADR 0007 + ADR 0008 (OCP-owned TUI spawn machinery). cli.js does NOT perform this operation — there is no cli.js citation and none is required: this is not an Anthropic API surface, it is OCP's own process management around the claude CLI, exactly as the existing tmux session lifecycle and reaper already are (ALIGNMENT.md Rule 2). TESTS: 294 passed / 0 failed (was 267). +27 covering acquire/hit/miss, single-use (a pane is never handed out twice), bounded + serialized refill, TTL + health-check drops, model retarget, drain/resume, boot-failure backoff, identity linkage, all three reaper invariants incl. post-drain kill-server restoration, and — the coverage gap that let both bugs ship — FIVE mid-boot tests: the booting pane is nameable/spareable, the sweep's drain kills it and resume starts a fresh boot with no bogus WARN, shutdown kills it synchronously (asserted WITHOUT awaiting, since process.exit runs in the same tick), a stale settle cannot clear a newer boot's slot, and a model switch cancels an in-flight boot for the old model. Live verification (temporary 20s reap interval, reverted): sweep drained both panes -> reaped -> refilled with NEW panes; a foreign tmux session survived untouched; with no foreign session kill-server fired and the pool still recovered and served the next request. Both review bugs reproduced against a PRIVATE tmux server (-L pr3repro, so the reaper's internal kill-server could not touch the host) before and after the fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(tui): kill a cancelled boot's pane when it settles + make async tests actually count Folds in the independent review's remaining nit — and, in proving the nit's fix, uncovers two defects in the test suite itself. ## The nit (latent M1b, second costume) `_cancelBooting` kills BY NAME, but the tmux session only EXISTS once `bootPane` has run — and `bootPane` is queued on a microtask. So a caller doing `refill()` then `drain()` in the SAME synchronous block leaves `_cancelBooting` with nothing to kill (a no-op); it bumps the generation, and the boot microtask then CREATES the session, succeeds, and — under the old bare `return` on a stale generation — walked away from a LIVE authenticated `claude` that nothing owns. Reproduced: reverted: drain() kills nothing (no session yet) -> boot creates it -> ORPHAN: ['p1'] fixed : drain() kills nothing (no session yet) -> boot creates it -> boot kills it -> [] Not reachable from any current call site, so this is defense-in-depth — but ADR 0008 and the reap-tick comment in server.mjs BOTH explicitly contemplate a boot-time pre-warm, which is exactly the shape that reaches it. Killing an already-dead session is a harmless no-op, so the fix is idempotent whichever way the race lands. ## Defect 1 in the suite: async tests were never awaited (44 of them) Writing the regression guard exposed this. `test()` called `fn()`, got a promise back, and IMMEDIATELY printed ✓ and incremented `passed` — without awaiting it. For all 44 tests written as `test("...", async () => {...})`: - ✓ meant "did not throw SYNCHRONOUSLY", not "passed"; - a failed assertion escaped as an unhandled rejection, crashing the process (CI stays red on the non-zero exit) but never being COUNTED — so the summary could print "0 failed" and be wrong. The suite's headline number was therefore not evidence for ANY async test, including this PR's own M1a/M1b guards. `test()` now settles an async body before counting it, and the summary awaits them. ## Defect 2, exposed the instant defect 1 was fixed: a false guard `"a boot that resolves AFTER a drain kills its own pane ... no orphan process left behind"` asserted `killed.length === 1` — i.e. that kill was CALLED once. But `_cancelBooting`'s kill-by-name on a not-yet-existent session is a NO-OP that still increments that counter. So "kill was called once" and "a live session is orphaned" were both true at the same time: a test named for the absence of an orphan was passing while the orphan was present. Now asserts LIVENESS (`live.size === 0`) — the only honest question. ## Evidence fix present : 295 passed, 0 failed, exit 0 fix reverted: 293 passed, 2 failed <- BOTH liveness guards fire (the old kill-count guard did not) Also: `dropped`'s doc comment now lists `cancelled` (a cancelled in-flight boot lands there via _drop). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VqgWJcjxrjjL9L9SkpZyXR --------- Co-authored-by: dtzp555 <dtzp555@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -960,6 +960,7 @@ See [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) and ADR 0007 PR-
|
|||||||
| `OCP_TUI_ENTRYPOINT` | `cli` | (TUI-mode) Billing-classifier labeling: `cli` (default) pins `cc_entrypoint=cli` deterministically; `auto` lets claude self-classify via TTY detection; `off` leaves the inherited env untouched. Honest only when the spawn is a genuine interactive PTY — see ADR 0007. |
|
| `OCP_TUI_ENTRYPOINT` | `cli` | (TUI-mode) Billing-classifier labeling: `cli` (default) pins `cc_entrypoint=cli` deterministically; `auto` lets claude self-classify via TTY detection; `off` leaves the inherited env untouched. Honest only when the spawn is a genuine interactive PTY — see ADR 0007. |
|
||||||
| `OCP_TUI_EFFORT` | `low` | (TUI-mode) Effort level passed to the interactive `claude` as an explicit `--effort` flag: `low` (default), `medium`, `high`, `xhigh`, `max`, or `inherit` to omit the flag (the pre-flag behaviour: the pane inherits a HOME-dependent effort — the operator's `~/.claude/settings.json` `effortLevel` in real-home mode, claude's built-in default in env-token scratch mode). Explicit `low` cuts measured TTFT p50 by ~40% and collapses run-to-run variance ~15× versus an inherited `xhigh` (see `docs/plans/2026-07-13-tui-latency/`); proxied requests rarely benefit from extended thinking. Banner-verified to stay on the subscription pool (`· Claude Max`). An invalid value logs a warning and falls back to `low`. |
|
| `OCP_TUI_EFFORT` | `low` | (TUI-mode) Effort level passed to the interactive `claude` as an explicit `--effort` flag: `low` (default), `medium`, `high`, `xhigh`, `max`, or `inherit` to omit the flag (the pre-flag behaviour: the pane inherits a HOME-dependent effort — the operator's `~/.claude/settings.json` `effortLevel` in real-home mode, claude's built-in default in env-token scratch mode). Explicit `low` cuts measured TTFT p50 by ~40% and collapses run-to-run variance ~15× versus an inherited `xhigh` (see `docs/plans/2026-07-13-tui-latency/`); proxied requests rarely benefit from extended thinking. Banner-verified to stay on the subscription pool (`· Claude Max`). An invalid value logs a warning and falls back to `low`. |
|
||||||
| `OCP_TUI_MAX_CONCURRENT` | `2` | (TUI-mode) Max concurrent interactive TUI turns. **Independent** of `CLAUDE_MAX_CONCURRENT` (which bounds the `-p`/stream-json path; TUI never uses it). A TUI turn is heavy (per-request cold-boot of tmux+claude + up to `CLAUDE_TUI_WALLCLOCK_MS` wallclock), so the default is low to keep small hosts (e.g. a Pi 4) alive under a burst. Excess turns **queue** (bounded); a full queue yields a 503. See ADR 0007 PR-B amendment. |
|
| `OCP_TUI_MAX_CONCURRENT` | `2` | (TUI-mode) Max concurrent interactive TUI turns. **Independent** of `CLAUDE_MAX_CONCURRENT` (which bounds the `-p`/stream-json path; TUI never uses it). A TUI turn is heavy (per-request cold-boot of tmux+claude + up to `CLAUDE_TUI_WALLCLOCK_MS` wallclock), so the default is low to keep small hosts (e.g. a Pi 4) alive under a burst. Excess turns **queue** (bounded); a full queue yields a 503. See ADR 0007 PR-B amendment. |
|
||||||
|
| `OCP_TUI_POOL_SIZE` | `0` (off) | (TUI-mode) Number of **pre-booted warm `claude` panes** kept ready, so a request does not pay the cold boot. `0` disables the pool entirely — the request path is then exactly the cold-boot path. Max `4`; an unparseable value disables it rather than guessing. **Measured on a Mac mini (Sonnet 4.6, `--effort low`): end-to-end p50 `10.17s` (n=6, pool off) → `6.00s` (n=12 warm hits) — −4.2 s / −41%** — the pool recovers both the ~1.2 s boot *and* ~2.9 s of post-input-bar init that a pane which has been idle a moment has already finished. **Cost:** each warm pane is a *live idle `claude` process* held whether or not a request ever arrives (peak processes ≈ pool size + `OCP_TUI_MAX_CONCURRENT` + 1 booting replacement) — which is why it is opt-in. Panes are **single-use**: one turn, then killed and replaced in the background. The **first request after start (and after any model switch) is always a cold miss** — the pool warms the most recently requested model, since OCP cannot know which model the next caller wants. See `docs/plans/2026-07-13-tui-latency/`. |
|
||||||
| `OCP_SKIP_AUTH_TEST` | *(unset)* | When `=1`, skip the `claude -p` auth probe during `setup.mjs`. After 2026-06-15 this probe draws from the Agent SDK credit pool; set this to avoid burning a metered credit on re-installs or `ocp update` runs. Auth is validated at the first real request. |
|
| `OCP_SKIP_AUTH_TEST` | *(unset)* | When `=1`, skip the `claude -p` auth probe during `setup.mjs`. After 2026-06-15 this probe draws from the Agent SDK credit pool; set this to avoid burning a metered credit on re-installs or `ocp update` runs. Auth is validated at the first real request. |
|
||||||
| `OCP_TUI_FULL_TOOLS` | *(unset)* | (TUI-mode, **single-user only**) When `=1`, grant the interactive session the **same tool surface as the `-p` path** — `--allowedTools` (+ optional `--mcp-config`, read from `CLAUDE_ALLOWED_TOOLS` / `CLAUDE_MCP_CONFIG`) — instead of the default MCP-walled, built-in-tools-only set. Lets a trusted single-operator TUI deployment run a **tool-using / MCP agent** (e.g. an OpenClaw assistant) on the subscription pool. Safe because TUI **refuses to boot under `AUTH_MODE=multi`** (hard exit) — no guest key can ever reach the TUI path, so this gate cannot expose tools to an untrusted caller. (Under `AUTH_MODE=shared` + `OCP_TUI_ALLOW_LAN=1`, anyone holding the single shared key reaches it — that is the existing TUI trust model, unchanged.) Note: `--dangerously-skip-permissions` / `CLAUDE_SKIP_PERMISSIONS` is **not** supported for TUI — claude v2.1.x shows an interactive bypass-acceptance screen in headless tmux that cannot be answered, bricking the pane. Use scratch-home `settings.json` `additionalDirectories` instead. See [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) and ADR 0007. |
|
| `OCP_TUI_FULL_TOOLS` | *(unset)* | (TUI-mode, **single-user only**) When `=1`, grant the interactive session the **same tool surface as the `-p` path** — `--allowedTools` (+ optional `--mcp-config`, read from `CLAUDE_ALLOWED_TOOLS` / `CLAUDE_MCP_CONFIG`) — instead of the default MCP-walled, built-in-tools-only set. Lets a trusted single-operator TUI deployment run a **tool-using / MCP agent** (e.g. an OpenClaw assistant) on the subscription pool. Safe because TUI **refuses to boot under `AUTH_MODE=multi`** (hard exit) — no guest key can ever reach the TUI path, so this gate cannot expose tools to an untrusted caller. (Under `AUTH_MODE=shared` + `OCP_TUI_ALLOW_LAN=1`, anyone holding the single shared key reaches it — that is the existing TUI trust model, unchanged.) Note: `--dangerously-skip-permissions` / `CLAUDE_SKIP_PERMISSIONS` is **not** supported for TUI — claude v2.1.x shows an interactive bypass-acceptance screen in headless tmux that cannot be answered, bricking the pane. Use scratch-home `settings.json` `additionalDirectories` instead. See [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) and ADR 0007. |
|
||||||
|
|
||||||
@@ -1040,6 +1041,7 @@ Then restart OCP. At boot you will see (with the env token set, isolated home au
|
|||||||
- **Stale tmux sessions are reaped.** The pane's `claude` is a child of the tmux server (not OCP), so OCP cannot reap it directly; `claude` zombies can otherwise accumulate as `<defunct>` over a long-running host. OCP reaps them at boot and on a 15-min idle sweep by issuing `tmux kill-server` — but **only when no foreign tmux session remains** (it never disrupts a co-hosted `olp-tui-*` instance). See ADR 0007 PR-C amendment.
|
- **Stale tmux sessions are reaped.** The pane's `claude` is a child of the tmux server (not OCP), so OCP cannot reap it directly; `claude` zombies can otherwise accumulate as `<defunct>` over a long-running host. OCP reaps them at boot and on a 15-min idle sweep by issuing `tmux kill-server` — but **only when no foreign tmux session remains** (it never disrupts a co-hosted `olp-tui-*` instance). See ADR 0007 PR-C amendment.
|
||||||
- **Default path unchanged.** Unset `CLAUDE_TUI_MODE` and restart → `callClaude` / `callClaudeStreaming` are used again, byte-for-byte identical to today.
|
- **Default path unchanged.** Unset `CLAUDE_TUI_MODE` and restart → `callClaude` / `callClaudeStreaming` are used again, byte-for-byte identical to today.
|
||||||
- **Concurrency is bounded separately.** TUI turns are heavy (per-request cold-boot + long wallclock), so the TUI path has its own limiter — `OCP_TUI_MAX_CONCURRENT` (default `2`), independent of `CLAUDE_MAX_CONCURRENT`. Excess turns queue; a full queue returns a 503. Tune it up only on a host that can run more interactive `claude` sessions at once.
|
- **Concurrency is bounded separately.** TUI turns are heavy (per-request cold-boot + long wallclock), so the TUI path has its own limiter — `OCP_TUI_MAX_CONCURRENT` (default `2`), independent of `CLAUDE_MAX_CONCURRENT`. Excess turns queue; a full queue returns a 503. Tune it up only on a host that can run more interactive `claude` sessions at once.
|
||||||
|
- **Optional warm pane pool (`OCP_TUI_POOL_SIZE`, default off).** Pre-boots panes so a request skips the cold boot — measured p50 `10.17s` → `6.00s` (−41%). Pooled panes are **single-use** (one turn, then killed and replaced in the background), each carrying its own fresh `--session-id`, so one session still means one exchange and no earlier-turn text can leak into a later answer. They are named `ocp-tui-<port>-p<hex>` and coexist with the reaper by design: the sweep **drains the pool first**, then reaps (so `kill-server` still flushes `<defunct>` zombies), then the pool refills in the background. Drain→reap→resume is synchronous, so no request can land mid-sweep; a request arriving while the pool is still re-booting simply misses it and cold-boots. A live pooled pane is never reaped — **including one that is still booting**, whose tmux session already exists — while an *orphaned* one (left by a previous process generation) still is.
|
||||||
|
|
||||||
### ⚠️ Latency: TUI mode has a ~6-second floor, and it is immovable
|
### ⚠️ Latency: TUI mode has a ~6-second floor, and it is immovable
|
||||||
|
|
||||||
@@ -1082,12 +1084,26 @@ a sub-5-second budget. Full measurements and methodology:
|
|||||||
"entrypointMismatches": 0, // count of cli-expected-but-got-other turns — ALERT if this climbs
|
"entrypointMismatches": 0, // count of cli-expected-but-got-other turns — ALERT if this climbs
|
||||||
"inflight": 1, // TUI turns running right now
|
"inflight": 1, // TUI turns running right now
|
||||||
"queued": 0, // TUI turns waiting for a concurrency slot
|
"queued": 0, // TUI turns waiting for a concurrency slot
|
||||||
"maxConcurrent": 2 // OCP_TUI_MAX_CONCURRENT
|
"maxConcurrent": 2, // OCP_TUI_MAX_CONCURRENT
|
||||||
|
"pool": { // warm pane pool — null when OCP_TUI_POOL_SIZE=0 (the default)
|
||||||
|
"size": 2, // target warm panes (OCP_TUI_POOL_SIZE)
|
||||||
|
"warm": 2, // panes ready right now — each is a LIVE idle claude process
|
||||||
|
"booting": 0, // replacement panes currently pre-booting
|
||||||
|
"model": "claude-sonnet-4-6", // the model being warmed (the most recently requested one)
|
||||||
|
"hits": 12, // requests served by a warm pane
|
||||||
|
"misses": 1, // requests that fell back to the cold boot (the 1st is always one)
|
||||||
|
"boots": 14, // panes successfully pre-booted
|
||||||
|
"bootFailures": 0, // pre-boots that genuinely never reached the input bar — WATCH this
|
||||||
|
"cancelled": 4, // in-flight boots OCP killed on purpose (drain / model switch) — not faults
|
||||||
|
"dropped": 8 // panes discarded unused (drain sweep / expired / unhealthy)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
Alert on `entrypointMismatches > 0` (or `lastEntrypoint !== "cli"`): it means a turn drew from the metered Agent SDK pool instead of the subscription. `inflight` / `queued` show how close the TUI path is to its concurrency cap.
|
Alert on `entrypointMismatches > 0` (or `lastEntrypoint !== "cli"`): it means a turn drew from the metered Agent SDK pool instead of the subscription. `inflight` / `queued` show how close the TUI path is to its concurrency cap.
|
||||||
|
|
||||||
|
With the pool on, `hits` / `misses` is the hit rate (a steady single-model consumer should sit near 100% after the first request), and `warm` is your standing idle-process cost. A climbing `bootFailures` means panes are not reaching their input bar — the pool then degrades safely to the cold path, but latency reverts to the un-pooled numbers. `cancelled` counts boots OCP killed *on purpose* (a drain, a model switch) and is **not** a fault signal — do not alert on it. A steadily climbing `dropped` is likewise normal: the 15-min reap sweep drains and re-boots the pool on every tick so `kill-server` can still flush `<defunct>` zombies.
|
||||||
|
|
||||||
### Kill-switch
|
### Kill-switch
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -0,0 +1,208 @@
|
|||||||
|
# ADR 0008 — TUI Warm Pane Pool
|
||||||
|
|
||||||
|
**Date:** 2026-07-13
|
||||||
|
**Status:** Proposed
|
||||||
|
**Extends:** [ADR 0007](0007-tui-interactive-mode.md) (TUI interactive mode). This ADR does not
|
||||||
|
change ADR 0007's billing-pool argument, security posture, or kill-switch — it adds a latency
|
||||||
|
optimization *inside* the TUI spawn machinery ADR 0007 owns.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
TUI mode (ADR 0007) serves every request by cold-booting a fresh `tmux` session running an
|
||||||
|
interactive `claude`, submitting one prompt, reading the native transcript, and killing the
|
||||||
|
session. That cold boot is paid on **every** request.
|
||||||
|
|
||||||
|
[`docs/plans/2026-07-13-tui-latency/`](../plans/2026-07-13-tui-latency/README.md) measured the
|
||||||
|
TUI path and listed a warm pane pool as backlog item #3, costed at "**~1.0 s**" (the observed
|
||||||
|
boot-to-input-bar time). Instrumenting the real request path showed that estimate is **~4×
|
||||||
|
too low**. Phase decomposition of the cold path (n=6 medians, Sonnet 4.6, `--effort low`,
|
||||||
|
through a real OCP instance):
|
||||||
|
|
||||||
|
| Phase | Median |
|
||||||
|
|---|---|
|
||||||
|
| prep (trust cwd, write prompt file) | 2 ms |
|
||||||
|
| `tmux new-session` | 27 ms |
|
||||||
|
| **boot → input bar ready** | **1232 ms** |
|
||||||
|
| paste (`load-buffer` + `paste-buffer`) | 8 ms |
|
||||||
|
| paste-verify poll | 426 ms |
|
||||||
|
| **submit → transcript terminal** | **8458 ms** |
|
||||||
|
| teardown | 8 ms |
|
||||||
|
| **total** | **10162 ms** |
|
||||||
|
| *claude's own reported `turn_duration`* | *5539 ms* |
|
||||||
|
| **OCP-side overhead** | **4490 ms** |
|
||||||
|
|
||||||
|
The `submit → terminal` phase exceeds claude's own `turn_duration` by **~2.9 s**. That gap is
|
||||||
|
**post-input-bar initialization inside `claude`** — work that a pane which has merely *sat idle
|
||||||
|
for a few seconds* has already completed. A direct spike confirmed it: an identical pane, idle
|
||||||
|
12 s before receiving the same prompt, completed its turn in a median 5537 ms versus 7980 ms
|
||||||
|
cold.
|
||||||
|
|
||||||
|
So a warm pane recovers **~1.26 s of boot *and* ~2.9 s of in-`claude` cold start** — not the
|
||||||
|
~1.0 s the plan predicted.
|
||||||
|
|
||||||
|
The reason this was worth a pool rather than a "keep one session and reuse it" cache is a
|
||||||
|
hazard already flagged in the code. `lib/tui/transcript.mjs` returns the **last text-bearing
|
||||||
|
assistant entry in the whole transcript file**, which is correct *only* under OCP's
|
||||||
|
one-session-per-request model, and it says so:
|
||||||
|
|
||||||
|
> *"If a future warm-pool ever reuses a session WITHOUT a fresh session-id / clear, earlier-turn
|
||||||
|
> text could leak — that author must add user-line scoping here."*
|
||||||
|
|
||||||
|
Reusing a pane for a second turn puts two exchanges in one transcript and would leak the earlier
|
||||||
|
turn's text into the later turn's answer — a **cross-request data leak**, not merely a bug.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Add an **opt-in pool of pre-booted, single-use `claude` panes**, `OCP_TUI_POOL_SIZE` (default
|
||||||
|
`0` = off, max `4`). Implementation: `lib/tui/pool.mjs`.
|
||||||
|
|
||||||
|
### 1. Panes are SINGLE-USE. This is the load-bearing rule.
|
||||||
|
|
||||||
|
A pooled pane serves **exactly one turn**, then is killed and replaced in the background. Each
|
||||||
|
pane is booted with its **own fresh `--session-id`**, fixed at spawn, and the turn locates its
|
||||||
|
transcript by that id.
|
||||||
|
|
||||||
|
This preserves one-session-per-request exactly, so the `transcript.mjs` hazard above **does not
|
||||||
|
arise** and no user-line scoping was needed. The warning in `transcript.mjs` is deliberately
|
||||||
|
left standing, now annotated: it still binds anyone who later wants a pane to serve a second
|
||||||
|
turn, or to reset a session with `/clear` and reuse it. **Neither is permitted without first
|
||||||
|
adding user-line scoping to the transcript reader.**
|
||||||
|
|
||||||
|
Rejected alternative — *reuse a pane for N turns, `/clear` between* — is strictly cheaper
|
||||||
|
(no re-boot per request) and was rejected on exactly this basis. The latency win is not worth a
|
||||||
|
cross-request text-leak surface guarded only by a `/clear` that we cannot verify landed.
|
||||||
|
|
||||||
|
### 2. The pool is keyed by model, and a MISS is always safe.
|
||||||
|
|
||||||
|
`--model` is fixed at spawn, so a pane can only serve the model it booted with. A pool miss
|
||||||
|
falls back to the existing cold-boot path with **zero behavioural difference**. There is no
|
||||||
|
boot-time pre-warm and no configured model: OCP cannot know which model the next caller wants,
|
||||||
|
so the pool warms the **most recently requested** model. Consequence, stated plainly: **the
|
||||||
|
first request after start, and the first after any model switch, is always a cold miss.**
|
||||||
|
|
||||||
|
### 3. The pool and the session reaper coexist by an explicit invariant.
|
||||||
|
|
||||||
|
This is the subtle part. `reapStaleTuiSessions()` kills every session matching this instance's
|
||||||
|
`ocp-tui-<port>-` prefix, and issues `tmux kill-server` when no foreign session remains (the
|
||||||
|
only mechanism that can reap `<defunct>` `claude` zombies — the pane's `claude` is a child of
|
||||||
|
the tmux *server*, not of node). A warm pooled pane **is** one of our own sessions, alive and
|
||||||
|
idle **by design** — and the periodic sweep runs precisely **when the instance is idle**, i.e.
|
||||||
|
exactly when the pool is full.
|
||||||
|
|
||||||
|
The invariant, stated in a comment above `reapStaleTuiSessions` and pinned by tests:
|
||||||
|
|
||||||
|
1. **A live pooled pane is never reaped — including one that is still BOOTING.** The reaper
|
||||||
|
takes a `spare` set of **exact session names** supplied by the pool's live registry.
|
||||||
|
2. **An orphaned pooled pane IS still reaped.** Membership is by **exact name from a live
|
||||||
|
in-memory registry, never by name shape**. A pane the pool no longer owns — handed out,
|
||||||
|
dropped, cancelled, or left behind by a previous process generation (whose registry died with
|
||||||
|
it) — is absent from `spare` and is killed like any other stale session. **Fail-safe:
|
||||||
|
omitting `spare` reaps *more*, never less.** Pool panes are named `ocp-tui-<port>-p<hex>`
|
||||||
|
purely for operator legibility; that shape is *not* the exemption mechanism.
|
||||||
|
3. **`kill-server` is suppressed while any pane is spared** (it would kill a live child of the
|
||||||
|
tmux server). Therefore **the pool is DRAINED immediately before every sweep**, so `spare` is
|
||||||
|
empty on the normal tick and `kill-server` still fires. Without the drain, a permanently-full
|
||||||
|
pool would **permanently disable zombie reaping** — the pool would silently break the thing
|
||||||
|
the sweep exists to do. The drain costs one pane re-boot per tick (15 min).
|
||||||
|
|
||||||
|
The `spare` mechanism is belt-and-braces given the drain: it makes it impossible for a reap call
|
||||||
|
site that *forgets* to drain to kill a live pane.
|
||||||
|
|
||||||
|
### 4. The pool tracks its in-flight boot BY NAME, not as a count.
|
||||||
|
|
||||||
|
`bootTuiPane` creates the tmux session **synchronously** and only *then* waits (up to
|
||||||
|
`POOL_BOOT_MS`, 20 s) for the input bar. So **a pooled tmux session can be live for ~20 s before
|
||||||
|
its boot resolves.** A pool that tracked in-flight boots as a *count* could not name that
|
||||||
|
session, and this produced two real bugs (both caught in review, both now regression-tested):
|
||||||
|
|
||||||
|
- the periodic sweep **killed the booting pane** (it could not be spared), then left the pool
|
||||||
|
empty with nothing scheduled, and logged the exact `tui_pool_boot_failed` warning operators are
|
||||||
|
told to alert on — for a completely healthy drain;
|
||||||
|
- graceful shutdown **orphaned a live, authenticated, idle `claude`**: `gracefulShutdown` calls
|
||||||
|
`process.exit(0)` in the same tick as the drain (TUI panes are tmux children, so node's
|
||||||
|
`activeProcesses` set is empty and the "wait for children" path exits immediately), so any
|
||||||
|
cleanup deferred to a `.then()` never ran.
|
||||||
|
|
||||||
|
The pool therefore **mints each pane's identity up front** (`{sessionId, name}`, name derived
|
||||||
|
from the session-id so `tmux ls` correlates to the transcript file) and holds it in
|
||||||
|
`_bootingPane`. `liveNames()` includes it; `drain()` kills it **synchronously**. A generation
|
||||||
|
counter distinguishes *"cancelled by us"* from *"genuinely failed"*, so a drain never inflates
|
||||||
|
`bootFailures` and `resume()` reliably starts a fresh boot.
|
||||||
|
|
||||||
|
### 5. Refills take no concurrency slot, and are serialized.
|
||||||
|
|
||||||
|
A refill boot deliberately does **not** take a `TuiSemaphore` slot: those slots bound concurrent
|
||||||
|
*turns* and belong to real requests, and charging a background pre-boot against them would let
|
||||||
|
the pool starve the traffic it exists to speed up. It cannot leak a slot either, since it never
|
||||||
|
holds one. Boots are **serialized** (one at a time): two cold boots racing an in-flight turn were
|
||||||
|
observed to overrun even the generous pool readiness cap. A genuinely failed boot does **not**
|
||||||
|
re-kick the chain (backoff — a broken `claude` must not respawn forever).
|
||||||
|
|
||||||
|
Background boots get a more generous readiness cap (`POOL_BOOT_MS` = 5 × `BOOT_MS`): `BOOT_MS` is
|
||||||
|
tight because a *client* is blocked on it, which is not true of a pre-boot. Slow ≠ broken.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
### Cost — standing processes, paid whether or not a request arrives
|
||||||
|
|
||||||
|
**A warm pane is a live idle `claude` process.** Peak process count is
|
||||||
|
`OCP_TUI_POOL_SIZE` + `OCP_TUI_MAX_CONCURRENT` + 1 (booting replacement). This is the whole
|
||||||
|
reason the pool is **default-off**: an operator must opt into holding processes for traffic that
|
||||||
|
may never come. Size is clamped to `POOL_MAX_SIZE` = 4; an unparseable value **disables** the
|
||||||
|
pool rather than guessing.
|
||||||
|
|
||||||
|
Panes carry a 10-minute TTL and are health-checked at hand-out; a dead or degraded pane becomes
|
||||||
|
a **miss** (cold path), never a hung turn.
|
||||||
|
|
||||||
|
### Benefit
|
||||||
|
|
||||||
|
Measured end-to-end through a real OCP instance (Sonnet 4.6, `--effort low`):
|
||||||
|
**p50 10.17 s (n=6, pool off) → 6.00 s (n=12 warm hits) — −4.2 s / −41%.**
|
||||||
|
|
||||||
|
### The floor is unchanged
|
||||||
|
|
||||||
|
The pool does not touch the **~6 s TTFT floor** documented in the latency plan (claude always
|
||||||
|
prefills the full Claude Code system prompt). TUI mode remains unsuitable for interactive /
|
||||||
|
real-time consumers; it is for batch and background work. This ADR does not change that
|
||||||
|
conclusion.
|
||||||
|
|
||||||
|
### Observability
|
||||||
|
|
||||||
|
`/health`'s `tui` block gains a `pool` sub-object (`null` when off): `size`, `warm`, `booting`,
|
||||||
|
`model`, `hits`, `misses`, `boots`, `bootFailures`, `cancelled`, `dropped`. A climbing
|
||||||
|
`bootFailures` means panes are not reaching their input bar — the pool then degrades safely to
|
||||||
|
the cold path, but latency reverts to the un-pooled numbers. A steadily climbing `dropped` is
|
||||||
|
**normal** (the 15-min sweep drains and re-boots the pool on every tick, by design — see
|
||||||
|
Decision 3).
|
||||||
|
|
||||||
|
### ALIGNMENT authorization
|
||||||
|
|
||||||
|
- **Class B / OCP-owned.** The warm pool is process management around the `claude` CLI — the
|
||||||
|
same category as the existing tmux session lifecycle and the defunct-session reaper it extends.
|
||||||
|
**`cli.js` does not perform this operation, and no `cli.js` citation applies**; the authority
|
||||||
|
is ADR 0007 (which owns the TUI spawn machinery) plus this ADR. This is `ALIGNMENT.md` Rule 2's
|
||||||
|
Class B citation requirement, discharged explicitly rather than by silence.
|
||||||
|
- **The `/health` extension** adds sub-fields to the `tui` block. That block is **owned by ADR
|
||||||
|
0007** and post-dates ADR 0006's v3.16.4 grandfather snapshot, so it is not part of the frozen
|
||||||
|
B.2 inventory. The change is additive — every pre-existing `/health` field keeps a
|
||||||
|
byte-identical value, and `pool` is `null` unless the operator opts in — which is the
|
||||||
|
behaviour-preserving bar ADR 0006 sets. This ADR records that authorization.
|
||||||
|
- **No spawn argument changed.** `buildTuiCmd` is byte-identical; the pool calls it with the same
|
||||||
|
arguments. Banner-verified on live pooled panes: `· Claude Max`, never `API Usage Billing`
|
||||||
|
(the `--bare` trap documented in the latency plan).
|
||||||
|
|
||||||
|
### What a future contributor must not undo
|
||||||
|
|
||||||
|
- **Do not let a pane serve a second turn** (or `/clear`-and-reuse one) without first adding
|
||||||
|
user-line scoping to `lib/tui/transcript.mjs`. That is a cross-request text leak, not a perf
|
||||||
|
tweak. See Decision 1.
|
||||||
|
- **Do not remove the drain-before-sweep.** It is what keeps `kill-server` zombie reaping alive.
|
||||||
|
See Decision 3.
|
||||||
|
- **Do not go back to counting in-flight boots.** The pool must be able to *name* a session that
|
||||||
|
exists but has not finished booting. See Decision 4.
|
||||||
@@ -23,6 +23,8 @@ New ADRs increment from the highest existing number. Filenames are
|
|||||||
| [0004](0004-openclaw-auto-sync.md) | OpenClaw Auto-Sync | Why `scripts/sync-openclaw.mjs` runs on `ocp update`, what its scope boundary is (writes only `models.providers["claude-local"].models` and `agents.defaults.models["claude-local/*"]`), and the idempotency contract. |
|
| [0004](0004-openclaw-auto-sync.md) | OpenClaw Auto-Sync | Why `scripts/sync-openclaw.mjs` runs on `ocp update`, what its scope boundary is (writes only `models.providers["claude-local"].models` and `agents.defaults.models["claude-local/*"]`), and the idempotency contract. |
|
||||||
| [0005](0005-no-multi-provider.md) | No Multi-Provider | Why OCP stays single-provider (Anthropic-via-cli.js) and does not extend to OpenAI / Gemini / OpenRouter. Cost estimate: ~7 weeks for a v1 that buys neither moat nor commercial readiness. Separate commercial work starts in a separate repo. |
|
| [0005](0005-no-multi-provider.md) | No Multi-Provider | Why OCP stays single-provider (Anthropic-via-cli.js) and does not extend to OpenAI / Gemini / OpenRouter. Cost estimate: ~7 weeks for a v1 that buys neither moat nor commercial readiness. Separate commercial work starts in a separate repo. |
|
||||||
| [0006](0006-openai-shim-scope.md) | OpenAI Shim Scope | The Class A / Class B taxonomy. Class A endpoints (`cli.js`-mirror) keep Rules 1–5 verbatim; Class B endpoints (OCP-owned compatibility surface — `/v1/chat/completions`, `/v1/models`, admin endpoints) are anchored to OpenAI's spec (B.1) or to an authorizing ADR (B.2). Triggered by PR #99 (external `response_format` honoring). Grandfathers the existing B.2 inventory at v3.16.4. |
|
| [0006](0006-openai-shim-scope.md) | OpenAI Shim Scope | The Class A / Class B taxonomy. Class A endpoints (`cli.js`-mirror) keep Rules 1–5 verbatim; Class B endpoints (OCP-owned compatibility surface — `/v1/chat/completions`, `/v1/models`, admin endpoints) are anchored to OpenAI's spec (B.1) or to an authorizing ADR (B.2). Triggered by PR #99 (external `response_format` honoring). Grandfathers the existing B.2 inventory at v3.16.4. |
|
||||||
|
| [0007](0007-tui-interactive-mode.md) | TUI Interactive Mode | Why TUI-mode spawns an interactive `claude` in a tmux pane (no `-p`) to reach the **subscription** billing pool (`cc_entrypoint=cli`) rather than the metered Agent SDK pool. Owns the TUI spawn machinery: entrypoint labeling, credential-isolated home, MCP hard-disable, session namespace + defunct-session reaping, the independent concurrency bound, and the `/health` `tui` block. **Single-user only** — hard FATAL on multi-user configs. |
|
||||||
|
| [0008](0008-tui-warm-pane-pool.md) | TUI Warm Pane Pool | Why `OCP_TUI_POOL_SIZE` pre-boots **single-use** `claude` panes (one turn each, own `--session-id`) — and why reuse is forbidden (`transcript.mjs` returns the last assistant entry in the file, so a reused session leaks the earlier turn's text). Measured −41% end-to-end. Defines the pool↔reaper invariant (exemption by exact name from a live registry; drain before every sweep so `kill-server` zombie reaping survives) and the standing idle-process cost. Extends ADR 0007. |
|
||||||
|
|
||||||
## When to write a new ADR
|
## When to write a new ADR
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,309 @@
|
|||||||
|
// TUI warm pane pool (docs/plans/2026-07-13-tui-latency backlog #3).
|
||||||
|
//
|
||||||
|
// WHAT IT IS: a small set of PRE-BOOTED `claude` panes, each already sitting at its
|
||||||
|
// input bar, so a request does not pay the cold boot. Opt-in: OCP_TUI_POOL_SIZE=0
|
||||||
|
// (default) disables it entirely and the request path is byte-for-byte today's.
|
||||||
|
//
|
||||||
|
// ── SINGLE-USE IS THE LOAD-BEARING RULE ─────────────────────────────────────
|
||||||
|
// A pooled pane serves EXACTLY ONE turn and is then killed and replaced in the
|
||||||
|
// background. Each pane carries its OWN fresh `--session-id`, fixed at boot, and the
|
||||||
|
// turn locates its transcript by that id. So OCP's one-session-per-request model is
|
||||||
|
// preserved: a session's transcript still holds exactly one logical exchange.
|
||||||
|
// That is what keeps lib/tui/transcript.mjs's extractLatestAssistantText (which returns
|
||||||
|
// the LAST text-bearing assistant entry in the whole file, not "text since the matching
|
||||||
|
// user line") correct — see the scoping note there. A pane MUST NEVER serve a second
|
||||||
|
// turn, and a session MUST NEVER be reset with /clear and reused: either would put two
|
||||||
|
// exchanges in one transcript and leak the earlier turn's text into the later turn's
|
||||||
|
// answer. Nothing here reuses a pane; keep it that way.
|
||||||
|
//
|
||||||
|
// ── WHY IT'S WORTH MORE THAN THE BOOT TIME ──────────────────────────────────
|
||||||
|
// Measured on this host (n=6 through OCP, Sonnet 4.6, --effort low): the cold path
|
||||||
|
// spends ~1.23 s reaching the input bar, but ALSO ~2.9 s inside the first turn beyond
|
||||||
|
// what claude itself reports as the turn duration — post-input-bar init that a pane
|
||||||
|
// which has been idle for a few seconds has already finished. A warm pane recovers both.
|
||||||
|
//
|
||||||
|
// ── COST (bounded, and paid whether or not a request arrives) ───────────────
|
||||||
|
// Each warm pane is a LIVE `claude` process (plus its tmux pane) sitting idle. Peak
|
||||||
|
// process count is (pool size) + (OCP_TUI_MAX_CONCURRENT in-flight turns) + (panes
|
||||||
|
// currently booting as replacements). Pool size is clamped to POOL_MAX_SIZE.
|
||||||
|
//
|
||||||
|
// Pure + injectable (bootPane / killPane / paneHealthy / now) so test-features.mjs can
|
||||||
|
// assert acquire / miss / refill / TTL / reaper-exemption with no tmux and no claude.
|
||||||
|
|
||||||
|
// Hard cap on OCP_TUI_POOL_SIZE. Each pane is an idle claude process; 4 is already a
|
||||||
|
// lot of resident memory on a small host (a Pi serving a family) for zero in-flight work.
|
||||||
|
export const POOL_MAX_SIZE = 4;
|
||||||
|
|
||||||
|
// A warm pane older than this is dropped on acquire rather than handed out. The periodic
|
||||||
|
// reap tick (server.mjs) drains the pool every 15 min anyway, so this only bites when
|
||||||
|
// that tick kept getting skipped because the TUI path was never idle. Guards against
|
||||||
|
// handing out a pane whose `claude` has been sitting so long it may have drifted
|
||||||
|
// (auto-compaction prompts, an idle-disconnect banner, an expired in-pane token).
|
||||||
|
export const POOL_MAX_AGE_MS = 10 * 60 * 1000;
|
||||||
|
|
||||||
|
// Clamp the operator-supplied size into [0, POOL_MAX_SIZE]. A garbage value disables the
|
||||||
|
// pool rather than guessing — an unparseable size must never silently boot 4 processes.
|
||||||
|
export function resolvePoolSize(raw) {
|
||||||
|
const n = parseInt(raw, 10);
|
||||||
|
if (!Number.isFinite(n) || n <= 0) return 0;
|
||||||
|
return Math.min(n, POOL_MAX_SIZE);
|
||||||
|
}
|
||||||
|
|
||||||
|
export class TuiPanePool {
|
||||||
|
// size: target number of warm panes (0 = disabled).
|
||||||
|
// maxAgeMs: per-pane TTL (see POOL_MAX_AGE_MS).
|
||||||
|
// mintPane: () => ({ sessionId, name }) — mints the identity of the NEXT pane. The POOL,
|
||||||
|
// not the boot function, owns this: the tmux session springs into existence the
|
||||||
|
// instant bootPane starts, so the pool must already know its NAME (see
|
||||||
|
// _bootingPane below). Deriving the name from the sessionId also makes `tmux ls`
|
||||||
|
// correlate to the transcript file.
|
||||||
|
// bootPane: async (model, {sessionId, name}) => { name, sessionId, model, bootedAt } —
|
||||||
|
// boots ONE pane under exactly that identity and resolves only once it is
|
||||||
|
// input-ready; throws if it never becomes ready.
|
||||||
|
// killPane: (name) => void — tmux kill-session. MUST be synchronous (see drain).
|
||||||
|
// paneHealthy:(name) => bool — pane still exists AND is still at its input bar.
|
||||||
|
constructor({ size, maxAgeMs = POOL_MAX_AGE_MS, mintPane, bootPane, killPane, paneHealthy, now = Date.now, log = () => {} }) {
|
||||||
|
this.size = Math.max(0, Math.min(parseInt(size, 10) || 0, POOL_MAX_SIZE));
|
||||||
|
// Fail fast at CONSTRUCTION, not at request time. refill() is called synchronously from
|
||||||
|
// the request path (runTuiTurn), so a missing collaborator would otherwise surface as a
|
||||||
|
// 500 on a live request instead of a loud error at boot.
|
||||||
|
if (this.size > 0) {
|
||||||
|
for (const [k, fn] of [["mintPane", mintPane], ["bootPane", bootPane], ["killPane", killPane], ["paneHealthy", paneHealthy]]) {
|
||||||
|
if (typeof fn !== "function") throw new TypeError(`TuiPanePool: ${k} must be a function`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
this.maxAgeMs = maxAgeMs;
|
||||||
|
this._mintPane = mintPane;
|
||||||
|
this._bootPane = bootPane;
|
||||||
|
this._killPane = killPane;
|
||||||
|
this._paneHealthy = paneHealthy;
|
||||||
|
this._now = now;
|
||||||
|
this._log = log;
|
||||||
|
|
||||||
|
this._panes = []; // warm, available panes: { name, sessionId, model, bootedAt }
|
||||||
|
// The pane currently BOOTING, BY NAME ({sessionId, name, model}) — or null.
|
||||||
|
//
|
||||||
|
// WHY A NAME AND NOT A COUNT (this is a fixed bug, don't regress it): bootTuiPane creates
|
||||||
|
// the tmux session SYNCHRONOUSLY and only THEN waits up to POOL_BOOT_MS (20 s) for the
|
||||||
|
// input bar. So for up to 20 s there is a LIVE pooled tmux session. When the pool tracked
|
||||||
|
// only a count, it could not NAME that session, so:
|
||||||
|
// - liveNames() could not spare it and the periodic reap sweep KILLED it (and
|
||||||
|
// kill-server'd on top), leaving the pool empty with nothing scheduled and firing the
|
||||||
|
// very tui_pool_boot_failed WARN operators are told to alert on; and
|
||||||
|
// - drain() could not kill it, so on shutdown it ORPHANED a live authenticated `claude`
|
||||||
|
// (the boot's .then that was supposed to clean up never runs — gracefulShutdown calls
|
||||||
|
// process.exit in the same tick).
|
||||||
|
// Both are fixed by holding the identity here, before the session exists.
|
||||||
|
this._bootingPane = null;
|
||||||
|
// Generation counter. Bumped whenever an in-flight boot is CANCELLED (drain / model
|
||||||
|
// switch). A boot compares the generation it started under against the current one:
|
||||||
|
// if they differ, its pane was already killed by us and its settle is inert — in
|
||||||
|
// particular a rejection is a CANCELLATION, not an operator-visible boot failure.
|
||||||
|
this._gen = 0;
|
||||||
|
this._paused = false; // true while drained; refill() is a no-op until resume()
|
||||||
|
this.warmModel = null; // the model the pool currently warms — learned from traffic (see acquire)
|
||||||
|
|
||||||
|
this.hits = 0; // requests served by a warm pane
|
||||||
|
this.misses = 0; // requests that fell back to the cold path
|
||||||
|
this.boots = 0; // panes successfully pre-booted
|
||||||
|
this.bootFailures = 0; // pre-boots that genuinely never reached the input bar
|
||||||
|
this.cancelled = 0; // in-flight boots WE killed (drain / model switch) — not failures
|
||||||
|
this.dropped = 0; // panes discarded unused (unhealthy / expired / wrong model / drained /
|
||||||
|
// cancelled — a cancelled in-flight boot also lands here via _drop)
|
||||||
|
}
|
||||||
|
|
||||||
|
get enabled() { return this.size > 0; }
|
||||||
|
get warm() { return this._panes.length; }
|
||||||
|
get booting() { return this._bootingPane ? 1 : 0; }
|
||||||
|
|
||||||
|
// The reaper's spare set: the EXACT names of every pane the pool currently owns and has NOT
|
||||||
|
// handed out — the warm ones AND the one currently booting (whose tmux session is already
|
||||||
|
// live; see _bootingPane). See the POOL/REAPER INVARIANT in lib/tui/session.mjs.
|
||||||
|
// Fail-safe by construction: a pane leaves this set the instant it is acquired, dropped, or
|
||||||
|
// cancelled, and if the pool is empty (or the process restarted) the set is empty — so an
|
||||||
|
// orphaned pooled pane looks exactly like any other stale session and IS reaped.
|
||||||
|
liveNames() {
|
||||||
|
const names = new Set(this._panes.map((p) => p.name));
|
||||||
|
if (this._bootingPane) names.add(this._bootingPane.name);
|
||||||
|
return names;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Take a warm pane for `model`, or null (caller must fall back to the cold path — a MISS
|
||||||
|
// is always safe, never an error). Synchronous: paneHealthy is a cheap tmux capture.
|
||||||
|
//
|
||||||
|
// The pool warms the MOST RECENTLY REQUESTED model (`warmModel`). There is no boot-time
|
||||||
|
// pre-warm and no configured model: OCP cannot know which model the next caller wants, and
|
||||||
|
// pre-booting a process for a model nobody asks for is pure waste. Consequence, stated
|
||||||
|
// plainly: the FIRST request after start (and the first after a model switch) is always a
|
||||||
|
// MISS. The pool pays off for the steady repeat traffic it exists to serve.
|
||||||
|
acquire(model) {
|
||||||
|
if (!this.enabled) return null;
|
||||||
|
|
||||||
|
// Retarget on a model switch: --model is fixed at spawn, so panes for another model are
|
||||||
|
// useless. Drop them now (they are replaced by the next refill) rather than holding
|
||||||
|
// processes for a model that is no longer being asked for. This includes any pane
|
||||||
|
// currently BOOTING for the old model — its tmux session already exists, so leaving it to
|
||||||
|
// die on resolve would both hold a useless process and block the next refill (one boot at
|
||||||
|
// a time) for up to POOL_BOOT_MS.
|
||||||
|
if (model !== this.warmModel) {
|
||||||
|
for (const p of this._panes) { this._drop(p, "model_switch"); }
|
||||||
|
this._panes = [];
|
||||||
|
this._cancelBooting("model_switch");
|
||||||
|
this.warmModel = model;
|
||||||
|
}
|
||||||
|
|
||||||
|
while (this._panes.length) {
|
||||||
|
const p = this._panes.shift();
|
||||||
|
if (this._now() - p.bootedAt > this.maxAgeMs) { this._drop(p, "expired"); continue; }
|
||||||
|
if (!this._paneHealthy(p.name)) { this._drop(p, "unhealthy"); continue; }
|
||||||
|
this.hits++;
|
||||||
|
return p; // caller OWNS it now: it is out of the registry (so out of the spare set),
|
||||||
|
// and the caller's finally MUST kill it. Single-use — never returned here.
|
||||||
|
}
|
||||||
|
this.misses++;
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Bring the pool back up to `size` warm panes for `warmModel`. Fire-and-forget: never
|
||||||
|
// awaited on the request path and never throws into it.
|
||||||
|
//
|
||||||
|
// SLOT ACCOUNTING: a refill boot deliberately does NOT take a TuiSemaphore slot. Those
|
||||||
|
// slots bound concurrent *turns* (each up to the 120 s wallclock) and belong to real
|
||||||
|
// requests; charging a background pre-boot against them would let the pool starve the
|
||||||
|
// traffic it exists to speed up. It cannot leak a slot either, because it never holds one.
|
||||||
|
//
|
||||||
|
// SERIALIZED, ONE BOOT AT A TIME (and re-kicked on success until the pool is at target).
|
||||||
|
// An earlier version launched all `want` boots at once; live at size=2 that put two cold
|
||||||
|
// `claude` boots plus an in-flight turn on the CPU together, and a refill overran even the
|
||||||
|
// generous pool readiness cap (tui_pool_boot_failed). Booting sequentially keeps each boot
|
||||||
|
// near its uncontended ~1.2 s, bounds the CPU burst the pool can cause, and still has the
|
||||||
|
// replacement pane warm long before the next request arrives.
|
||||||
|
//
|
||||||
|
// A genuinely FAILED boot deliberately does NOT re-kick the chain — that is the backoff. A
|
||||||
|
// persistently failing boot (bad claude binary, no auth) would otherwise spin, respawning
|
||||||
|
// forever. The next natural trigger (the following request's refill, or the reap tick's
|
||||||
|
// resume) retries it. A CANCELLED boot is different: we killed it on purpose, nothing is
|
||||||
|
// wrong, and resume() is expected to start a fresh one immediately.
|
||||||
|
refill() {
|
||||||
|
if (!this.enabled || this._paused || !this.warmModel) return;
|
||||||
|
if (this._bootingPane) return; // one boot in flight at a time
|
||||||
|
if (this._panes.length >= this.size) return; // already at target
|
||||||
|
|
||||||
|
const model = this.warmModel;
|
||||||
|
const gen = this._gen;
|
||||||
|
// Mint the identity BEFORE booting: bootPane creates the tmux session synchronously, so
|
||||||
|
// the pool must be able to name (and therefore spare, and kill) it from this moment on.
|
||||||
|
const ident = this._mintPane();
|
||||||
|
this._bootingPane = { ...ident, model };
|
||||||
|
let enlisted = false;
|
||||||
|
Promise.resolve()
|
||||||
|
.then(() => this._bootPane(model, ident))
|
||||||
|
.then((pane) => {
|
||||||
|
// The world may have moved while we booted. If our generation was cancelled, kill the
|
||||||
|
// pane here rather than ASSUMING _cancelBooting already did.
|
||||||
|
//
|
||||||
|
// Why not just `return`: _cancelBooting kills by name, but the tmux session only EXISTS
|
||||||
|
// once _bootPane has actually run — and _bootPane is queued on a microtask (above). A
|
||||||
|
// caller that does refill() and then drain() in the SAME synchronous block would have
|
||||||
|
// _cancelBooting find nothing to kill (a no-op), bump the generation, and then this
|
||||||
|
// microtask would create the session, boot it fine, and — under a bare `return` — walk
|
||||||
|
// away from a LIVE authenticated `claude` that nothing owns. That is M1b in a new costume.
|
||||||
|
// No current call site does that, so this is defense-in-depth, not a live bug — but ADR
|
||||||
|
// 0008 and the reap-tick comment in server.mjs both explicitly contemplate a boot-time
|
||||||
|
// pre-warm, which is exactly the shape that would reach it.
|
||||||
|
//
|
||||||
|
// Killing an already-dead session is a harmless no-op (_drop swallows it), so this is
|
||||||
|
// idempotent whether or not _cancelBooting got there first.
|
||||||
|
if (gen !== this._gen) { this._drop(pane, "cancelled_late"); return; }
|
||||||
|
// Otherwise: still possible the pool filled or retargeted without a cancellation.
|
||||||
|
if (this._paused || model !== this.warmModel || this._panes.length >= this.size) {
|
||||||
|
this._drop(pane, "stale_boot");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this._panes.push(pane);
|
||||||
|
this.boots++;
|
||||||
|
enlisted = true;
|
||||||
|
})
|
||||||
|
.catch((e) => {
|
||||||
|
// A rejection from a CANCELLED generation is not a fault: it is almost always
|
||||||
|
// "tui_pane_not_ready", thrown because WE killed the pane out from under the boot.
|
||||||
|
// Counting it as a bootFailure would fire the exact WARN operators are told to alert
|
||||||
|
// on, for a completely healthy drain. Stay silent — _cancelBooting already counted
|
||||||
|
// this as a cancellation, so do NOT count it again here.
|
||||||
|
if (gen !== this._gen) return;
|
||||||
|
this.bootFailures++;
|
||||||
|
this._log("warn", "tui_pool_boot_failed", { model, error: e && e.message });
|
||||||
|
})
|
||||||
|
.finally(() => {
|
||||||
|
// ONLY the current generation's boot owns the booting slot. A stale settle must not
|
||||||
|
// clear a slot that a newer boot (started by resume()) already holds.
|
||||||
|
if (gen === this._gen) this._bootingPane = null;
|
||||||
|
if (enlisted) this.refill(); // continue toward target, still one at a time
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Kill the in-flight boot's pane, SYNCHRONOUSLY, and invalidate its generation. Returns 1
|
||||||
|
// if there was one, else 0. The tmux session already exists (bootPane created it before it
|
||||||
|
// started waiting for readiness), so this is a real kill, not a cancellation flag.
|
||||||
|
_cancelBooting(reason) {
|
||||||
|
if (!this._bootingPane) return 0;
|
||||||
|
this._gen++; // the in-flight boot's settle is now inert
|
||||||
|
this._drop(this._bootingPane, reason); // synchronous kill-session
|
||||||
|
this._bootingPane = null;
|
||||||
|
this.cancelled++;
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Kill every pane the pool owns — warm AND currently booting — and stop refilling. Returns
|
||||||
|
// how many were killed.
|
||||||
|
//
|
||||||
|
// Called (a) before the periodic reap sweep — reapStaleTuiSessions can only reap defunct
|
||||||
|
// `claude` zombies via kill-server, and kill-server is suppressed while any live pooled pane
|
||||||
|
// exists (including a booting one), so without this drain the pool would permanently disable
|
||||||
|
// zombie reaping; and (b) on graceful shutdown, so no pane outlives the process as an orphan.
|
||||||
|
//
|
||||||
|
// EVERY KILL HERE IS SYNCHRONOUS, and that is load-bearing. It is NOT safe to leave the
|
||||||
|
// booting pane to clean itself up on resolve: gracefulShutdown calls process.exit() in the
|
||||||
|
// same tick as this drain (TUI panes are children of the tmux SERVER, not of node, so
|
||||||
|
// node's activeProcesses set is empty on a TUI host and the "wait for children" path exits
|
||||||
|
// immediately). A .then()/.catch() scheduled here would never run, and the pane would
|
||||||
|
// survive as an orphaned, authenticated, idle `claude`.
|
||||||
|
drain() {
|
||||||
|
this._paused = true;
|
||||||
|
let n = this._panes.length;
|
||||||
|
for (const p of this._panes) this._drop(p, "drain");
|
||||||
|
this._panes = [];
|
||||||
|
n += this._cancelBooting("drain_booting");
|
||||||
|
return n;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Undo drain() and start refilling again. Because drain() CANCELLED the in-flight boot
|
||||||
|
// (rather than leaving it pending), the booting slot is free and this really does start a
|
||||||
|
// fresh boot — the pool is never left empty with nothing scheduled.
|
||||||
|
resume() {
|
||||||
|
this._paused = false;
|
||||||
|
this.refill();
|
||||||
|
}
|
||||||
|
|
||||||
|
// /health surface (additive).
|
||||||
|
stats() {
|
||||||
|
return {
|
||||||
|
size: this.size,
|
||||||
|
warm: this._panes.length,
|
||||||
|
booting: this.booting,
|
||||||
|
model: this.warmModel,
|
||||||
|
hits: this.hits,
|
||||||
|
misses: this.misses,
|
||||||
|
boots: this.boots,
|
||||||
|
bootFailures: this.bootFailures,
|
||||||
|
cancelled: this.cancelled,
|
||||||
|
dropped: this.dropped,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
_drop(pane, reason) {
|
||||||
|
this.dropped++;
|
||||||
|
try { this._killPane(pane.name); } catch { /* already gone */ }
|
||||||
|
this._log("info", "tui_pool_pane_dropped", { name: pane.name, reason });
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -139,7 +139,12 @@ export function recordTuiEntrypoint(tuiStats, observed, expectedMode = "cli") {
|
|||||||
// Build the additive /health `tui` block (ADR 0007 PR-B amendment). Pure: given the
|
// Build the additive /health `tui` block (ADR 0007 PR-B amendment). Pure: given the
|
||||||
// config + live counters, returns the exact object embedded in /health. New fields only —
|
// config + live counters, returns the exact object embedded in /health. New fields only —
|
||||||
// behaviour-preserving for existing /health consumers (grandfathered B.2 under ADR 0006).
|
// behaviour-preserving for existing /health consumers (grandfathered B.2 under ADR 0006).
|
||||||
export function buildTuiHealthBlock({ enabled, entrypointMode, maxConcurrent }, tuiStats, semaphore) {
|
//
|
||||||
|
// `pool` (optional, warm pane pool — lib/tui/pool.mjs): a TuiPanePool, or null/undefined
|
||||||
|
// when the pool is off (the default). Reported as `pool: null` when off so the block's
|
||||||
|
// shape stays stable, and as the pool's stats (size / warm / hits / misses / …) when on —
|
||||||
|
// the operator's window onto both the hit rate and the standing idle-process cost.
|
||||||
|
export function buildTuiHealthBlock({ enabled, entrypointMode, maxConcurrent }, tuiStats, semaphore, pool = null) {
|
||||||
return {
|
return {
|
||||||
enabled,
|
enabled,
|
||||||
entrypointMode, // cli | auto | off
|
entrypointMode, // cli | auto | off
|
||||||
@@ -148,5 +153,6 @@ export function buildTuiHealthBlock({ enabled, entrypointMode, maxConcurrent },
|
|||||||
inflight: semaphore.inflight, // current concurrent TUI turns
|
inflight: semaphore.inflight, // current concurrent TUI turns
|
||||||
queued: semaphore.queued, // turns waiting for a slot
|
queued: semaphore.queued, // turns waiting for a slot
|
||||||
maxConcurrent,
|
maxConcurrent,
|
||||||
|
pool: pool ? pool.stats() : null, // warm pane pool, or null when disabled
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|||||||
+174
-50
@@ -73,6 +73,36 @@ const defaultTmux = (args, opts = {}) =>
|
|||||||
// `port` (required) is this instance's own listen port (server.mjs's PORT / lib/constants.mjs
|
// `port` (required) is this instance's own listen port (server.mjs's PORT / lib/constants.mjs
|
||||||
// DEFAULT_PORT resolution) — the SPOT for "which sessions are ours."
|
// DEFAULT_PORT resolution) — the SPOT for "which sessions are ours."
|
||||||
//
|
//
|
||||||
|
// ── POOL/REAPER INVARIANT (warm pane pool — lib/tui/pool.mjs) ───────────────────────────
|
||||||
|
// A warm pooled pane is one of OUR OWN `ocp-tui-<port>-*` sessions that is ALIVE AND IDLE
|
||||||
|
// BY DESIGN — and the periodic sweep runs precisely when the instance is idle, i.e. exactly
|
||||||
|
// when the pool is full. Without an exemption the sweep would kill every warm pane on every
|
||||||
|
// tick (and kill-server on top). The exemption is `spare`: a set of EXACT session names the
|
||||||
|
// caller declares live. Three properties, all load-bearing:
|
||||||
|
//
|
||||||
|
// 1. A LIVE POOLED PANE IS NEVER REAPED — INCLUDING ONE THAT IS STILL BOOTING. It is in
|
||||||
|
// `spare` (the pool's live registry), so it is skipped by name. The booting case is not
|
||||||
|
// a footnote, it is the one that bit us: bootTuiPane creates the tmux session
|
||||||
|
// SYNCHRONOUSLY and only then waits up to POOL_BOOT_MS for the input bar, so a pooled
|
||||||
|
// session can be live for ~20 s before its boot resolves. The pool therefore mints the
|
||||||
|
// pane's NAME up front and holds it in `_bootingPane`, so liveNames() can name — and
|
||||||
|
// spare — a session whose boot has not finished. (An earlier version tracked only a
|
||||||
|
// COUNT of in-flight boots; the sweep could not name that session and killed it.)
|
||||||
|
// 2. A LEAKED/ORPHANED POOLED PANE IS STILL REAPED. Membership is by EXACT NAME from a
|
||||||
|
// live in-memory registry — NOT by "looks pooled" (name shape). A pane the pool no
|
||||||
|
// longer owns (handed out, dropped, cancelled, or left behind by a previous process
|
||||||
|
// generation — whose registry died with it) is absent from `spare` and is killed like
|
||||||
|
// any other stale session. Fail-safe: forgetting to pass `spare` reaps MORE, never less.
|
||||||
|
// 3. KILL-SERVER NEVER KILLS A LIVE POOL PANE. A spared session suppresses kill-server
|
||||||
|
// exactly as a foreign session does (it is a live child of the tmux server). The
|
||||||
|
// consequence — that a permanently-full pool would permanently disable the defunct-
|
||||||
|
// zombie reaping that ONLY kill-server can do — is resolved in server.mjs by DRAINING
|
||||||
|
// the pool immediately before the sweep, so `spare` is empty on the normal tick and
|
||||||
|
// kill-server still fires. `spare` is the belt-and-braces: a reap call site that
|
||||||
|
// forgets to drain still cannot kill a live pane.
|
||||||
|
//
|
||||||
|
// `spare` (default: none) — iterable of session names, or a Set. Ignored when the pool is off.
|
||||||
|
//
|
||||||
// `includeLegacy` (default false): when true, sessions matching the exact OLD bare-prefix
|
// `includeLegacy` (default false): when true, sessions matching the exact OLD bare-prefix
|
||||||
// shape (LEGACY_SESSION_NAME_RE) are ALSO treated as ours for kill-session purposes. This is
|
// shape (LEGACY_SESSION_NAME_RE) are ALSO treated as ours for kill-session purposes. This is
|
||||||
// the boot-time legacy migration: an operator upgrading past this fix could otherwise be left
|
// the boot-time legacy migration: an operator upgrading past this fix could otherwise be left
|
||||||
@@ -88,14 +118,20 @@ const defaultTmux = (args, opts = {}) =>
|
|||||||
// same class of residual risk the audit finding itself accepts ("no live instance of the new
|
// same class of residual risk the audit finding itself accepts ("no live instance of the new
|
||||||
// version creates them"); this PR does not regress that scenario, it only removes the far
|
// version creates them"); this PR does not regress that scenario, it only removes the far
|
||||||
// more common same-version collision (the actual F7 finding).
|
// more common same-version collision (the actual F7 finding).
|
||||||
export function reapStaleTuiSessions({ tmux = defaultTmux, port, includeLegacy = false } = {}) {
|
export function reapStaleTuiSessions({ tmux = defaultTmux, port, includeLegacy = false, spare = null } = {}) {
|
||||||
const r = tmux(["list-sessions", "-F", "#{session_name}"]);
|
const r = tmux(["list-sessions", "-F", "#{session_name}"]);
|
||||||
if (!r || r.status !== 0) return 0; // no tmux server / no sessions
|
if (!r || r.status !== 0) return 0; // no tmux server / no sessions
|
||||||
const names = String(r.stdout || "").split("\n").map((s) => s.trim()).filter(Boolean);
|
const names = String(r.stdout || "").split("\n").map((s) => s.trim()).filter(Boolean);
|
||||||
const ownPrefix = sessionPrefixForPort(port);
|
const ownPrefix = sessionPrefixForPort(port);
|
||||||
|
const spared = spare instanceof Set ? spare : new Set(spare || []);
|
||||||
let killed = 0;
|
let killed = 0;
|
||||||
let othersRemain = false;
|
let othersRemain = false;
|
||||||
|
let sparedLive = 0;
|
||||||
for (const name of names) {
|
for (const name of names) {
|
||||||
|
// Property 1+2: exemption is by EXACT NAME from the pool's live registry. A pooled-
|
||||||
|
// LOOKING name that is not in the registry is an orphan and falls through to the
|
||||||
|
// normal kill path below.
|
||||||
|
if (spared.has(name)) { sparedLive++; continue; }
|
||||||
const isOwn = name.startsWith(ownPrefix);
|
const isOwn = name.startsWith(ownPrefix);
|
||||||
const isLegacyOwn = includeLegacy && LEGACY_SESSION_NAME_RE.test(name);
|
const isLegacyOwn = includeLegacy && LEGACY_SESSION_NAME_RE.test(name);
|
||||||
if (isOwn || isLegacyOwn) {
|
if (isOwn || isLegacyOwn) {
|
||||||
@@ -109,7 +145,11 @@ export function reapStaleTuiSessions({ tmux = defaultTmux, port, includeLegacy =
|
|||||||
// Reap defunct `claude` zombies: safe ONLY when the server is now ours-only/empty.
|
// Reap defunct `claude` zombies: safe ONLY when the server is now ours-only/empty.
|
||||||
// kill-server is what actually reaps (server exit reparents survivors to init); a
|
// kill-server is what actually reaps (server exit reparents survivors to init); a
|
||||||
// per-session kill cannot, since node is not the zombies' parent.
|
// per-session kill cannot, since node is not the zombies' parent.
|
||||||
if (!othersRemain) {
|
//
|
||||||
|
// Property 3: a SPARED session is a live child of this tmux server, so kill-server would
|
||||||
|
// kill it — it therefore suppresses kill-server exactly as a foreign session does. On the
|
||||||
|
// normal sweep the pool is drained first, so sparedLive is 0 and kill-server still fires.
|
||||||
|
if (!othersRemain && sparedLive === 0) {
|
||||||
tmux(["kill-server"]);
|
tmux(["kill-server"]);
|
||||||
}
|
}
|
||||||
return killed;
|
return killed;
|
||||||
@@ -119,6 +159,12 @@ export function reapStaleTuiSessions({ tmux = defaultTmux, port, includeLegacy =
|
|||||||
|
|
||||||
// Boot + paste-settle timing. Conservative defaults validated on PI231; env-tunable.
|
// Boot + paste-settle timing. Conservative defaults validated on PI231; env-tunable.
|
||||||
const BOOT_MS = parseInt(process.env.OCP_TUI_BOOT_MS || "4000", 10); // max wait for input-ready
|
const BOOT_MS = parseInt(process.env.OCP_TUI_BOOT_MS || "4000", 10); // max wait for input-ready
|
||||||
|
// Readiness cap for a POOL pre-boot. Deliberately far more generous than BOOT_MS: BOOT_MS is
|
||||||
|
// tight because a client is blocked on it, whereas a warm-pane boot happens in the background
|
||||||
|
// with nobody waiting. Observed live at size=2: a refill booting alongside an in-flight turn
|
||||||
|
// exceeded 4000 ms and was discarded (tui_pool_boot_failed), quietly costing hit rate for a
|
||||||
|
// pane that was merely slow, not broken. Scales with OCP_TUI_BOOT_MS if an operator raises it.
|
||||||
|
export const POOL_BOOT_MS = BOOT_MS * 5;
|
||||||
const READY_POLL_MS = parseInt(process.env.OCP_TUI_READY_POLL_MS || "400", 10); // readiness / paste-verify poll interval
|
const READY_POLL_MS = parseInt(process.env.OCP_TUI_READY_POLL_MS || "400", 10); // readiness / paste-verify poll interval
|
||||||
const PASTE_VERIFY_MS = parseInt(process.env.OCP_TUI_PASTE_VERIFY_MS || "5000", 10); // max wait for pasted prompt to render
|
const PASTE_VERIFY_MS = parseInt(process.env.OCP_TUI_PASTE_VERIFY_MS || "5000", 10); // max wait for pasted prompt to render
|
||||||
|
|
||||||
@@ -412,38 +458,60 @@ export function buildTuiCmd(claudeBin, model, sessionId, ehome, entrypointMode)
|
|||||||
].join(" ");
|
].join(" ");
|
||||||
}
|
}
|
||||||
|
|
||||||
// Full per-request TUI lifecycle:
|
// Is a pane alive AND still sitting at its input bar? Used by the warm pool to decide,
|
||||||
// 1. Pre-trust the scratch cwd (no trust dialog will appear).
|
// at hand-out time, whether a pre-booted pane is still usable (a dead/degraded pane must
|
||||||
// 2. Write prompt to a 0600 temp file (no shell injection from prompt content).
|
// become a MISS → cold path, never a hung turn). capture-pane exits non-zero when the
|
||||||
// 3. Boot an interactive `claude` in a fresh tmux session in the scratch cwd; poll
|
// session no longer exists, so this covers "pane gone" and "pane not ready" in one call.
|
||||||
// capture-pane until the `? for shortcuts` input bar appears (readiness-poll
|
export function tuiPaneHealthy(tmux, tmuxName) {
|
||||||
// replaces the old blind boot sleep). BOOT_MS is the max wait, not a fixed delay.
|
const r = tmux(["capture-pane", "-p", "-t", tmuxName]);
|
||||||
// 4. Paste the prompt via tmux load-buffer + paste-buffer -p (bracketed paste) —
|
if (!r || r.status !== 0 || typeof r.stdout !== "string") return false;
|
||||||
// reliable for large multi-line prompts where send-keys -l is not (issue #130).
|
return tuiInputReady(r.stdout);
|
||||||
// Poll-verify the prompt landed in the input (placeholder gone / [Pasted text]);
|
}
|
||||||
// fast-fail with tui_paste_not_landed if it never lands (prevents the 120s
|
|
||||||
// wallclock "stuck typing" hang). Then submit with a SEPARATE Enter key event.
|
// Pool pane names carry a "p" marker after the port-scoped prefix:
|
||||||
// 5. Block on the native JSONL transcript (located by session-id) until terminal
|
// turn pane: ocp-tui-<port>-<8hex> (unchanged)
|
||||||
// marker or wall-clock cap.
|
// pool pane: ocp-tui-<port>-p<8hex>
|
||||||
// 6. Always teardown: kill session + rm temp dir (even on throw).
|
// Purely for operator legibility (`tmux ls` shows which panes are warm). It is NOT the
|
||||||
// Returns { text, entrypoint } from readTuiTranscript (entrypoint is the billing-pool
|
// reaper's exemption mechanism — that is the exact-name spare set (see the POOL/REAPER
|
||||||
// classifier, e.g. "cli", or null if the transcript did not include a turn_duration).
|
// INVARIANT above), so a pooled-LOOKING orphan is still reaped. Both shapes start with
|
||||||
export async function runTuiTurn({
|
// sessionPrefixForPort(port), so both remain reapable as "ours", and neither can match
|
||||||
prompt,
|
// LEGACY_SESSION_NAME_RE.
|
||||||
model,
|
export function poolPaneName(port, sessionId) {
|
||||||
claudeBin,
|
return sessionPrefixForPort(port) + "p" + sessionId.slice(0, 8);
|
||||||
home,
|
}
|
||||||
realHome,
|
|
||||||
cwd,
|
// Boot ONE interactive `claude` pane and wait for its input bar. Shared by the cold
|
||||||
port,
|
// request path (runTuiTurn) and the warm pool (lib/tui/pool.mjs) so a pooled pane is
|
||||||
wallclockMs = 120000,
|
// spawned with byte-for-byte the same argv, HOME, cwd and trust preparation as a
|
||||||
entrypointMode = "cli",
|
// cold-booted one — the pool must not become a second, drifting spawn path.
|
||||||
tmux = defaultTmux,
|
//
|
||||||
|
// Each pane gets its OWN fresh randomUUID() --session-id, fixed at boot. That is what
|
||||||
|
// keeps a pooled pane single-use-safe: its transcript holds exactly one exchange.
|
||||||
|
//
|
||||||
|
// requireReady: the cold path tolerates a readiness timeout (it falls through and lets
|
||||||
|
// the paste-verify decide — pre-existing behaviour, unchanged). The POOL sets it, because
|
||||||
|
// a pane that never reached its input bar is worthless as a warm pane and must not be
|
||||||
|
// enlisted: throw, let the pool count a bootFailure, and leave the request path to
|
||||||
|
// cold-boot as usual.
|
||||||
|
// bootMs: max wait for the input bar. Defaults to BOOT_MS (the REQUEST path's cap, which is
|
||||||
|
// deliberately tight — a client is blocked on it). The POOL passes POOL_BOOT_MS instead: a
|
||||||
|
// background pre-boot has nobody waiting on it, and capping it at the request-path's 4 s
|
||||||
|
// made real refills fail (observed live: a refill booting alongside an in-flight turn took
|
||||||
|
// >4 s and was discarded, silently lowering the hit rate). Slow != broken for a pre-boot.
|
||||||
|
// `sessionId` / `name` (both optional): the caller may supply the pane's identity instead of
|
||||||
|
// letting bootTuiPane mint it. The POOL does, because it must know the tmux session's NAME
|
||||||
|
// before this function runs — the session is created synchronously below, well before the
|
||||||
|
// readiness wait returns, so a pool that only learned the name on resolve could neither spare
|
||||||
|
// the session from the reaper nor kill it on shutdown. Supplying BOTH also keeps the name's
|
||||||
|
// hex suffix equal to the session-id's, so `tmux ls` correlates to the transcript file.
|
||||||
|
export async function bootTuiPane({
|
||||||
|
model, claudeBin, home, realHome, cwd, port, entrypointMode = "cli",
|
||||||
|
tmux = defaultTmux, sessionId = null, name = null, requireReady = false, bootMs = BOOT_MS,
|
||||||
}) {
|
}) {
|
||||||
const sessionId = randomUUID();
|
const sid = sessionId || randomUUID();
|
||||||
// Port-scoped session name (F7 fix) — see sessionPrefixForPort / reapStaleTuiSessions
|
// Port-scoped session name (F7 fix) — see sessionPrefixForPort / reapStaleTuiSessions
|
||||||
// for why this instance's own listen port is the namespace discriminator.
|
// for why this instance's own listen port is the namespace discriminator.
|
||||||
const tmuxName = sessionPrefixForPort(port) + sessionId.slice(0, 8);
|
const tmuxName = name || (sessionPrefixForPort(port) + sid.slice(0, 8));
|
||||||
const ehome = home || process.env.HOME; // HOME claude runs under (scratch or real)
|
const ehome = home || process.env.HOME; // HOME claude runs under (scratch or real)
|
||||||
const rhome = realHome || process.env.HOME; // real home (OAuth + onboarded config source)
|
const rhome = realHome || process.env.HOME; // real home (OAuth + onboarded config source)
|
||||||
|
|
||||||
@@ -460,42 +528,97 @@ export async function runTuiTurn({
|
|||||||
if (!existsSync(cwd)) mkdirSync(cwd, { recursive: true });
|
if (!existsSync(cwd)) mkdirSync(cwd, { recursive: true });
|
||||||
prepareTuiHome(rhome, ehome, cwd, { envTokenMode });
|
prepareTuiHome(rhome, ehome, cwd, { envTokenMode });
|
||||||
|
|
||||||
// Write prompt to a temp file (mode 0600) so the content never touches argv.
|
|
||||||
const tmpDir = mkdtempSync(`${tmpdir()}/ocp-tui-`);
|
|
||||||
const promptFile = `${tmpDir}/prompt.txt`;
|
|
||||||
writeFileSync(promptFile, prompt, { mode: 0o600 });
|
|
||||||
|
|
||||||
// Minimal env for spawnSync (tmux itself). The pane's claude env comes exclusively
|
// Minimal env for spawnSync (tmux itself). The pane's claude env comes exclusively
|
||||||
// from the `env` prefix string built inside buildTuiCmd — tmux does NOT forward the
|
// from the `env` prefix string built inside buildTuiCmd — tmux does NOT forward the
|
||||||
// spawning process's env to the pane, so the {env} here is intentionally minimal.
|
// spawning process's env to the pane, so the {env} here is intentionally minimal.
|
||||||
const env = { ...process.env };
|
const env = { ...process.env };
|
||||||
env.HOME = ehome; // tmux needs HOME; all claude-specific vars go via buildTuiCmd prefix
|
env.HOME = ehome; // tmux needs HOME; all claude-specific vars go via buildTuiCmd prefix
|
||||||
|
|
||||||
try {
|
// Boot the interactive session inside tmux, rooted at the scratch cwd.
|
||||||
// 1. Boot the interactive session inside tmux, rooted at the scratch cwd.
|
// Capture the result: if tmux new-session fails (status !== 0) there is no PTY, no
|
||||||
// Capture the result: if tmux new-session fails (status !== 0) there is no
|
// interactive spawn — abort BEFORE the boot wait rather than paste into a non-existent
|
||||||
// PTY, no interactive spawn — abort BEFORE the boot sleep rather than paste
|
// session or issue a billing request without a verified interactive context.
|
||||||
// into a non-existent session or issue a billing request without a verified
|
|
||||||
// interactive context. The finally teardown is still harmless (kill-session
|
|
||||||
// is a no-op when the session never existed).
|
|
||||||
const spawnResult = tmux(
|
const spawnResult = tmux(
|
||||||
["new-session", "-d", "-s", tmuxName, "-x", "220", "-y", "50", "-c", cwd,
|
["new-session", "-d", "-s", tmuxName, "-x", "220", "-y", "50", "-c", cwd,
|
||||||
buildTuiCmd(claudeBin, model, sessionId, ehome, entrypointMode)],
|
buildTuiCmd(claudeBin, model, sid, ehome, entrypointMode)],
|
||||||
{ env },
|
{ env },
|
||||||
);
|
);
|
||||||
if (!spawnResult || spawnResult.status !== 0) {
|
if (!spawnResult || spawnResult.status !== 0) {
|
||||||
throw new Error("tui_spawn_failed: tmux session not created");
|
throw new Error("tui_spawn_failed: tmux session not created");
|
||||||
}
|
}
|
||||||
|
|
||||||
// 2. Wait until claude's input bar is actually ready (was: blind sleep(BOOT_MS)).
|
// Wait until claude's input bar is actually ready (not a blind sleep).
|
||||||
// BOOT_MS is now the MAX readiness wait, not a fixed delay.
|
// bootMs is the MAX readiness wait, not a fixed delay.
|
||||||
const ready = await pollUntil(() => tuiInputReady(tuiCapturePane(tmux, tmuxName)),
|
const ready = await pollUntil(() => tuiInputReady(tuiCapturePane(tmux, tmuxName)),
|
||||||
{ timeoutMs: BOOT_MS, intervalMs: READY_POLL_MS });
|
{ timeoutMs: bootMs, intervalMs: READY_POLL_MS });
|
||||||
if (!ready) {
|
if (!ready) {
|
||||||
// (readiness timed out; relying on paste-verify)
|
if (requireReady) {
|
||||||
|
try { tmux(["kill-session", "-t", tmuxName]); } catch { /* already gone */ }
|
||||||
|
throw new Error("tui_pane_not_ready: input bar did not appear within " + bootMs + "ms");
|
||||||
|
}
|
||||||
|
// Cold path (pre-existing behaviour): readiness timed out; rely on paste-verify.
|
||||||
console.error("[tui] input_not_ready", tmuxName);
|
console.error("[tui] input_not_ready", tmuxName);
|
||||||
}
|
}
|
||||||
|
return { name: tmuxName, sessionId: sid, model, ehome, bootedAt: Date.now() };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Full per-request TUI lifecycle:
|
||||||
|
// 1. Take a WARM pane from the pool if one is available for this model (opt-in;
|
||||||
|
// OCP_TUI_POOL_SIZE=0 => always null => steps 2-3 below are exactly today's path).
|
||||||
|
// A pooled pane is SINGLE-USE: it already carries its own fresh --session-id, it
|
||||||
|
// serves this one turn, and it is killed in the finally like any other pane.
|
||||||
|
// 2. On a MISS: pre-trust the scratch cwd, boot an interactive `claude` in a fresh tmux
|
||||||
|
// session in the scratch cwd, poll capture-pane until the `? for shortcuts` input bar
|
||||||
|
// appears (bootTuiPane). BOOT_MS is the max wait, not a fixed delay.
|
||||||
|
// 3. Write prompt to a 0600 temp file (no shell injection from prompt content).
|
||||||
|
// 4. Paste the prompt via tmux load-buffer + paste-buffer -p (bracketed paste) —
|
||||||
|
// reliable for large multi-line prompts where send-keys -l is not (issue #130).
|
||||||
|
// Poll-verify the prompt landed in the input (placeholder gone / [Pasted text]);
|
||||||
|
// fast-fail with tui_paste_not_landed if it never lands (prevents the 120s
|
||||||
|
// wallclock "stuck typing" hang). Then submit with a SEPARATE Enter key event.
|
||||||
|
// 5. Block on the native JSONL transcript (located by THIS pane's session-id) until
|
||||||
|
// terminal marker or wall-clock cap.
|
||||||
|
// 6. Always teardown: kill session + rm temp dir (even on throw), and kick a background
|
||||||
|
// pool refill so the next request finds a warm pane.
|
||||||
|
// Returns { text, entrypoint } from readTuiTranscript (entrypoint is the billing-pool
|
||||||
|
// classifier, e.g. "cli", or null if the transcript did not include a turn_duration).
|
||||||
|
export async function runTuiTurn({
|
||||||
|
prompt,
|
||||||
|
model,
|
||||||
|
claudeBin,
|
||||||
|
home,
|
||||||
|
realHome,
|
||||||
|
cwd,
|
||||||
|
port,
|
||||||
|
wallclockMs = 120000,
|
||||||
|
entrypointMode = "cli",
|
||||||
|
tmux = defaultTmux,
|
||||||
|
pool = null, // TuiPanePool | null — null (default) === today's cold-boot-only path
|
||||||
|
onPane = null, // optional observer: ({ warm }) => void, for logging/metrics
|
||||||
|
}) {
|
||||||
|
// 1. Warm pane, or cold boot. A MISS is never an error — it is exactly today's path.
|
||||||
|
let pane = pool ? pool.acquire(model) : null;
|
||||||
|
const warm = !!pane;
|
||||||
|
// Kick the refill IMMEDIATELY (not after the turn): the replacement pane then boots
|
||||||
|
// CONCURRENTLY with this turn and is warm by the time the next request arrives. Also
|
||||||
|
// runs on a MISS — acquire() has just retargeted the pool to this model, so the miss
|
||||||
|
// that cold-boots today warms the pool for the next caller. Fire-and-forget; it takes
|
||||||
|
// no TuiSemaphore slot (see pool.refill's SLOT ACCOUNTING note).
|
||||||
|
if (pool) pool.refill();
|
||||||
|
if (onPane) { try { onPane({ warm }); } catch { /* observer must never break a turn */ } }
|
||||||
|
if (!pane) {
|
||||||
|
pane = await bootTuiPane({ model, claudeBin, home, realHome, cwd, port, entrypointMode, tmux });
|
||||||
|
}
|
||||||
|
const tmuxName = pane.name;
|
||||||
|
const sessionId = pane.sessionId; // THIS pane's own session-id — one session, one turn
|
||||||
|
const ehome = pane.ehome || home || process.env.HOME;
|
||||||
|
|
||||||
|
// Write prompt to a temp file (mode 0600) so the content never touches argv.
|
||||||
|
const tmpDir = mkdtempSync(`${tmpdir()}/ocp-tui-`);
|
||||||
|
const promptFile = `${tmpDir}/prompt.txt`;
|
||||||
|
writeFileSync(promptFile, prompt, { mode: 0o600 });
|
||||||
|
|
||||||
|
try {
|
||||||
// 3. Paste the prompt via a tmux PASTE BUFFER with bracketed paste (-p), NOT
|
// 3. Paste the prompt via a tmux PASTE BUFFER with bracketed paste (-p), NOT
|
||||||
// `send-keys -l`. send-keys of a large multi-line prompt is unreliable: the
|
// `send-keys -l`. send-keys of a large multi-line prompt is unreliable: the
|
||||||
// embedded newlines arrive as separate key events (effectively repeated Enter),
|
// embedded newlines arrive as separate key events (effectively repeated Enter),
|
||||||
@@ -521,11 +644,12 @@ export async function runTuiTurn({
|
|||||||
// Submit (separate Enter key event).
|
// Submit (separate Enter key event).
|
||||||
tmux(["send-keys", "-t", tmuxName, "Enter"]);
|
tmux(["send-keys", "-t", tmuxName, "Enter"]);
|
||||||
|
|
||||||
// 4. Block on the native transcript (resolved by session-id) until terminal.
|
// 5. Block on the native transcript (resolved by THIS pane's session-id) until terminal.
|
||||||
// Returns { text, entrypoint } from readTuiTranscript.
|
// Returns { text, entrypoint } from readTuiTranscript.
|
||||||
return await readTuiTranscript({ home: ehome, sessionId, wallclockMs });
|
return await readTuiTranscript({ home: ehome, sessionId, wallclockMs });
|
||||||
} finally {
|
} finally {
|
||||||
// 5. Teardown — always, even on throw.
|
// 6. Teardown — always, even on throw. A pooled pane is torn down here exactly like a
|
||||||
|
// cold-booted one: SINGLE-USE, never returned to the pool (see pool.mjs).
|
||||||
try { tmux(["kill-session", "-t", tmuxName]); } catch { /* already gone */ }
|
try { tmux(["kill-session", "-t", tmuxName]); } catch { /* already gone */ }
|
||||||
try { rmSync(tmpDir, { recursive: true, force: true }); } catch { /* best effort */ }
|
try { rmSync(tmpDir, { recursive: true, force: true }); } catch { /* best effort */ }
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -73,6 +73,17 @@ export function isTerminalLine(obj) {
|
|||||||
// transcript holding one logical exchange). If a future warm-pool ever reuses a
|
// transcript holding one logical exchange). If a future warm-pool ever reuses a
|
||||||
// session WITHOUT a fresh session-id / clear, earlier-turn text could leak — that
|
// session WITHOUT a fresh session-id / clear, earlier-turn text could leak — that
|
||||||
// author must add user-line scoping here. See spec §7.2.
|
// author must add user-line scoping here. See spec §7.2.
|
||||||
|
//
|
||||||
|
// STATUS (warm pool, lib/tui/pool.mjs — the "future warm-pool" this note anticipated):
|
||||||
|
// the pool does NOT reuse sessions, so the precondition above still holds and no
|
||||||
|
// user-line scoping was added. Each pooled pane is booted with its OWN fresh
|
||||||
|
// randomUUID() --session-id (bootTuiPane) and is SINGLE-USE: it serves exactly one turn
|
||||||
|
// and is then killed and replaced. One session still means one logical exchange, so the
|
||||||
|
// last assistant entry is still that request's answer.
|
||||||
|
// The warning therefore stands UNCHANGED for anyone who later wants a pane to serve a
|
||||||
|
// SECOND turn (or to reset one with /clear and reuse it): that is a leak, and it needs
|
||||||
|
// user-line scoping HERE before it can be safe. Do not relax pool.mjs's single-use rule
|
||||||
|
// without doing that work first.
|
||||||
export function extractLatestAssistantText(events) {
|
export function extractLatestAssistantText(events) {
|
||||||
let text = "";
|
let text = "";
|
||||||
for (const ev of events) {
|
for (const ev of events) {
|
||||||
|
|||||||
+117
-5
@@ -22,6 +22,8 @@
|
|||||||
* CLAUDE_MAX_CONCURRENT — max concurrent claude processes, -p/stream-json path (default: 8)
|
* CLAUDE_MAX_CONCURRENT — max concurrent claude processes, -p/stream-json path (default: 8)
|
||||||
* CLAUDE_MAX_QUEUE — max requests waiting for a -p slot before HTTP 429 (default: 16)
|
* CLAUDE_MAX_QUEUE — max requests waiting for a -p slot before HTTP 429 (default: 16)
|
||||||
* OCP_TUI_MAX_CONCURRENT — max concurrent interactive TUI turns, TUI-mode path (default: 2)
|
* OCP_TUI_MAX_CONCURRENT — max concurrent interactive TUI turns, TUI-mode path (default: 2)
|
||||||
|
* OCP_TUI_POOL_SIZE — pre-booted warm `claude` panes held for TUI-mode (default: 0 = off;
|
||||||
|
* max 4). Each is a live idle process; cuts ~3-4s per request.
|
||||||
* OCP_SPAWN_REAL_HOME — "1" forces the -p spawn to use the real HOME (disables the
|
* OCP_SPAWN_REAL_HOME — "1" forces the -p spawn to use the real HOME (disables the
|
||||||
* latency spawn-home isolation; default: isolated when a token exists)
|
* latency spawn-home isolation; default: isolated when a token exists)
|
||||||
* CLAUDE_BREAKER_THRESHOLD — failures in window before circuit opens (default: 6)
|
* CLAUDE_BREAKER_THRESHOLD — failures in window before circuit opens (default: 6)
|
||||||
@@ -32,7 +34,7 @@
|
|||||||
* CLAUDE_HEARTBEAT_INTERVAL — SSE heartbeat interval in ms on streaming path (default: 0 = disabled)
|
* CLAUDE_HEARTBEAT_INTERVAL — SSE heartbeat interval in ms on streaming path (default: 0 = disabled)
|
||||||
*/
|
*/
|
||||||
import { createServer } from "node:http";
|
import { createServer } from "node:http";
|
||||||
import { spawn, execFileSync } from "node:child_process";
|
import { spawn, execFileSync, spawnSync } from "node:child_process";
|
||||||
import { randomUUID, timingSafeEqual } from "node:crypto";
|
import { randomUUID, timingSafeEqual } from "node:crypto";
|
||||||
import { readFileSync, readdirSync, accessSync, existsSync, constants, chmodSync, statSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
|
import { readFileSync, readdirSync, accessSync, existsSync, constants, chmodSync, statSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
|
||||||
import { fileURLToPath } from "node:url";
|
import { fileURLToPath } from "node:url";
|
||||||
@@ -41,9 +43,10 @@ import { homedir } from "node:os";
|
|||||||
import { validateKey, recordUsage, getUsageByKey, getUsageTimeline, getRecentUsage, createKey, listKeys, revokeKey, closeDb, checkQuota, updateKeyQuota, getKeyQuota, findKey, cacheHash, getCachedResponse, setCachedResponse, clearCache, getCacheStats, hasCacheControl, singleflight, getInflightStats } from "./keys.mjs";
|
import { validateKey, recordUsage, getUsageByKey, getUsageTimeline, getRecentUsage, createKey, listKeys, revokeKey, closeDb, checkQuota, updateKeyQuota, getKeyQuota, findKey, cacheHash, getCachedResponse, setCachedResponse, clearCache, getCacheStats, hasCacheControl, singleflight, getInflightStats } from "./keys.mjs";
|
||||||
import { DEFAULT_PORT } from "./lib/constants.mjs";
|
import { DEFAULT_PORT } from "./lib/constants.mjs";
|
||||||
import { isLoopbackBind } from "./lib/net.mjs";
|
import { isLoopbackBind } from "./lib/net.mjs";
|
||||||
import { runTuiTurn, reapStaleTuiSessions, resolveTuiHome } from "./lib/tui/session.mjs";
|
import { runTuiTurn, reapStaleTuiSessions, resolveTuiHome, bootTuiPane, tuiPaneHealthy, poolPaneName, POOL_BOOT_MS } from "./lib/tui/session.mjs";
|
||||||
import { detectTuiUpstreamError } from "./lib/tui/transcript.mjs";
|
import { detectTuiUpstreamError } from "./lib/tui/transcript.mjs";
|
||||||
import { TuiSemaphore, SemaphoreAbortError, recordTuiEntrypoint, buildTuiHealthBlock } from "./lib/tui/semaphore.mjs";
|
import { TuiSemaphore, SemaphoreAbortError, recordTuiEntrypoint, buildTuiHealthBlock } from "./lib/tui/semaphore.mjs";
|
||||||
|
import { TuiPanePool, resolvePoolSize, POOL_MAX_SIZE } from "./lib/tui/pool.mjs";
|
||||||
import { createSerialMutex, createTtlCache, isTokenExpiring, orderLabelsLastGoodFirst } from "./lib/spawn-auth.mjs";
|
import { createSerialMutex, createTtlCache, isTokenExpiring, orderLabelsLastGoodFirst } from "./lib/spawn-auth.mjs";
|
||||||
|
|
||||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||||
@@ -351,6 +354,48 @@ const tuiStats = {
|
|||||||
entrypointMismatches: 0, // count of cli-expected-but-got-other turns
|
entrypointMismatches: 0, // count of cli-expected-but-got-other turns
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// ── Warm pane pool (docs/plans/2026-07-13-tui-latency #3) — opt-in; default OFF ─────────
|
||||||
|
// OCP_TUI_POOL_SIZE=0 (default) => tuiPool is null => runTuiTurn's cold-boot path is
|
||||||
|
// byte-for-byte unchanged. Set it to N (clamped to POOL_MAX_SIZE) to keep N pre-booted
|
||||||
|
// `claude` panes warm, each SINGLE-USE (see lib/tui/pool.mjs for why single-use is the
|
||||||
|
// load-bearing rule, and lib/tui/session.mjs for the POOL/REAPER INVARIANT).
|
||||||
|
//
|
||||||
|
// Default-off is deliberate on a stable production path: a warm pane is a LIVE idle
|
||||||
|
// `claude` process held whether or not a request ever arrives, so the operator must opt
|
||||||
|
// in to that standing cost. Measured saving when on (this host, Sonnet 4.6, --effort low):
|
||||||
|
// end-to-end p50 10.17 s (n=6, pool off) -> 6.00 s (n=12 warm hits), i.e. -41%.
|
||||||
|
// cli.js does NOT perform this operation (Class B, OCP-owned TUI spawn) — see ADR 0007.
|
||||||
|
const TUI_POOL_SIZE = TUI_MODE ? resolvePoolSize(process.env.OCP_TUI_POOL_SIZE) : 0;
|
||||||
|
const tuiPool = TUI_POOL_SIZE > 0
|
||||||
|
? new TuiPanePool({
|
||||||
|
size: TUI_POOL_SIZE,
|
||||||
|
// The POOL mints the pane's identity, not bootTuiPane: the tmux session exists the
|
||||||
|
// instant the boot starts, so the pool must be able to name (hence spare, hence kill)
|
||||||
|
// it before then. Name is derived from the session-id, so `tmux ls` correlates to the
|
||||||
|
// transcript file <HOME>/.claude/projects/*/<sessionId>.jsonl.
|
||||||
|
mintPane: () => {
|
||||||
|
const sessionId = randomUUID();
|
||||||
|
return { sessionId, name: poolPaneName(PORT, sessionId) };
|
||||||
|
},
|
||||||
|
bootPane: (model, ident) => bootTuiPane({
|
||||||
|
model,
|
||||||
|
claudeBin: CLAUDE,
|
||||||
|
home: TUI_HOME,
|
||||||
|
realHome: process.env.HOME,
|
||||||
|
cwd: TUI_CWD,
|
||||||
|
port: PORT,
|
||||||
|
entrypointMode: TUI_ENTRYPOINT,
|
||||||
|
sessionId: ident.sessionId,
|
||||||
|
name: ident.name,
|
||||||
|
requireReady: true, // a pane that never reached its input bar must not be enlisted
|
||||||
|
bootMs: POOL_BOOT_MS, // background pre-boot — no client is blocked, so be patient
|
||||||
|
}),
|
||||||
|
killPane: (name) => { try { spawnSync(process.env.OCP_TUI_TMUX_BIN || "tmux", ["kill-session", "-t", name]); } catch { /* already gone */ } },
|
||||||
|
paneHealthy: (name) => tuiPaneHealthy((args) => spawnSync(process.env.OCP_TUI_TMUX_BIN || "tmux", args, { encoding: "utf8" }), name),
|
||||||
|
log: (level, event, data) => logEvent(level, event, data),
|
||||||
|
})
|
||||||
|
: null;
|
||||||
|
|
||||||
// ── FIX ③ (latency): default-path (-p / stream-json) spawn-home isolation ──────────────
|
// ── FIX ③ (latency): default-path (-p / stream-json) spawn-home isolation ──────────────
|
||||||
// PROBLEM (measured, not theoretical): OCP's default spawn inherits the operator's real HOME
|
// PROBLEM (measured, not theoretical): OCP's default spawn inherits the operator's real HOME
|
||||||
// (loading the global ~/.claude — plugins, skills, hooks) and runs with cwd=~/ocp (loading the
|
// (loading the global ~/.claude — plugins, skills, hooks) and runs with cwd=~/ocp (loading the
|
||||||
@@ -770,19 +815,45 @@ const cacheCleanupInterval = setInterval(() => {
|
|||||||
// mechanism and the 15-min cadence makes the window negligible).
|
// mechanism and the 15-min cadence makes the window negligible).
|
||||||
// Gated on TUI_MODE — zero effect (no kill-server, no list-sessions) when TUI is off.
|
// Gated on TUI_MODE — zero effect (no kill-server, no list-sessions) when TUI is off.
|
||||||
// cli.js does NOT perform this operation (Class B, OCP-owned TUI spawn) — see ADR 0007.
|
// cli.js does NOT perform this operation (Class B, OCP-owned TUI spawn) — see ADR 0007.
|
||||||
|
//
|
||||||
|
// WARM POOL INTERACTION (the crux — see the POOL/REAPER INVARIANT in lib/tui/session.mjs).
|
||||||
|
// A warm pooled pane is one of OUR OWN ocp-tui-<port>-* sessions that is alive and idle BY
|
||||||
|
// DESIGN, and this sweep fires precisely when the instance is idle — i.e. exactly when the
|
||||||
|
// pool is full. Two things are therefore required, and both are done here:
|
||||||
|
// (a) DRAIN the pool BEFORE the sweep. Zombie reaping is possible ONLY via kill-server,
|
||||||
|
// and a live pooled pane suppresses kill-server (it is a live child of the tmux
|
||||||
|
// server). A permanently-full pool would otherwise permanently disable the very
|
||||||
|
// thing this tick exists to do. Draining costs one pane re-boot per tick (~1.2 s of
|
||||||
|
// background work every 15 min) and is invisible to callers: a request landing in the
|
||||||
|
// drain→refill gap simply MISSES the pool and takes today's cold path.
|
||||||
|
// (b) Pass the pool's live registry as `spare` anyway. After (a) it is empty, so this is
|
||||||
|
// belt-and-braces — it makes it impossible for THIS call site (or a future one) to
|
||||||
|
// kill a live pooled pane even if the drain were ever removed or reordered.
|
||||||
|
// RESIDUAL (unchanged in kind from the pre-pool code, and explicitly accepted there): a
|
||||||
|
// request arriving in the narrow window between the idle-check and kill-server has its pane
|
||||||
|
// torn down and fails cleanly via runTuiTurn's honesty gates. The drain widens that window
|
||||||
|
// by the cost of N kill-session calls (single-digit ms), not materially.
|
||||||
const TUI_REAP_INTERVAL_MS = 15 * 60 * 1000;
|
const TUI_REAP_INTERVAL_MS = 15 * 60 * 1000;
|
||||||
const tuiReapInterval = TUI_MODE ? setInterval(() => {
|
const tuiReapInterval = TUI_MODE ? setInterval(() => {
|
||||||
if (tuiSemaphore.inflight > 0 || tuiSemaphore.queued > 0) return; // a turn is live — defer
|
if (tuiSemaphore.inflight > 0 || tuiSemaphore.queued > 0) return; // a turn is live — defer
|
||||||
try {
|
try {
|
||||||
|
const drained = tuiPool ? tuiPool.drain() : 0;
|
||||||
// F7 fix: scope to THIS instance's own port; a sibling ocp-tui-<otherPort>-* session
|
// F7 fix: scope to THIS instance's own port; a sibling ocp-tui-<otherPort>-* session
|
||||||
// (a second OCP instance on the same host) is treated as foreign, same as olp-tui-*.
|
// (a second OCP instance on the same host) is treated as foreign, same as olp-tui-*.
|
||||||
// includeLegacy is NOT set here — see reapStaleTuiSessions' comment: the periodic sweep
|
// includeLegacy is NOT set here — see reapStaleTuiSessions' comment: the periodic sweep
|
||||||
// conservatively treats any lingering bare-prefix legacy session as foreign so it can
|
// conservatively treats any lingering bare-prefix legacy session as foreign so it can
|
||||||
// never trigger kill-server on a steady-state tick; only the one-time boot reap below
|
// never trigger kill-server on a steady-state tick; only the one-time boot reap below
|
||||||
// claims legacy-shaped zombies.
|
// claims legacy-shaped zombies.
|
||||||
const n = reapStaleTuiSessions({ port: PORT });
|
const n = reapStaleTuiSessions({ port: PORT, spare: tuiPool ? tuiPool.liveNames() : null });
|
||||||
if (n) logEvent("info", "tui_reaped_stale_sessions", { count: n, trigger: "periodic" });
|
if (n || drained) {
|
||||||
|
logEvent("info", "tui_reaped_stale_sessions", { count: n, poolDrained: drained, trigger: "periodic" });
|
||||||
|
}
|
||||||
} catch (e) { logEvent("error", "tui_periodic_reap_failed", { error: e.message }); }
|
} catch (e) { logEvent("error", "tui_periodic_reap_failed", { error: e.message }); }
|
||||||
|
finally {
|
||||||
|
// Refill in the background regardless of how the sweep went — a throw mid-sweep must not
|
||||||
|
// leave the pool permanently paused (it would silently degrade to the cold path forever).
|
||||||
|
if (tuiPool) { try { tuiPool.resume(); } catch { /* best effort */ } }
|
||||||
|
}
|
||||||
}, TUI_REAP_INTERVAL_MS) : null;
|
}, TUI_REAP_INTERVAL_MS) : null;
|
||||||
if (tuiReapInterval && typeof tuiReapInterval.unref === "function") tuiReapInterval.unref();
|
if (tuiReapInterval && typeof tuiReapInterval.unref === "function") tuiReapInterval.unref();
|
||||||
|
|
||||||
@@ -1323,6 +1394,15 @@ async function callClaudeTui(model, messages, _conversationId, _keyName, res) {
|
|||||||
// different port never collides with this instance's reap/kill-server logic.
|
// different port never collides with this instance's reap/kill-server logic.
|
||||||
wallclockMs: TUI_WALLCLOCK_MS,
|
wallclockMs: TUI_WALLCLOCK_MS,
|
||||||
entrypointMode: TUI_ENTRYPOINT,
|
entrypointMode: TUI_ENTRYPOINT,
|
||||||
|
// Warm pane pool (null unless OCP_TUI_POOL_SIZE > 0 → today's cold path exactly).
|
||||||
|
// A pooled pane is single-use: runTuiTurn kills it in its finally like any other.
|
||||||
|
pool: tuiPool,
|
||||||
|
// Only observe when the pool is ON — with it off (the default) no new log line is
|
||||||
|
// emitted, so the disabled path stays byte-for-byte today's, logs included.
|
||||||
|
onPane: tuiPool
|
||||||
|
? ({ warm }) => logEvent("info", warm ? "tui_pool_hit" : "tui_pool_miss",
|
||||||
|
{ model: cliModel, warmRemaining: tuiPool.warm })
|
||||||
|
: null,
|
||||||
});
|
});
|
||||||
// ── Honesty gates (issue #133) ─ run BEFORE recordModelSuccess / cache write-back.
|
// ── Honesty gates (issue #133) ─ run BEFORE recordModelSuccess / cache write-back.
|
||||||
// A throw here propagates to the catch below (recordModelError + reject), so the
|
// A throw here propagates to the catch below (recordModelError + reject), so the
|
||||||
@@ -2558,9 +2638,12 @@ const server = createServer(async (req, res) => {
|
|||||||
// still appears with enabled:false (cheap, harmless) so the shape is stable.
|
// still appears with enabled:false (cheap, harmless) so the shape is stable.
|
||||||
// entrypointMismatches/lastEntrypoint exist so an operator can poll /health to catch a
|
// entrypointMismatches/lastEntrypoint exist so an operator can poll /health to catch a
|
||||||
// silent metered-pool drift (the audit's top risk after the 6/15 billing flip).
|
// silent metered-pool drift (the audit's top risk after the 6/15 billing flip).
|
||||||
|
// `pool` is a NEW nested field inside the (already additive) tui block: null when the
|
||||||
|
// warm pool is off (the default), so the disabled shape is unchanged apart from one
|
||||||
|
// explicit null. Lets the operator confirm hit rate + standing process cost.
|
||||||
tui: buildTuiHealthBlock(
|
tui: buildTuiHealthBlock(
|
||||||
{ enabled: TUI_MODE, entrypointMode: TUI_ENTRYPOINT, maxConcurrent: TUI_MAX_CONCURRENT },
|
{ enabled: TUI_MODE, entrypointMode: TUI_ENTRYPOINT, maxConcurrent: TUI_MAX_CONCURRENT },
|
||||||
tuiStats, tuiSemaphore,
|
tuiStats, tuiSemaphore, tuiPool,
|
||||||
),
|
),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -2797,6 +2880,27 @@ function gracefulShutdown(signal) {
|
|||||||
if (tuiReapInterval) clearInterval(tuiReapInterval);
|
if (tuiReapInterval) clearInterval(tuiReapInterval);
|
||||||
closeDb();
|
closeDb();
|
||||||
|
|
||||||
|
// 2b. Drain the warm pane pool. A pooled `claude` is a child of the tmux SERVER, not of
|
||||||
|
// this node process, so it is NOT in activeProcesses and step 3 below cannot reach it —
|
||||||
|
// without this explicit drain every warm pane would outlive OCP as an orphan (and the
|
||||||
|
// pool's in-memory registry dies with the process, so nothing would remember it owned them).
|
||||||
|
//
|
||||||
|
// drain() kills the pane that is currently BOOTING too, and it does so SYNCHRONOUSLY. That
|
||||||
|
// is required, not incidental: step 4 below calls process.exit(0) in THIS SAME TICK whenever
|
||||||
|
// activeProcesses is empty — which on a TUI host it always is — so any cleanup a boot
|
||||||
|
// deferred to a .then()/.catch() would simply never run. (That was a real bug: the pool used
|
||||||
|
// to track in-flight boots as a count, could not name the booting session, and orphaned a
|
||||||
|
// live authenticated `claude` on every shutdown that landed mid-boot.)
|
||||||
|
//
|
||||||
|
// Orphans that survive anyway (SIGKILL, power loss) are still caught by the next instance's
|
||||||
|
// boot reap — this makes the graceful path clean, it is not the only safety net.
|
||||||
|
if (tuiPool) {
|
||||||
|
try {
|
||||||
|
const drained = tuiPool.drain();
|
||||||
|
if (drained) logEvent("info", "tui_pool_drained", { count: drained, trigger: "shutdown" });
|
||||||
|
} catch (e) { logEvent("error", "tui_pool_drain_failed", { error: e.message }); }
|
||||||
|
}
|
||||||
|
|
||||||
// 3. Kill all active child processes
|
// 3. Kill all active child processes
|
||||||
for (const proc of activeProcesses) {
|
for (const proc of activeProcesses) {
|
||||||
try { proc.kill("SIGTERM"); } catch {}
|
try { proc.kill("SIGTERM"); } catch {}
|
||||||
@@ -2866,11 +2970,19 @@ server.listen(PORT, BIND_ADDRESS, () => {
|
|||||||
? (TUI_HOME === process.env.HOME ? "env-token (real home — unset OCP_TUI_HOME for credential isolation)" : "env-token (credential-isolated home — no credentials.json)")
|
? (TUI_HOME === process.env.HOME ? "env-token (real home — unset OCP_TUI_HOME for credential isolation)" : "env-token (credential-isolated home — no credentials.json)")
|
||||||
: "credentials.json (no CLAUDE_CODE_OAUTH_TOKEN — see Troubleshooting #401)";
|
: "credentials.json (no CLAUDE_CODE_OAUTH_TOKEN — see Troubleshooting #401)";
|
||||||
console.log(` TUI-mode: ON home=${TUI_HOME} cwd=${TUI_CWD} auth=${tuiAuth} wallclock=${TUI_WALLCLOCK_MS}ms maxConcurrent=${TUI_MAX_CONCURRENT}`);
|
console.log(` TUI-mode: ON home=${TUI_HOME} cwd=${TUI_CWD} auth=${tuiAuth} wallclock=${TUI_WALLCLOCK_MS}ms maxConcurrent=${TUI_MAX_CONCURRENT}`);
|
||||||
|
console.log(TUI_POOL_SIZE > 0
|
||||||
|
? ` TUI warm pool: ON size=${TUI_POOL_SIZE} — ${TUI_POOL_SIZE} idle \`claude\` process(es) held warm; first request per model is still a cold MISS`
|
||||||
|
: ` TUI warm pool: OFF (set OCP_TUI_POOL_SIZE=1..${POOL_MAX_SIZE} to pre-boot panes and cut ~3-4s per request)`);
|
||||||
try {
|
try {
|
||||||
// F7 fix: scope to THIS instance's own port (see reapStaleTuiSessions). includeLegacy:
|
// F7 fix: scope to THIS instance's own port (see reapStaleTuiSessions). includeLegacy:
|
||||||
// true ONLY here — the one-time boot reap is the designated point to claim orphaned
|
// true ONLY here — the one-time boot reap is the designated point to claim orphaned
|
||||||
// bare-prefix ("ocp-tui-<uuid8>") zombie sessions left by a PRE-fix process generation
|
// bare-prefix ("ocp-tui-<uuid8>") zombie sessions left by a PRE-fix process generation
|
||||||
// of this same instance (no live post-fix instance ever creates that shape again).
|
// of this same instance (no live post-fix instance ever creates that shape again).
|
||||||
|
// No `spare`: the warm pool is EMPTY at boot (there is no boot-time pre-warm — the pool
|
||||||
|
// learns its model from the first request), so this reap has no live pane to protect and
|
||||||
|
// it is exactly what SHOULD claim any ocp-tui-<port>-p* pool orphans left by a previous
|
||||||
|
// process generation of this instance (POOL/REAPER INVARIANT property 2). If a future
|
||||||
|
// change ever pre-warms at boot, this call MUST start passing tuiPool.liveNames().
|
||||||
const n = reapStaleTuiSessions({ port: PORT, includeLegacy: true });
|
const n = reapStaleTuiSessions({ port: PORT, includeLegacy: true });
|
||||||
if (n) logEvent("info", "tui_reaped_stale_sessions", { count: n });
|
if (n) logEvent("info", "tui_reaped_stale_sessions", { count: n });
|
||||||
} catch {}
|
} catch {}
|
||||||
|
|||||||
+514
-3
@@ -23,9 +23,30 @@ process.env.HOME = homedir(); // ensure consistent
|
|||||||
let passed = 0;
|
let passed = 0;
|
||||||
let failed = 0;
|
let failed = 0;
|
||||||
|
|
||||||
|
// Pending promises from tests declared `async` but registered through the SYNC `test()` helper.
|
||||||
|
// 44 tests in this file are written that way. Before this, `test()` called fn(), got a promise back,
|
||||||
|
// and immediately printed ✓ and incremented `passed` — WITHOUT AWAITING IT. So for every async test:
|
||||||
|
// - ✓ meant "did not throw synchronously", NOT "passed";
|
||||||
|
// - a failed assertion escaped as an unhandled rejection, which crashes the process (CI still goes
|
||||||
|
// red on the non-zero exit) but is NOT counted, so the summary could print "N passed, 0 failed"
|
||||||
|
// and be wrong.
|
||||||
|
// The suite's own headline number was therefore not evidence for any async test — including the
|
||||||
|
// regression guards in this PR. Collected here and awaited before the summary prints.
|
||||||
|
const pendingAsync = [];
|
||||||
|
|
||||||
function test(name, fn) {
|
function test(name, fn) {
|
||||||
try {
|
try {
|
||||||
fn();
|
const r = fn();
|
||||||
|
if (r && typeof r.then === "function") {
|
||||||
|
// Async body: settle it before counting. Do NOT print ✓ yet.
|
||||||
|
pendingAsync.push(
|
||||||
|
r.then(
|
||||||
|
() => { passed++; console.log(` ✓ ${name}`); },
|
||||||
|
(e) => { failed++; console.log(` ✗ ${name}: ${e.message}`); },
|
||||||
|
),
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
passed++;
|
passed++;
|
||||||
console.log(` ✓ ${name}`);
|
console.log(` ✓ ${name}`);
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
@@ -2027,6 +2048,489 @@ test("reaper with includeLegacy=true still spares a sibling instance's port-scop
|
|||||||
assert.ok(!calls.includes("kill-server"), "sibling instance's live session still blocks kill-server");
|
assert.ok(!calls.includes("kill-server"), "sibling instance's live session still blocks kill-server");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ── TUI warm pane pool (docs/plans/2026-07-13-tui-latency #3) ────────────
|
||||||
|
import { TuiPanePool, resolvePoolSize, POOL_MAX_SIZE, POOL_MAX_AGE_MS } from "./lib/tui/pool.mjs";
|
||||||
|
import { poolPaneName as poolName } from "./lib/tui/session.mjs";
|
||||||
|
|
||||||
|
// A pool wired to fakes: no tmux, no claude. bootPane resolves on the microtask queue; use
|
||||||
|
// `await settle()` after a refill() to let the SERIALIZED boot chain run to target.
|
||||||
|
// `live` models the real tmux server: bootTuiPane creates the session SYNCHRONOUSLY and only
|
||||||
|
// THEN waits (up to POOL_BOOT_MS) for the input bar, so the fake boot registers the session
|
||||||
|
// immediately and only afterwards resolves. `opts.hold` keeps a boot in that mid-flight window
|
||||||
|
// so tests can act on a pane that is live-but-not-yet-warm — the state that hid two bugs.
|
||||||
|
function makeFakePool(opts = {}) {
|
||||||
|
const killed = [];
|
||||||
|
const booted = [];
|
||||||
|
const live = new Set(); // "tmux sessions" that currently exist
|
||||||
|
let seq = 0;
|
||||||
|
let clock = 1_000_000;
|
||||||
|
const healthy = new Set();
|
||||||
|
// FIFO gate queue — one entry per in-flight held boot. A single `release` slot would be
|
||||||
|
// OVERWRITTEN by a later boot, so releasing "the first boot" would silently release the
|
||||||
|
// second instead (and mask the stale-settle bug this harness exists to test).
|
||||||
|
const gates = [];
|
||||||
|
const pool = new TuiPanePool({
|
||||||
|
size: opts.size ?? 2,
|
||||||
|
maxAgeMs: opts.maxAgeMs ?? POOL_MAX_AGE_MS,
|
||||||
|
now: () => clock,
|
||||||
|
mintPane: () => {
|
||||||
|
const n = ++seq;
|
||||||
|
return { sessionId: `sid-${n}`, name: `ocp-tui-3456-p${String(n).padStart(8, "0")}` };
|
||||||
|
},
|
||||||
|
bootPane: async (model, { sessionId, name }) => {
|
||||||
|
live.add(name); // session exists NOW
|
||||||
|
booted.push({ name, model });
|
||||||
|
if (opts.hold) {
|
||||||
|
await new Promise((r) => gates.push(r)); // ...stuck waiting for readiness
|
||||||
|
}
|
||||||
|
if (opts.bootThrows) { live.delete(name); throw new Error("boom"); }
|
||||||
|
// A pane whose session was killed while booting can never become ready — exactly what
|
||||||
|
// the real bootTuiPane does (it throws tui_pane_not_ready).
|
||||||
|
if (!live.has(name)) throw new Error("tui_pane_not_ready");
|
||||||
|
healthy.add(name);
|
||||||
|
return { name, sessionId, model, bootedAt: clock };
|
||||||
|
},
|
||||||
|
killPane: (name) => { killed.push(name); healthy.delete(name); live.delete(name); },
|
||||||
|
paneHealthy: (name) => healthy.has(name),
|
||||||
|
});
|
||||||
|
return {
|
||||||
|
pool, killed, booted, healthy, live,
|
||||||
|
releaseBoot: () => { const r = gates.shift(); if (r) r(); }, // release the OLDEST held boot
|
||||||
|
advance: (ms) => { clock += ms; }, at: () => clock,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
const tick = () => new Promise((r) => setImmediate(r));
|
||||||
|
// Refills are SERIALIZED (one boot at a time, re-kicked on success), so settling the pool
|
||||||
|
// takes a chain of microtask turns, not one. 40 is far more than POOL_MAX_SIZE needs.
|
||||||
|
const settle = async () => { for (let i = 0; i < 40; i++) await tick(); };
|
||||||
|
|
||||||
|
console.log("\nTUI warm pane pool (acquire / miss / refill / TTL / reaper exemption):");
|
||||||
|
|
||||||
|
test("resolvePoolSize: default/garbage/negative disable the pool; size is clamped to POOL_MAX_SIZE", () => {
|
||||||
|
assert.equal(resolvePoolSize(undefined), 0, "unset => off (byte-for-byte today's cold path)");
|
||||||
|
assert.equal(resolvePoolSize("0"), 0);
|
||||||
|
assert.equal(resolvePoolSize("-3"), 0);
|
||||||
|
assert.equal(resolvePoolSize("banana"), 0, "garbage disables rather than guessing a size");
|
||||||
|
assert.equal(resolvePoolSize("2"), 2);
|
||||||
|
assert.equal(resolvePoolSize("99"), POOL_MAX_SIZE, "clamped — never boot an unbounded number of idle claudes");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("pool size 0 is inert: acquire always misses and refill never boots", async () => {
|
||||||
|
const { pool, booted } = makeFakePool({ size: 0 });
|
||||||
|
assert.equal(pool.enabled, false);
|
||||||
|
assert.equal(pool.acquire("m1"), null, "disabled pool always MISSES → caller cold-boots");
|
||||||
|
pool.refill();
|
||||||
|
await settle();
|
||||||
|
assert.equal(booted.length, 0, "a disabled pool must never spawn a process");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("acquire MISSES on an empty pool, and the miss refills for the requested model", async () => {
|
||||||
|
const { pool, booted } = makeFakePool({ size: 2 });
|
||||||
|
assert.equal(pool.acquire("sonnet"), null, "first request is always a MISS (no boot-time pre-warm)");
|
||||||
|
assert.equal(pool.misses, 1);
|
||||||
|
pool.refill();
|
||||||
|
await settle();
|
||||||
|
assert.equal(pool.warm, 2, "refilled to target");
|
||||||
|
assert.deepEqual(booted.map((b) => b.model), ["sonnet", "sonnet"], "warmed for the model that missed");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("acquire HITS a warm pane, hands it out ONCE, and never returns it (single-use)", async () => {
|
||||||
|
const { pool } = makeFakePool({ size: 2 });
|
||||||
|
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||||
|
assert.equal(pool.warm, 2);
|
||||||
|
|
||||||
|
const a = pool.acquire("sonnet");
|
||||||
|
assert.ok(a && a.name && a.sessionId, "warm pane handed out");
|
||||||
|
assert.equal(pool.hits, 1);
|
||||||
|
assert.equal(pool.warm, 1, "the pane LEAVES the registry when acquired");
|
||||||
|
|
||||||
|
const b = pool.acquire("sonnet");
|
||||||
|
assert.notEqual(b.name, a.name, "a pane is NEVER handed out twice — single-use");
|
||||||
|
assert.notEqual(b.sessionId, a.sessionId, "each pane carries its OWN fresh session-id (transcript.mjs scoping)");
|
||||||
|
assert.equal(pool.warm, 0);
|
||||||
|
assert.equal(pool.acquire("sonnet"), null, "exhausted pool MISSES rather than reusing a pane");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("refill is bounded: never more than `size` panes, and concurrent refills do not overshoot", async () => {
|
||||||
|
const { pool, booted } = makeFakePool({ size: 2 });
|
||||||
|
pool.acquire("sonnet");
|
||||||
|
pool.refill(); pool.refill(); pool.refill(); // hammer it
|
||||||
|
await settle();
|
||||||
|
assert.equal(pool.warm, 2, "still exactly `size` warm panes");
|
||||||
|
assert.equal(booted.length, 2, "the _booting guard prevented duplicate boots");
|
||||||
|
});
|
||||||
|
|
||||||
|
// Live finding at size=2: two cold `claude` boots racing an in-flight turn made a refill
|
||||||
|
// overrun even the generous pool readiness cap. Boots are therefore SERIALIZED.
|
||||||
|
test("refill boots panes ONE AT A TIME (never two claude cold-boots racing each other)", async () => {
|
||||||
|
let concurrent = 0, peak = 0;
|
||||||
|
let seq = 0;
|
||||||
|
const pool = new TuiPanePool({
|
||||||
|
size: 3,
|
||||||
|
mintPane: () => { const n = ++seq; return { sessionId: `s${n}`, name: `p${n}` }; },
|
||||||
|
bootPane: async (model, { sessionId, name }) => {
|
||||||
|
concurrent++; peak = Math.max(peak, concurrent);
|
||||||
|
await new Promise((r) => setImmediate(r)); // simulate boot latency
|
||||||
|
concurrent--;
|
||||||
|
return { name, sessionId, model, bootedAt: Date.now() };
|
||||||
|
},
|
||||||
|
killPane: () => {},
|
||||||
|
paneHealthy: () => true,
|
||||||
|
});
|
||||||
|
pool.acquire("sonnet");
|
||||||
|
pool.refill();
|
||||||
|
await settle();
|
||||||
|
assert.equal(pool.warm, 3, "chain still reaches the target size");
|
||||||
|
assert.equal(peak, 1, "at most ONE boot in flight at any moment");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a FAILED boot does not re-kick the chain (backoff — a broken claude must not spin)", async () => {
|
||||||
|
const { pool, booted } = makeFakePool({ size: 3, bootThrows: true });
|
||||||
|
pool.acquire("sonnet");
|
||||||
|
pool.refill();
|
||||||
|
await settle();
|
||||||
|
assert.equal(pool.bootFailures, 1, "counted as a genuine failure (nobody cancelled it)");
|
||||||
|
assert.equal(booted.length, 1, "exactly ONE attempt — a failure stops the chain, it does not respawn forever");
|
||||||
|
assert.equal(pool.warm, 0);
|
||||||
|
assert.equal(pool.booting, 0, "and the booting slot is released, so the next trigger can retry");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("acquire drops an UNHEALTHY warm pane (kills it) and falls through to a MISS", async () => {
|
||||||
|
const { pool, killed, healthy, booted } = makeFakePool({ size: 1 });
|
||||||
|
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||||
|
const dead = booted[0].name;
|
||||||
|
healthy.delete(dead); // pane died / stopped being input-ready while idle
|
||||||
|
|
||||||
|
assert.equal(pool.acquire("sonnet"), null, "a dead pane must MISS, never hang a turn");
|
||||||
|
assert.ok(killed.includes(dead), "the dead pane is killed, not leaked");
|
||||||
|
assert.equal(pool.misses, 2);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("acquire drops an EXPIRED warm pane (older than maxAgeMs)", async () => {
|
||||||
|
const { pool, killed, booted, advance } = makeFakePool({ size: 1, maxAgeMs: 60_000 });
|
||||||
|
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||||
|
advance(60_001);
|
||||||
|
assert.equal(pool.acquire("sonnet"), null, "a pane past its TTL is not handed out");
|
||||||
|
assert.ok(killed.includes(booted[0].name), "expired pane is killed");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a model switch drops the wrong-model panes and retargets the pool (--model is fixed at spawn)", async () => {
|
||||||
|
const { pool, killed, booted } = makeFakePool({ size: 2 });
|
||||||
|
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||||
|
const sonnetPanes = booted.map((b) => b.name);
|
||||||
|
|
||||||
|
assert.equal(pool.acquire("opus"), null, "different model => MISS (a sonnet pane cannot serve opus)");
|
||||||
|
assert.equal(pool.warm, 0, "sonnet panes dropped");
|
||||||
|
for (const p of sonnetPanes) assert.ok(killed.includes(p), "wrong-model pane killed, not leaked");
|
||||||
|
assert.equal(pool.warmModel, "opus", "pool retargeted to the model actually being asked for");
|
||||||
|
|
||||||
|
pool.refill(); await settle();
|
||||||
|
assert.deepEqual(booted.slice(2).map((b) => b.model), ["opus", "opus"], "refilled for the NEW model");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a boot that resolves AFTER a drain kills its own pane instead of enlisting it", async () => {
|
||||||
|
const { pool, live } = makeFakePool({ size: 1 });
|
||||||
|
pool.acquire("sonnet");
|
||||||
|
pool.refill(); // boot is in flight...
|
||||||
|
pool.drain(); // ...pool drained before it resolves (shutdown / reap sweep)
|
||||||
|
await settle();
|
||||||
|
assert.equal(pool.warm, 0, "the late pane must NOT be enlisted into a drained pool");
|
||||||
|
// Assert LIVENESS, not the kill-call COUNT. This assertion used to read
|
||||||
|
// `assert.equal(killed.length, 1)` — and it PASSED while the orphan it is named after was
|
||||||
|
// actually present: _cancelBooting kills BY NAME, and at drain time the tmux session does not
|
||||||
|
// exist yet (bootPane runs on a microtask), so that kill is a NO-OP which still increments the
|
||||||
|
// counter. "kill was called once" and "a live session is orphaned" were both true at the same
|
||||||
|
// time. The only honest question is whether the session is dead.
|
||||||
|
assert.equal(live.size, 0, "it kills itself — no orphan process left behind");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("bootPane failure is counted, never thrown into the request path, and does not wedge refill", async () => {
|
||||||
|
const { pool } = makeFakePool({ size: 1, bootThrows: true });
|
||||||
|
pool.acquire("sonnet");
|
||||||
|
pool.refill();
|
||||||
|
await settle();
|
||||||
|
assert.equal(pool.warm, 0);
|
||||||
|
assert.equal(pool.bootFailures, 1);
|
||||||
|
assert.equal(pool.booting, 0, "the _booting counter is released on failure (else refill wedges forever)");
|
||||||
|
assert.equal(pool.acquire("sonnet"), null, "and the caller just MISSES → cold path");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("drain kills every warm pane and pauses refills; resume restarts them", async () => {
|
||||||
|
const { pool, killed } = makeFakePool({ size: 2 });
|
||||||
|
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||||
|
assert.equal(pool.warm, 2);
|
||||||
|
|
||||||
|
assert.equal(pool.drain(), 2, "drain reports how many it killed");
|
||||||
|
assert.equal(pool.warm, 0);
|
||||||
|
assert.equal(killed.length, 2, "both panes killed — none outlive the drain");
|
||||||
|
|
||||||
|
pool.refill(); await settle();
|
||||||
|
assert.equal(pool.warm, 0, "refill is a NO-OP while drained (paused)");
|
||||||
|
|
||||||
|
pool.resume(); await settle();
|
||||||
|
assert.equal(pool.warm, 2, "resume refills");
|
||||||
|
});
|
||||||
|
|
||||||
|
// ── The crux: pool ↔ reaper coexistence (POOL/REAPER INVARIANT, lib/tui/session.mjs) ──
|
||||||
|
console.log("\nTUI warm pool ↔ session reaper coexistence:");
|
||||||
|
|
||||||
|
test("INVARIANT 1: a LIVE pooled pane is NEVER reaped (it is in the spare set)", () => {
|
||||||
|
const killed = [];
|
||||||
|
const live = "ocp-tui-3456-pdeadbeef";
|
||||||
|
const fakeTmux = (args) => {
|
||||||
|
if (args[0] === "list-sessions") return { status: 0, stdout: `${live}\nocp-tui-3456-aaaa\n` };
|
||||||
|
if (args[0] === "kill-session") { killed.push(args[args.indexOf("-t") + 1]); return { status: 0 }; }
|
||||||
|
return { status: 0, stdout: "" };
|
||||||
|
};
|
||||||
|
const n = reapStaleTuiSessions({ tmux: fakeTmux, port: 3456, spare: new Set([live]) });
|
||||||
|
assert.equal(n, 1, "only the stale turn session was reaped");
|
||||||
|
assert.ok(!killed.includes(live), "the live warm pane must survive the sweep");
|
||||||
|
assert.ok(killed.includes("ocp-tui-3456-aaaa"), "a genuinely stale own session is still reaped");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("INVARIANT 2: an ORPHANED pooled pane (pool-shaped but NOT in the spare set) IS reaped", () => {
|
||||||
|
const killed = [];
|
||||||
|
// ocp-tui-3456-porphan01 LOOKS pooled but the live registry does not claim it — e.g. left
|
||||||
|
// behind by a previous process generation, whose in-memory registry died with it.
|
||||||
|
const fakeTmux = (args) => {
|
||||||
|
if (args[0] === "list-sessions") return { status: 0, stdout: "ocp-tui-3456-porphan01\nocp-tui-3456-plive0001\n" };
|
||||||
|
if (args[0] === "kill-session") { killed.push(args[args.indexOf("-t") + 1]); return { status: 0 }; }
|
||||||
|
return { status: 0, stdout: "" };
|
||||||
|
};
|
||||||
|
const n = reapStaleTuiSessions({ tmux: fakeTmux, port: 3456, spare: new Set(["ocp-tui-3456-plive0001"]) });
|
||||||
|
assert.equal(n, 1);
|
||||||
|
assert.deepEqual(killed, ["ocp-tui-3456-porphan01"], "exemption is by EXACT NAME, never by name shape");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("INVARIANT 2b: with NO spare set (the pre-pool call shape) pool-shaped panes are reaped — fail-safe", () => {
|
||||||
|
const killed = [];
|
||||||
|
const fakeTmux = (args) => {
|
||||||
|
if (args[0] === "list-sessions") return { status: 0, stdout: "ocp-tui-3456-pdeadbeef\n" };
|
||||||
|
if (args[0] === "kill-session") { killed.push(args[args.indexOf("-t") + 1]); return { status: 0 }; }
|
||||||
|
return { status: 0, stdout: "" };
|
||||||
|
};
|
||||||
|
const n = reapStaleTuiSessions({ tmux: fakeTmux, port: 3456 });
|
||||||
|
assert.equal(n, 1, "omitting `spare` reaps MORE, never less — forgetting it can't leak panes");
|
||||||
|
assert.deepEqual(killed, ["ocp-tui-3456-pdeadbeef"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("INVARIANT 3: kill-server is SUPPRESSED while a live pooled pane is spared", () => {
|
||||||
|
const calls = [];
|
||||||
|
const live = "ocp-tui-3456-plive0001";
|
||||||
|
const fakeTmux = (args) => {
|
||||||
|
calls.push(args.join(" "));
|
||||||
|
if (args[0] === "list-sessions") return { status: 0, stdout: `${live}\nocp-tui-3456-aaaa\n` };
|
||||||
|
return { status: 0, stdout: "" };
|
||||||
|
};
|
||||||
|
reapStaleTuiSessions({ tmux: fakeTmux, port: 3456, spare: new Set([live]) });
|
||||||
|
assert.ok(!calls.includes("kill-server"), "kill-server would kill the live pane (a child of the tmux server)");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("INVARIANT 3b: after a DRAIN the spare set is empty, so kill-server fires again (zombie reaping preserved)", async () => {
|
||||||
|
// This is the whole reason server.mjs drains BEFORE the periodic sweep: a permanently-full
|
||||||
|
// pool would otherwise permanently suppress the only mechanism that reaps defunct claudes.
|
||||||
|
const { pool } = makeFakePool({ size: 2 });
|
||||||
|
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||||
|
assert.equal(pool.liveNames().size, 2, "pool is full → the sweep would be suppressed");
|
||||||
|
|
||||||
|
pool.drain();
|
||||||
|
assert.equal(pool.liveNames().size, 0, "drain empties the live registry");
|
||||||
|
|
||||||
|
const calls = [];
|
||||||
|
const fakeTmux = (args) => {
|
||||||
|
calls.push(args.join(" "));
|
||||||
|
if (args[0] === "list-sessions") return { status: 0, stdout: "ocp-tui-3456-aaaa\n" };
|
||||||
|
return { status: 0, stdout: "" };
|
||||||
|
};
|
||||||
|
reapStaleTuiSessions({ tmux: fakeTmux, port: 3456, spare: pool.liveNames() });
|
||||||
|
assert.ok(calls.includes("kill-server"), "kill-server fires post-drain — defunct zombies still get reaped");
|
||||||
|
});
|
||||||
|
|
||||||
|
// ── MID-BOOT: the state that hid M1a + M1b ────────────────────────────────────────────
|
||||||
|
// bootTuiPane creates the tmux session SYNCHRONOUSLY, then waits up to POOL_BOOT_MS (20s)
|
||||||
|
// for the input bar. So a pooled session can be LIVE for ~20s before its boot resolves.
|
||||||
|
// Every reaper test above uses a pool that is either full or drained — never mid-boot.
|
||||||
|
// That gap is exactly why both bugs shipped past the first round of tests.
|
||||||
|
|
||||||
|
test("M1a: a reap tick during an IN-FLIGHT BOOT must not orphan-kill the booting pane", async () => {
|
||||||
|
const { pool, live } = makeFakePool({ size: 1, hold: true });
|
||||||
|
pool.acquire("sonnet");
|
||||||
|
pool.refill();
|
||||||
|
await tick(); // boot started; session live; NOT yet warm
|
||||||
|
assert.equal(pool.warm, 0, "not warm yet");
|
||||||
|
assert.equal(pool.booting, 1, "a boot is in flight");
|
||||||
|
assert.equal(live.size, 1, "...and its tmux session ALREADY EXISTS");
|
||||||
|
|
||||||
|
const bootingName = [...live][0];
|
||||||
|
assert.ok(pool.liveNames().has(bootingName),
|
||||||
|
"REGRESSION GUARD: the booting pane MUST be nameable, or the sweep cannot spare it " +
|
||||||
|
"(the pool used to track in-flight boots as a COUNT and this was empty)");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("M1a: the reap tick's drain kills the booting pane, and resume() starts a FRESH boot", async () => {
|
||||||
|
const warns = [];
|
||||||
|
const { pool, live, releaseBoot } = makeFakePool({ size: 1, hold: true });
|
||||||
|
pool._log = (lvl, ev) => { if (lvl === "warn") warns.push(ev); };
|
||||||
|
pool.acquire("sonnet");
|
||||||
|
pool.refill();
|
||||||
|
await tick();
|
||||||
|
const first = [...live][0];
|
||||||
|
|
||||||
|
// The reap tick, as server.mjs runs it: drain -> reap -> resume.
|
||||||
|
const drained = pool.drain();
|
||||||
|
assert.equal(drained, 1, "drain accounts for the booting pane");
|
||||||
|
assert.equal(live.size, 0, "its tmux session is killed — kill-server can now flush zombies");
|
||||||
|
assert.equal(pool.liveNames().size, 0, "nothing left to spare, so kill-server is not suppressed");
|
||||||
|
|
||||||
|
pool.resume();
|
||||||
|
await tick();
|
||||||
|
assert.equal(pool.booting, 1, "resume() started a FRESH boot — the pool is not left empty with nothing scheduled");
|
||||||
|
assert.notEqual([...live][0], first, "and it is a NEW pane, not the killed one");
|
||||||
|
|
||||||
|
// Now let the ORIGINAL (cancelled) boot settle. It rejects with tui_pane_not_ready because
|
||||||
|
// we killed its session — but that is OUR doing, not a fault.
|
||||||
|
releaseBoot();
|
||||||
|
await settle();
|
||||||
|
assert.equal(pool.bootFailures, 0,
|
||||||
|
"a cancelled boot must NOT be counted as a bootFailure — that is the WARN operators alert on");
|
||||||
|
assert.deepEqual(warns, [], "and it must not log tui_pool_boot_failed for a healthy drain");
|
||||||
|
assert.equal(pool.cancelled, 1, "it is counted as a cancellation instead (counted exactly once)");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("M1b: shutdown drain kills the booting pane SYNCHRONOUSLY — no orphaned claude", async () => {
|
||||||
|
const { pool, live } = makeFakePool({ size: 1, hold: true });
|
||||||
|
pool.acquire("sonnet");
|
||||||
|
pool.refill();
|
||||||
|
await tick();
|
||||||
|
assert.equal(live.size, 1, "a live pooled session exists");
|
||||||
|
|
||||||
|
// gracefulShutdown: drain() then process.exit(0) IN THE SAME TICK (TUI panes are tmux
|
||||||
|
// children, not node children, so activeProcesses is empty and the exit is immediate).
|
||||||
|
// Nothing scheduled on the microtask queue can run. So we assert WITHOUT awaiting.
|
||||||
|
pool.drain();
|
||||||
|
assert.equal(live.size, 0,
|
||||||
|
"REGRESSION GUARD: the pane must be dead BEFORE any await. A .then()-based cleanup would " +
|
||||||
|
"never run before process.exit and would orphan a live authenticated `claude`.");
|
||||||
|
});
|
||||||
|
|
||||||
|
// M1b, second costume: drain() in the SAME synchronous block as refill(). The tmux session does
|
||||||
|
// not exist yet at drain time (bootPane runs on a microtask), so _cancelBooting's kill-by-name is
|
||||||
|
// a no-op — and a `.then` that merely `return`s on a stale generation would then let the boot
|
||||||
|
// CREATE the session and walk away from it. Not reachable from any current call site, but ADR 0008
|
||||||
|
// and the reap-tick comment both contemplate a boot-time pre-warm, which is exactly this shape.
|
||||||
|
// NOTE: deliberately NOT `hold: true`. A held boot never settles, so its `.then` never runs and
|
||||||
|
// the guard would vacuously pass — the test must let the boot actually SUCCEED, because the bug is
|
||||||
|
// precisely that a SUCCESSFUL boot on a cancelled generation walks away from its live session.
|
||||||
|
test("M1b': a boot cancelled BEFORE its session existed is still killed when it settles", async () => {
|
||||||
|
const { pool, live } = makeFakePool({ size: 1 });
|
||||||
|
pool.acquire("sonnet"); // miss → learns the model
|
||||||
|
|
||||||
|
pool.refill(); // mints the identity; bootPane is queued on a microtask — no session YET
|
||||||
|
pool.drain(); // SAME sync block: kill-by-name finds nothing to kill (no-op), bumps the gen
|
||||||
|
assert.equal(live.size, 0, "precondition: the session genuinely did not exist at cancel time");
|
||||||
|
|
||||||
|
await tick(); // NOW the boot microtask runs, CREATES the session, and settles on a stale gen
|
||||||
|
await tick();
|
||||||
|
|
||||||
|
assert.equal(live.size, 0,
|
||||||
|
"REGRESSION GUARD: a stale-generation boot must KILL its pane, not assume _cancelBooting " +
|
||||||
|
"already did. _cancelBooting kills BY NAME, and the tmux session does not exist until the " +
|
||||||
|
"boot microtask runs — so a cancellation landing first is a no-op, and a bare `return` here " +
|
||||||
|
"orphans a live authenticated `claude` that nothing owns.");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a stale boot settling after drain+resume must not clear the NEW boot's slot", async () => {
|
||||||
|
const { pool, releaseBoot } = makeFakePool({ size: 1, hold: true });
|
||||||
|
pool.acquire("sonnet");
|
||||||
|
pool.refill();
|
||||||
|
await tick();
|
||||||
|
pool.drain(); // cancels boot #1 (generation bumped)
|
||||||
|
pool.resume();
|
||||||
|
await tick();
|
||||||
|
assert.equal(pool.booting, 1, "boot #2 owns the slot");
|
||||||
|
releaseBoot(); // boot #1 finally settles (rejects)
|
||||||
|
await settle();
|
||||||
|
assert.equal(pool.booting, 1, "boot #2 STILL owns the slot — a stale settle must not free it");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a model switch cancels an in-flight boot for the OLD model (kills it, frees the slot)", async () => {
|
||||||
|
const { pool, live, killed } = makeFakePool({ size: 1, hold: true });
|
||||||
|
pool.acquire("sonnet");
|
||||||
|
pool.refill();
|
||||||
|
await tick();
|
||||||
|
const sonnetPane = [...live][0];
|
||||||
|
|
||||||
|
pool.acquire("opus"); // retarget mid-boot
|
||||||
|
assert.ok(killed.includes(sonnetPane), "the old model's booting pane is killed, not left to linger");
|
||||||
|
assert.equal(pool.booting, 0, "and its slot is freed immediately, so the new model can boot now");
|
||||||
|
assert.equal(pool.warmModel, "opus");
|
||||||
|
|
||||||
|
pool.refill();
|
||||||
|
await tick();
|
||||||
|
assert.equal(pool.booting, 1, "a boot for the NEW model starts without waiting out the old one");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a pane handed out for a turn leaves the spare set immediately (so its teardown is authoritative)", async () => {
|
||||||
|
const { pool } = makeFakePool({ size: 2 });
|
||||||
|
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||||
|
const taken = pool.acquire("sonnet");
|
||||||
|
assert.ok(!pool.liveNames().has(taken.name),
|
||||||
|
"an acquired pane is the CALLER's — the pool must not also claim it live, or a crashed turn's pane would be spared forever");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("N1: the pool mints ONE identity — the tmux name's hex is the transcript session-id's hex", async () => {
|
||||||
|
// Without this, `tmux ls` shows a pane whose name has no relation to any transcript file,
|
||||||
|
// so a live pane cannot be correlated to <HOME>/.claude/projects/*/<sessionId>.jsonl.
|
||||||
|
const seen = [];
|
||||||
|
const pool = new TuiPanePool({
|
||||||
|
size: 1,
|
||||||
|
mintPane: () => {
|
||||||
|
const sessionId = "deadbeef-1111-2222-3333-444444444444";
|
||||||
|
return { sessionId, name: poolName(3456, sessionId) };
|
||||||
|
},
|
||||||
|
bootPane: async (model, ident) => {
|
||||||
|
seen.push(ident);
|
||||||
|
return { ...ident, model, bootedAt: Date.now() };
|
||||||
|
},
|
||||||
|
killPane: () => {},
|
||||||
|
paneHealthy: () => true,
|
||||||
|
});
|
||||||
|
pool.acquire("sonnet");
|
||||||
|
pool.refill();
|
||||||
|
await settle();
|
||||||
|
assert.equal(seen.length, 1, "bootPane received the pool-minted identity");
|
||||||
|
assert.equal(seen[0].name, "ocp-tui-3456-pdeadbeef");
|
||||||
|
assert.ok(seen[0].name.endsWith(seen[0].sessionId.slice(0, 8)),
|
||||||
|
"the tmux session name carries the session-id's own hex — `tmux ls` correlates to the transcript");
|
||||||
|
const pane = pool.acquire("sonnet");
|
||||||
|
assert.equal(pane.sessionId, seen[0].sessionId,
|
||||||
|
"and the turn reads the transcript under THAT session-id — one identity end to end");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("pool pane names are port-scoped (reapable as ours) and never match the legacy shape", () => {
|
||||||
|
const name = poolName(3456, "deadbeef-1111-2222-3333-444444444444");
|
||||||
|
assert.ok(name.startsWith(sessionPrefixForPort(3456)), "pool panes are OURS → reapable when orphaned");
|
||||||
|
assert.equal(name, "ocp-tui-3456-pdeadbeef");
|
||||||
|
assert.ok(!LEGACY_SESSION_NAME_RE.test(name), "must never be mistaken for a legacy bare-prefix session");
|
||||||
|
assert.ok(!poolName(9999, "aaaaaaaa-0000-0000-0000-000000000000").startsWith(sessionPrefixForPort(3456)),
|
||||||
|
"a sibling instance's pool pane is foreign to us");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("buildTuiHealthBlock reports pool:null when off, and the pool's stats when on", async () => {
|
||||||
|
const st = { lastEntrypoint: "cli", entrypointMismatches: 0 };
|
||||||
|
const sem = { inflight: 0, queued: 0 };
|
||||||
|
const off = buildTuiHealthBlock({ enabled: true, entrypointMode: "cli", maxConcurrent: 2 }, st, sem, null);
|
||||||
|
assert.equal(off.pool, null, "pool disabled → explicit null (stable /health shape)");
|
||||||
|
|
||||||
|
const { pool } = makeFakePool({ size: 2 });
|
||||||
|
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||||
|
const on = buildTuiHealthBlock({ enabled: true, entrypointMode: "cli", maxConcurrent: 2 }, st, sem, pool);
|
||||||
|
assert.equal(on.pool.size, 2);
|
||||||
|
assert.equal(on.pool.warm, 2);
|
||||||
|
assert.equal(on.pool.misses, 1);
|
||||||
|
assert.equal(on.pool.model, "sonnet");
|
||||||
|
});
|
||||||
|
|
||||||
// ── TUI home preparation (scratch vs real) ───────────────────────────────
|
// ── TUI home preparation (scratch vs real) ───────────────────────────────
|
||||||
import { prepareTuiHome, ensureTuiCwdTrusted } from "./lib/tui/session.mjs";
|
import { prepareTuiHome, ensureTuiCwdTrusted } from "./lib/tui/session.mjs";
|
||||||
import { mkdtempSync as hMkdtemp, mkdirSync as hMkdir, writeFileSync as hWrite, readFileSync as hRead, existsSync as hExists, readlinkSync as hReadlink } from "node:fs";
|
import { mkdtempSync as hMkdtemp, mkdirSync as hMkdir, writeFileSync as hWrite, readFileSync as hRead, existsSync as hExists, readlinkSync as hReadlink } from "node:fs";
|
||||||
@@ -2474,8 +2978,13 @@ test("buildTuiHealthBlock: shape + live counters (the additive /health tui block
|
|||||||
const ts = { lastEntrypoint: "cli", entrypointMismatches: 3 };
|
const ts = { lastEntrypoint: "cli", entrypointMismatches: 3 };
|
||||||
const block = buildTuiHealthBlock(
|
const block = buildTuiHealthBlock(
|
||||||
{ enabled: true, entrypointMode: "cli", maxConcurrent: 2 }, ts, sem);
|
{ enabled: true, entrypointMode: "cli", maxConcurrent: 2 }, ts, sem);
|
||||||
|
// `pool` joined this key set with the warm pane pool. The tui block is ADR-0007-owned (it
|
||||||
|
// did not exist at v3.16.4, so it is outside ADR 0006's grandfather freeze), and the
|
||||||
|
// addition is purely additive: every pre-existing key below still carries a byte-identical
|
||||||
|
// value, and `pool` is null unless the operator opts in via OCP_TUI_POOL_SIZE.
|
||||||
assert.deepEqual(Object.keys(block).sort(),
|
assert.deepEqual(Object.keys(block).sort(),
|
||||||
["enabled", "entrypointMismatches", "entrypointMode", "inflight", "lastEntrypoint", "maxConcurrent", "queued"]);
|
["enabled", "entrypointMismatches", "entrypointMode", "inflight", "lastEntrypoint", "maxConcurrent", "pool", "queued"]);
|
||||||
|
assert.equal(block.pool, null, "no pool passed → null (the default, pool disabled)");
|
||||||
assert.equal(block.enabled, true);
|
assert.equal(block.enabled, true);
|
||||||
assert.equal(block.entrypointMode, "cli");
|
assert.equal(block.entrypointMode, "cli");
|
||||||
assert.equal(block.lastEntrypoint, "cli");
|
assert.equal(block.lastEntrypoint, "cli");
|
||||||
@@ -2934,7 +3443,9 @@ async function runAsyncTests() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ── Cleanup ──
|
// ── Cleanup ──
|
||||||
runAsyncTests().then(() => {
|
// Settle the async-bodied tests registered through the sync `test()` helper BEFORE summarizing —
|
||||||
|
// otherwise their pass/fail is not reflected in the counts (see the `pendingAsync` comment above).
|
||||||
|
runAsyncTests().then(() => Promise.all(pendingAsync)).then(() => {
|
||||||
closeDb();
|
closeDb();
|
||||||
console.log(`\n=== Results: ${passed} passed, ${failed} failed ===\n`);
|
console.log(`\n=== Results: ${passed} passed, ${failed} failed ===\n`);
|
||||||
process.exit(failed > 0 ? 1 : 0);
|
process.exit(failed > 0 ? 1 : 0);
|
||||||
|
|||||||
Reference in New Issue
Block a user