mirror of
https://github.com/dtzp555-max/ocp.git
synced 2026-07-21 21:15:09 +00:00
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
64150e0408 | ||
|
|
ed9abb19fe |
@@ -960,6 +960,7 @@ See [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) and ADR 0007 PR-
|
||||
| `OCP_TUI_ENTRYPOINT` | `cli` | (TUI-mode) Billing-classifier labeling: `cli` (default) pins `cc_entrypoint=cli` deterministically; `auto` lets claude self-classify via TTY detection; `off` leaves the inherited env untouched. Honest only when the spawn is a genuine interactive PTY — see ADR 0007. |
|
||||
| `OCP_TUI_EFFORT` | `low` | (TUI-mode) Effort level passed to the interactive `claude` as an explicit `--effort` flag: `low` (default), `medium`, `high`, `xhigh`, `max`, or `inherit` to omit the flag (the pre-flag behaviour: the pane inherits a HOME-dependent effort — the operator's `~/.claude/settings.json` `effortLevel` in real-home mode, claude's built-in default in env-token scratch mode). Explicit `low` cuts measured TTFT p50 by ~40% and collapses run-to-run variance ~15× versus an inherited `xhigh` (see `docs/plans/2026-07-13-tui-latency/`); proxied requests rarely benefit from extended thinking. Banner-verified to stay on the subscription pool (`· Claude Max`). An invalid value logs a warning and falls back to `low`. |
|
||||
| `OCP_TUI_MAX_CONCURRENT` | `2` | (TUI-mode) Max concurrent interactive TUI turns. **Independent** of `CLAUDE_MAX_CONCURRENT` (which bounds the `-p`/stream-json path; TUI never uses it). A TUI turn is heavy (per-request cold-boot of tmux+claude + up to `CLAUDE_TUI_WALLCLOCK_MS` wallclock), so the default is low to keep small hosts (e.g. a Pi 4) alive under a burst. Excess turns **queue** (bounded); a full queue yields a 503. See ADR 0007 PR-B amendment. |
|
||||
| `OCP_TUI_POOL_SIZE` | `0` (off) | (TUI-mode) Number of **pre-booted warm `claude` panes** kept ready, so a request does not pay the cold boot. `0` disables the pool entirely — the request path is then exactly the cold-boot path. Max `4`; an unparseable value disables it rather than guessing. **Measured on a Mac mini (Sonnet 4.6, `--effort low`): end-to-end p50 `10.17s` (n=6, pool off) → `6.00s` (n=12 warm hits) — −4.2 s / −41%** — the pool recovers both the ~1.2 s boot *and* ~2.9 s of post-input-bar init that a pane which has been idle a moment has already finished. **Cost:** each warm pane is a *live idle `claude` process* held whether or not a request ever arrives (peak processes ≈ pool size + `OCP_TUI_MAX_CONCURRENT` + 1 booting replacement) — which is why it is opt-in. Panes are **single-use**: one turn, then killed and replaced in the background. The **first request after start (and after any model switch) is always a cold miss** — the pool warms the most recently requested model, since OCP cannot know which model the next caller wants. See `docs/plans/2026-07-13-tui-latency/`. |
|
||||
| `OCP_SKIP_AUTH_TEST` | *(unset)* | When `=1`, skip the `claude -p` auth probe during `setup.mjs`. After 2026-06-15 this probe draws from the Agent SDK credit pool; set this to avoid burning a metered credit on re-installs or `ocp update` runs. Auth is validated at the first real request. |
|
||||
| `OCP_TUI_FULL_TOOLS` | *(unset)* | (TUI-mode, **single-user only**) When `=1`, grant the interactive session the **same tool surface as the `-p` path** — `--allowedTools` (+ optional `--mcp-config`, read from `CLAUDE_ALLOWED_TOOLS` / `CLAUDE_MCP_CONFIG`) — instead of the default MCP-walled, built-in-tools-only set. Lets a trusted single-operator TUI deployment run a **tool-using / MCP agent** (e.g. an OpenClaw assistant) on the subscription pool. Safe because TUI **refuses to boot under `AUTH_MODE=multi`** (hard exit) — no guest key can ever reach the TUI path, so this gate cannot expose tools to an untrusted caller. (Under `AUTH_MODE=shared` + `OCP_TUI_ALLOW_LAN=1`, anyone holding the single shared key reaches it — that is the existing TUI trust model, unchanged.) Note: `--dangerously-skip-permissions` / `CLAUDE_SKIP_PERMISSIONS` is **not** supported for TUI — claude v2.1.x shows an interactive bypass-acceptance screen in headless tmux that cannot be answered, bricking the pane. Use scratch-home `settings.json` `additionalDirectories` instead. See [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) and ADR 0007. |
|
||||
|
||||
@@ -1040,6 +1041,7 @@ Then restart OCP. At boot you will see (with the env token set, isolated home au
|
||||
- **Stale tmux sessions are reaped.** The pane's `claude` is a child of the tmux server (not OCP), so OCP cannot reap it directly; `claude` zombies can otherwise accumulate as `<defunct>` over a long-running host. OCP reaps them at boot and on a 15-min idle sweep by issuing `tmux kill-server` — but **only when no foreign tmux session remains** (it never disrupts a co-hosted `olp-tui-*` instance). See ADR 0007 PR-C amendment.
|
||||
- **Default path unchanged.** Unset `CLAUDE_TUI_MODE` and restart → `callClaude` / `callClaudeStreaming` are used again, byte-for-byte identical to today.
|
||||
- **Concurrency is bounded separately.** TUI turns are heavy (per-request cold-boot + long wallclock), so the TUI path has its own limiter — `OCP_TUI_MAX_CONCURRENT` (default `2`), independent of `CLAUDE_MAX_CONCURRENT`. Excess turns queue; a full queue returns a 503. Tune it up only on a host that can run more interactive `claude` sessions at once.
|
||||
- **Optional warm pane pool (`OCP_TUI_POOL_SIZE`, default off).** Pre-boots panes so a request skips the cold boot — measured p50 `10.17s` → `6.00s` (−41%). Pooled panes are **single-use** (one turn, then killed and replaced in the background), each carrying its own fresh `--session-id`, so one session still means one exchange and no earlier-turn text can leak into a later answer. They are named `ocp-tui-<port>-p<hex>` and coexist with the reaper by design: the sweep **drains the pool first**, then reaps (so `kill-server` still flushes `<defunct>` zombies), then the pool refills in the background. Drain→reap→resume is synchronous, so no request can land mid-sweep; a request arriving while the pool is still re-booting simply misses it and cold-boots. A live pooled pane is never reaped — **including one that is still booting**, whose tmux session already exists — while an *orphaned* one (left by a previous process generation) still is.
|
||||
|
||||
### ⚠️ Latency: TUI mode has a ~6-second floor, and it is immovable
|
||||
|
||||
@@ -1082,12 +1084,26 @@ a sub-5-second budget. Full measurements and methodology:
|
||||
"entrypointMismatches": 0, // count of cli-expected-but-got-other turns — ALERT if this climbs
|
||||
"inflight": 1, // TUI turns running right now
|
||||
"queued": 0, // TUI turns waiting for a concurrency slot
|
||||
"maxConcurrent": 2 // OCP_TUI_MAX_CONCURRENT
|
||||
"maxConcurrent": 2, // OCP_TUI_MAX_CONCURRENT
|
||||
"pool": { // warm pane pool — null when OCP_TUI_POOL_SIZE=0 (the default)
|
||||
"size": 2, // target warm panes (OCP_TUI_POOL_SIZE)
|
||||
"warm": 2, // panes ready right now — each is a LIVE idle claude process
|
||||
"booting": 0, // replacement panes currently pre-booting
|
||||
"model": "claude-sonnet-4-6", // the model being warmed (the most recently requested one)
|
||||
"hits": 12, // requests served by a warm pane
|
||||
"misses": 1, // requests that fell back to the cold boot (the 1st is always one)
|
||||
"boots": 14, // panes successfully pre-booted
|
||||
"bootFailures": 0, // pre-boots that genuinely never reached the input bar — WATCH this
|
||||
"cancelled": 4, // in-flight boots OCP killed on purpose (drain / model switch) — not faults
|
||||
"dropped": 8 // panes discarded unused (drain sweep / expired / unhealthy)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Alert on `entrypointMismatches > 0` (or `lastEntrypoint !== "cli"`): it means a turn drew from the metered Agent SDK pool instead of the subscription. `inflight` / `queued` show how close the TUI path is to its concurrency cap.
|
||||
|
||||
With the pool on, `hits` / `misses` is the hit rate (a steady single-model consumer should sit near 100% after the first request), and `warm` is your standing idle-process cost. A climbing `bootFailures` means panes are not reaching their input bar — the pool then degrades safely to the cold path, but latency reverts to the un-pooled numbers. `cancelled` counts boots OCP killed *on purpose* (a drain, a model switch) and is **not** a fault signal — do not alert on it. A steadily climbing `dropped` is likewise normal: the 15-min reap sweep drains and re-boots the pool on every tick so `kill-server` can still flush `<defunct>` zombies.
|
||||
|
||||
### Kill-switch
|
||||
|
||||
```bash
|
||||
|
||||
@@ -0,0 +1,208 @@
|
||||
# ADR 0008 — TUI Warm Pane Pool
|
||||
|
||||
**Date:** 2026-07-13
|
||||
**Status:** Proposed
|
||||
**Extends:** [ADR 0007](0007-tui-interactive-mode.md) (TUI interactive mode). This ADR does not
|
||||
change ADR 0007's billing-pool argument, security posture, or kill-switch — it adds a latency
|
||||
optimization *inside* the TUI spawn machinery ADR 0007 owns.
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
TUI mode (ADR 0007) serves every request by cold-booting a fresh `tmux` session running an
|
||||
interactive `claude`, submitting one prompt, reading the native transcript, and killing the
|
||||
session. That cold boot is paid on **every** request.
|
||||
|
||||
[`docs/plans/2026-07-13-tui-latency/`](../plans/2026-07-13-tui-latency/README.md) measured the
|
||||
TUI path and listed a warm pane pool as backlog item #3, costed at "**~1.0 s**" (the observed
|
||||
boot-to-input-bar time). Instrumenting the real request path showed that estimate is **~4×
|
||||
too low**. Phase decomposition of the cold path (n=6 medians, Sonnet 4.6, `--effort low`,
|
||||
through a real OCP instance):
|
||||
|
||||
| Phase | Median |
|
||||
|---|---|
|
||||
| prep (trust cwd, write prompt file) | 2 ms |
|
||||
| `tmux new-session` | 27 ms |
|
||||
| **boot → input bar ready** | **1232 ms** |
|
||||
| paste (`load-buffer` + `paste-buffer`) | 8 ms |
|
||||
| paste-verify poll | 426 ms |
|
||||
| **submit → transcript terminal** | **8458 ms** |
|
||||
| teardown | 8 ms |
|
||||
| **total** | **10162 ms** |
|
||||
| *claude's own reported `turn_duration`* | *5539 ms* |
|
||||
| **OCP-side overhead** | **4490 ms** |
|
||||
|
||||
The `submit → terminal` phase exceeds claude's own `turn_duration` by **~2.9 s**. That gap is
|
||||
**post-input-bar initialization inside `claude`** — work that a pane which has merely *sat idle
|
||||
for a few seconds* has already completed. A direct spike confirmed it: an identical pane, idle
|
||||
12 s before receiving the same prompt, completed its turn in a median 5537 ms versus 7980 ms
|
||||
cold.
|
||||
|
||||
So a warm pane recovers **~1.26 s of boot *and* ~2.9 s of in-`claude` cold start** — not the
|
||||
~1.0 s the plan predicted.
|
||||
|
||||
The reason this was worth a pool rather than a "keep one session and reuse it" cache is a
|
||||
hazard already flagged in the code. `lib/tui/transcript.mjs` returns the **last text-bearing
|
||||
assistant entry in the whole transcript file**, which is correct *only* under OCP's
|
||||
one-session-per-request model, and it says so:
|
||||
|
||||
> *"If a future warm-pool ever reuses a session WITHOUT a fresh session-id / clear, earlier-turn
|
||||
> text could leak — that author must add user-line scoping here."*
|
||||
|
||||
Reusing a pane for a second turn puts two exchanges in one transcript and would leak the earlier
|
||||
turn's text into the later turn's answer — a **cross-request data leak**, not merely a bug.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
Add an **opt-in pool of pre-booted, single-use `claude` panes**, `OCP_TUI_POOL_SIZE` (default
|
||||
`0` = off, max `4`). Implementation: `lib/tui/pool.mjs`.
|
||||
|
||||
### 1. Panes are SINGLE-USE. This is the load-bearing rule.
|
||||
|
||||
A pooled pane serves **exactly one turn**, then is killed and replaced in the background. Each
|
||||
pane is booted with its **own fresh `--session-id`**, fixed at spawn, and the turn locates its
|
||||
transcript by that id.
|
||||
|
||||
This preserves one-session-per-request exactly, so the `transcript.mjs` hazard above **does not
|
||||
arise** and no user-line scoping was needed. The warning in `transcript.mjs` is deliberately
|
||||
left standing, now annotated: it still binds anyone who later wants a pane to serve a second
|
||||
turn, or to reset a session with `/clear` and reuse it. **Neither is permitted without first
|
||||
adding user-line scoping to the transcript reader.**
|
||||
|
||||
Rejected alternative — *reuse a pane for N turns, `/clear` between* — is strictly cheaper
|
||||
(no re-boot per request) and was rejected on exactly this basis. The latency win is not worth a
|
||||
cross-request text-leak surface guarded only by a `/clear` that we cannot verify landed.
|
||||
|
||||
### 2. The pool is keyed by model, and a MISS is always safe.
|
||||
|
||||
`--model` is fixed at spawn, so a pane can only serve the model it booted with. A pool miss
|
||||
falls back to the existing cold-boot path with **zero behavioural difference**. There is no
|
||||
boot-time pre-warm and no configured model: OCP cannot know which model the next caller wants,
|
||||
so the pool warms the **most recently requested** model. Consequence, stated plainly: **the
|
||||
first request after start, and the first after any model switch, is always a cold miss.**
|
||||
|
||||
### 3. The pool and the session reaper coexist by an explicit invariant.
|
||||
|
||||
This is the subtle part. `reapStaleTuiSessions()` kills every session matching this instance's
|
||||
`ocp-tui-<port>-` prefix, and issues `tmux kill-server` when no foreign session remains (the
|
||||
only mechanism that can reap `<defunct>` `claude` zombies — the pane's `claude` is a child of
|
||||
the tmux *server*, not of node). A warm pooled pane **is** one of our own sessions, alive and
|
||||
idle **by design** — and the periodic sweep runs precisely **when the instance is idle**, i.e.
|
||||
exactly when the pool is full.
|
||||
|
||||
The invariant, stated in a comment above `reapStaleTuiSessions` and pinned by tests:
|
||||
|
||||
1. **A live pooled pane is never reaped — including one that is still BOOTING.** The reaper
|
||||
takes a `spare` set of **exact session names** supplied by the pool's live registry.
|
||||
2. **An orphaned pooled pane IS still reaped.** Membership is by **exact name from a live
|
||||
in-memory registry, never by name shape**. A pane the pool no longer owns — handed out,
|
||||
dropped, cancelled, or left behind by a previous process generation (whose registry died with
|
||||
it) — is absent from `spare` and is killed like any other stale session. **Fail-safe:
|
||||
omitting `spare` reaps *more*, never less.** Pool panes are named `ocp-tui-<port>-p<hex>`
|
||||
purely for operator legibility; that shape is *not* the exemption mechanism.
|
||||
3. **`kill-server` is suppressed while any pane is spared** (it would kill a live child of the
|
||||
tmux server). Therefore **the pool is DRAINED immediately before every sweep**, so `spare` is
|
||||
empty on the normal tick and `kill-server` still fires. Without the drain, a permanently-full
|
||||
pool would **permanently disable zombie reaping** — the pool would silently break the thing
|
||||
the sweep exists to do. The drain costs one pane re-boot per tick (15 min).
|
||||
|
||||
The `spare` mechanism is belt-and-braces given the drain: it makes it impossible for a reap call
|
||||
site that *forgets* to drain to kill a live pane.
|
||||
|
||||
### 4. The pool tracks its in-flight boot BY NAME, not as a count.
|
||||
|
||||
`bootTuiPane` creates the tmux session **synchronously** and only *then* waits (up to
|
||||
`POOL_BOOT_MS`, 20 s) for the input bar. So **a pooled tmux session can be live for ~20 s before
|
||||
its boot resolves.** A pool that tracked in-flight boots as a *count* could not name that
|
||||
session, and this produced two real bugs (both caught in review, both now regression-tested):
|
||||
|
||||
- the periodic sweep **killed the booting pane** (it could not be spared), then left the pool
|
||||
empty with nothing scheduled, and logged the exact `tui_pool_boot_failed` warning operators are
|
||||
told to alert on — for a completely healthy drain;
|
||||
- graceful shutdown **orphaned a live, authenticated, idle `claude`**: `gracefulShutdown` calls
|
||||
`process.exit(0)` in the same tick as the drain (TUI panes are tmux children, so node's
|
||||
`activeProcesses` set is empty and the "wait for children" path exits immediately), so any
|
||||
cleanup deferred to a `.then()` never ran.
|
||||
|
||||
The pool therefore **mints each pane's identity up front** (`{sessionId, name}`, name derived
|
||||
from the session-id so `tmux ls` correlates to the transcript file) and holds it in
|
||||
`_bootingPane`. `liveNames()` includes it; `drain()` kills it **synchronously**. A generation
|
||||
counter distinguishes *"cancelled by us"* from *"genuinely failed"*, so a drain never inflates
|
||||
`bootFailures` and `resume()` reliably starts a fresh boot.
|
||||
|
||||
### 5. Refills take no concurrency slot, and are serialized.
|
||||
|
||||
A refill boot deliberately does **not** take a `TuiSemaphore` slot: those slots bound concurrent
|
||||
*turns* and belong to real requests, and charging a background pre-boot against them would let
|
||||
the pool starve the traffic it exists to speed up. It cannot leak a slot either, since it never
|
||||
holds one. Boots are **serialized** (one at a time): two cold boots racing an in-flight turn were
|
||||
observed to overrun even the generous pool readiness cap. A genuinely failed boot does **not**
|
||||
re-kick the chain (backoff — a broken `claude` must not respawn forever).
|
||||
|
||||
Background boots get a more generous readiness cap (`POOL_BOOT_MS` = 5 × `BOOT_MS`): `BOOT_MS` is
|
||||
tight because a *client* is blocked on it, which is not true of a pre-boot. Slow ≠ broken.
|
||||
|
||||
---
|
||||
|
||||
## Consequences
|
||||
|
||||
### Cost — standing processes, paid whether or not a request arrives
|
||||
|
||||
**A warm pane is a live idle `claude` process.** Peak process count is
|
||||
`OCP_TUI_POOL_SIZE` + `OCP_TUI_MAX_CONCURRENT` + 1 (booting replacement). This is the whole
|
||||
reason the pool is **default-off**: an operator must opt into holding processes for traffic that
|
||||
may never come. Size is clamped to `POOL_MAX_SIZE` = 4; an unparseable value **disables** the
|
||||
pool rather than guessing.
|
||||
|
||||
Panes carry a 10-minute TTL and are health-checked at hand-out; a dead or degraded pane becomes
|
||||
a **miss** (cold path), never a hung turn.
|
||||
|
||||
### Benefit
|
||||
|
||||
Measured end-to-end through a real OCP instance (Sonnet 4.6, `--effort low`):
|
||||
**p50 10.17 s (n=6, pool off) → 6.00 s (n=12 warm hits) — −4.2 s / −41%.**
|
||||
|
||||
### The floor is unchanged
|
||||
|
||||
The pool does not touch the **~6 s TTFT floor** documented in the latency plan (claude always
|
||||
prefills the full Claude Code system prompt). TUI mode remains unsuitable for interactive /
|
||||
real-time consumers; it is for batch and background work. This ADR does not change that
|
||||
conclusion.
|
||||
|
||||
### Observability
|
||||
|
||||
`/health`'s `tui` block gains a `pool` sub-object (`null` when off): `size`, `warm`, `booting`,
|
||||
`model`, `hits`, `misses`, `boots`, `bootFailures`, `cancelled`, `dropped`. A climbing
|
||||
`bootFailures` means panes are not reaching their input bar — the pool then degrades safely to
|
||||
the cold path, but latency reverts to the un-pooled numbers. A steadily climbing `dropped` is
|
||||
**normal** (the 15-min sweep drains and re-boots the pool on every tick, by design — see
|
||||
Decision 3).
|
||||
|
||||
### ALIGNMENT authorization
|
||||
|
||||
- **Class B / OCP-owned.** The warm pool is process management around the `claude` CLI — the
|
||||
same category as the existing tmux session lifecycle and the defunct-session reaper it extends.
|
||||
**`cli.js` does not perform this operation, and no `cli.js` citation applies**; the authority
|
||||
is ADR 0007 (which owns the TUI spawn machinery) plus this ADR. This is `ALIGNMENT.md` Rule 2's
|
||||
Class B citation requirement, discharged explicitly rather than by silence.
|
||||
- **The `/health` extension** adds sub-fields to the `tui` block. That block is **owned by ADR
|
||||
0007** and post-dates ADR 0006's v3.16.4 grandfather snapshot, so it is not part of the frozen
|
||||
B.2 inventory. The change is additive — every pre-existing `/health` field keeps a
|
||||
byte-identical value, and `pool` is `null` unless the operator opts in — which is the
|
||||
behaviour-preserving bar ADR 0006 sets. This ADR records that authorization.
|
||||
- **No spawn argument changed.** `buildTuiCmd` is byte-identical; the pool calls it with the same
|
||||
arguments. Banner-verified on live pooled panes: `· Claude Max`, never `API Usage Billing`
|
||||
(the `--bare` trap documented in the latency plan).
|
||||
|
||||
### What a future contributor must not undo
|
||||
|
||||
- **Do not let a pane serve a second turn** (or `/clear`-and-reuse one) without first adding
|
||||
user-line scoping to `lib/tui/transcript.mjs`. That is a cross-request text leak, not a perf
|
||||
tweak. See Decision 1.
|
||||
- **Do not remove the drain-before-sweep.** It is what keeps `kill-server` zombie reaping alive.
|
||||
See Decision 3.
|
||||
- **Do not go back to counting in-flight boots.** The pool must be able to *name* a session that
|
||||
exists but has not finished booting. See Decision 4.
|
||||
@@ -23,6 +23,8 @@ New ADRs increment from the highest existing number. Filenames are
|
||||
| [0004](0004-openclaw-auto-sync.md) | OpenClaw Auto-Sync | Why `scripts/sync-openclaw.mjs` runs on `ocp update`, what its scope boundary is (writes only `models.providers["claude-local"].models` and `agents.defaults.models["claude-local/*"]`), and the idempotency contract. |
|
||||
| [0005](0005-no-multi-provider.md) | No Multi-Provider | Why OCP stays single-provider (Anthropic-via-cli.js) and does not extend to OpenAI / Gemini / OpenRouter. Cost estimate: ~7 weeks for a v1 that buys neither moat nor commercial readiness. Separate commercial work starts in a separate repo. |
|
||||
| [0006](0006-openai-shim-scope.md) | OpenAI Shim Scope | The Class A / Class B taxonomy. Class A endpoints (`cli.js`-mirror) keep Rules 1–5 verbatim; Class B endpoints (OCP-owned compatibility surface — `/v1/chat/completions`, `/v1/models`, admin endpoints) are anchored to OpenAI's spec (B.1) or to an authorizing ADR (B.2). Triggered by PR #99 (external `response_format` honoring). Grandfathers the existing B.2 inventory at v3.16.4. |
|
||||
| [0007](0007-tui-interactive-mode.md) | TUI Interactive Mode | Why TUI-mode spawns an interactive `claude` in a tmux pane (no `-p`) to reach the **subscription** billing pool (`cc_entrypoint=cli`) rather than the metered Agent SDK pool. Owns the TUI spawn machinery: entrypoint labeling, credential-isolated home, MCP hard-disable, session namespace + defunct-session reaping, the independent concurrency bound, and the `/health` `tui` block. **Single-user only** — hard FATAL on multi-user configs. |
|
||||
| [0008](0008-tui-warm-pane-pool.md) | TUI Warm Pane Pool | Why `OCP_TUI_POOL_SIZE` pre-boots **single-use** `claude` panes (one turn each, own `--session-id`) — and why reuse is forbidden (`transcript.mjs` returns the last assistant entry in the file, so a reused session leaks the earlier turn's text). Measured −41% end-to-end. Defines the pool↔reaper invariant (exemption by exact name from a live registry; drain before every sweep so `kill-server` zombie reaping survives) and the standing idle-process cost. Extends ADR 0007. |
|
||||
|
||||
## When to write a new ADR
|
||||
|
||||
|
||||
@@ -0,0 +1,309 @@
|
||||
// TUI warm pane pool (docs/plans/2026-07-13-tui-latency backlog #3).
|
||||
//
|
||||
// WHAT IT IS: a small set of PRE-BOOTED `claude` panes, each already sitting at its
|
||||
// input bar, so a request does not pay the cold boot. Opt-in: OCP_TUI_POOL_SIZE=0
|
||||
// (default) disables it entirely and the request path is byte-for-byte today's.
|
||||
//
|
||||
// ── SINGLE-USE IS THE LOAD-BEARING RULE ─────────────────────────────────────
|
||||
// A pooled pane serves EXACTLY ONE turn and is then killed and replaced in the
|
||||
// background. Each pane carries its OWN fresh `--session-id`, fixed at boot, and the
|
||||
// turn locates its transcript by that id. So OCP's one-session-per-request model is
|
||||
// preserved: a session's transcript still holds exactly one logical exchange.
|
||||
// That is what keeps lib/tui/transcript.mjs's extractLatestAssistantText (which returns
|
||||
// the LAST text-bearing assistant entry in the whole file, not "text since the matching
|
||||
// user line") correct — see the scoping note there. A pane MUST NEVER serve a second
|
||||
// turn, and a session MUST NEVER be reset with /clear and reused: either would put two
|
||||
// exchanges in one transcript and leak the earlier turn's text into the later turn's
|
||||
// answer. Nothing here reuses a pane; keep it that way.
|
||||
//
|
||||
// ── WHY IT'S WORTH MORE THAN THE BOOT TIME ──────────────────────────────────
|
||||
// Measured on this host (n=6 through OCP, Sonnet 4.6, --effort low): the cold path
|
||||
// spends ~1.23 s reaching the input bar, but ALSO ~2.9 s inside the first turn beyond
|
||||
// what claude itself reports as the turn duration — post-input-bar init that a pane
|
||||
// which has been idle for a few seconds has already finished. A warm pane recovers both.
|
||||
//
|
||||
// ── COST (bounded, and paid whether or not a request arrives) ───────────────
|
||||
// Each warm pane is a LIVE `claude` process (plus its tmux pane) sitting idle. Peak
|
||||
// process count is (pool size) + (OCP_TUI_MAX_CONCURRENT in-flight turns) + (panes
|
||||
// currently booting as replacements). Pool size is clamped to POOL_MAX_SIZE.
|
||||
//
|
||||
// Pure + injectable (bootPane / killPane / paneHealthy / now) so test-features.mjs can
|
||||
// assert acquire / miss / refill / TTL / reaper-exemption with no tmux and no claude.
|
||||
|
||||
// Hard cap on OCP_TUI_POOL_SIZE. Each pane is an idle claude process; 4 is already a
|
||||
// lot of resident memory on a small host (a Pi serving a family) for zero in-flight work.
|
||||
export const POOL_MAX_SIZE = 4;
|
||||
|
||||
// A warm pane older than this is dropped on acquire rather than handed out. The periodic
|
||||
// reap tick (server.mjs) drains the pool every 15 min anyway, so this only bites when
|
||||
// that tick kept getting skipped because the TUI path was never idle. Guards against
|
||||
// handing out a pane whose `claude` has been sitting so long it may have drifted
|
||||
// (auto-compaction prompts, an idle-disconnect banner, an expired in-pane token).
|
||||
export const POOL_MAX_AGE_MS = 10 * 60 * 1000;
|
||||
|
||||
// Clamp the operator-supplied size into [0, POOL_MAX_SIZE]. A garbage value disables the
|
||||
// pool rather than guessing — an unparseable size must never silently boot 4 processes.
|
||||
export function resolvePoolSize(raw) {
|
||||
const n = parseInt(raw, 10);
|
||||
if (!Number.isFinite(n) || n <= 0) return 0;
|
||||
return Math.min(n, POOL_MAX_SIZE);
|
||||
}
|
||||
|
||||
export class TuiPanePool {
|
||||
// size: target number of warm panes (0 = disabled).
|
||||
// maxAgeMs: per-pane TTL (see POOL_MAX_AGE_MS).
|
||||
// mintPane: () => ({ sessionId, name }) — mints the identity of the NEXT pane. The POOL,
|
||||
// not the boot function, owns this: the tmux session springs into existence the
|
||||
// instant bootPane starts, so the pool must already know its NAME (see
|
||||
// _bootingPane below). Deriving the name from the sessionId also makes `tmux ls`
|
||||
// correlate to the transcript file.
|
||||
// bootPane: async (model, {sessionId, name}) => { name, sessionId, model, bootedAt } —
|
||||
// boots ONE pane under exactly that identity and resolves only once it is
|
||||
// input-ready; throws if it never becomes ready.
|
||||
// killPane: (name) => void — tmux kill-session. MUST be synchronous (see drain).
|
||||
// paneHealthy:(name) => bool — pane still exists AND is still at its input bar.
|
||||
constructor({ size, maxAgeMs = POOL_MAX_AGE_MS, mintPane, bootPane, killPane, paneHealthy, now = Date.now, log = () => {} }) {
|
||||
this.size = Math.max(0, Math.min(parseInt(size, 10) || 0, POOL_MAX_SIZE));
|
||||
// Fail fast at CONSTRUCTION, not at request time. refill() is called synchronously from
|
||||
// the request path (runTuiTurn), so a missing collaborator would otherwise surface as a
|
||||
// 500 on a live request instead of a loud error at boot.
|
||||
if (this.size > 0) {
|
||||
for (const [k, fn] of [["mintPane", mintPane], ["bootPane", bootPane], ["killPane", killPane], ["paneHealthy", paneHealthy]]) {
|
||||
if (typeof fn !== "function") throw new TypeError(`TuiPanePool: ${k} must be a function`);
|
||||
}
|
||||
}
|
||||
this.maxAgeMs = maxAgeMs;
|
||||
this._mintPane = mintPane;
|
||||
this._bootPane = bootPane;
|
||||
this._killPane = killPane;
|
||||
this._paneHealthy = paneHealthy;
|
||||
this._now = now;
|
||||
this._log = log;
|
||||
|
||||
this._panes = []; // warm, available panes: { name, sessionId, model, bootedAt }
|
||||
// The pane currently BOOTING, BY NAME ({sessionId, name, model}) — or null.
|
||||
//
|
||||
// WHY A NAME AND NOT A COUNT (this is a fixed bug, don't regress it): bootTuiPane creates
|
||||
// the tmux session SYNCHRONOUSLY and only THEN waits up to POOL_BOOT_MS (20 s) for the
|
||||
// input bar. So for up to 20 s there is a LIVE pooled tmux session. When the pool tracked
|
||||
// only a count, it could not NAME that session, so:
|
||||
// - liveNames() could not spare it and the periodic reap sweep KILLED it (and
|
||||
// kill-server'd on top), leaving the pool empty with nothing scheduled and firing the
|
||||
// very tui_pool_boot_failed WARN operators are told to alert on; and
|
||||
// - drain() could not kill it, so on shutdown it ORPHANED a live authenticated `claude`
|
||||
// (the boot's .then that was supposed to clean up never runs — gracefulShutdown calls
|
||||
// process.exit in the same tick).
|
||||
// Both are fixed by holding the identity here, before the session exists.
|
||||
this._bootingPane = null;
|
||||
// Generation counter. Bumped whenever an in-flight boot is CANCELLED (drain / model
|
||||
// switch). A boot compares the generation it started under against the current one:
|
||||
// if they differ, its pane was already killed by us and its settle is inert — in
|
||||
// particular a rejection is a CANCELLATION, not an operator-visible boot failure.
|
||||
this._gen = 0;
|
||||
this._paused = false; // true while drained; refill() is a no-op until resume()
|
||||
this.warmModel = null; // the model the pool currently warms — learned from traffic (see acquire)
|
||||
|
||||
this.hits = 0; // requests served by a warm pane
|
||||
this.misses = 0; // requests that fell back to the cold path
|
||||
this.boots = 0; // panes successfully pre-booted
|
||||
this.bootFailures = 0; // pre-boots that genuinely never reached the input bar
|
||||
this.cancelled = 0; // in-flight boots WE killed (drain / model switch) — not failures
|
||||
this.dropped = 0; // panes discarded unused (unhealthy / expired / wrong model / drained /
|
||||
// cancelled — a cancelled in-flight boot also lands here via _drop)
|
||||
}
|
||||
|
||||
get enabled() { return this.size > 0; }
|
||||
get warm() { return this._panes.length; }
|
||||
get booting() { return this._bootingPane ? 1 : 0; }
|
||||
|
||||
// The reaper's spare set: the EXACT names of every pane the pool currently owns and has NOT
|
||||
// handed out — the warm ones AND the one currently booting (whose tmux session is already
|
||||
// live; see _bootingPane). See the POOL/REAPER INVARIANT in lib/tui/session.mjs.
|
||||
// Fail-safe by construction: a pane leaves this set the instant it is acquired, dropped, or
|
||||
// cancelled, and if the pool is empty (or the process restarted) the set is empty — so an
|
||||
// orphaned pooled pane looks exactly like any other stale session and IS reaped.
|
||||
liveNames() {
|
||||
const names = new Set(this._panes.map((p) => p.name));
|
||||
if (this._bootingPane) names.add(this._bootingPane.name);
|
||||
return names;
|
||||
}
|
||||
|
||||
// Take a warm pane for `model`, or null (caller must fall back to the cold path — a MISS
|
||||
// is always safe, never an error). Synchronous: paneHealthy is a cheap tmux capture.
|
||||
//
|
||||
// The pool warms the MOST RECENTLY REQUESTED model (`warmModel`). There is no boot-time
|
||||
// pre-warm and no configured model: OCP cannot know which model the next caller wants, and
|
||||
// pre-booting a process for a model nobody asks for is pure waste. Consequence, stated
|
||||
// plainly: the FIRST request after start (and the first after a model switch) is always a
|
||||
// MISS. The pool pays off for the steady repeat traffic it exists to serve.
|
||||
acquire(model) {
|
||||
if (!this.enabled) return null;
|
||||
|
||||
// Retarget on a model switch: --model is fixed at spawn, so panes for another model are
|
||||
// useless. Drop them now (they are replaced by the next refill) rather than holding
|
||||
// processes for a model that is no longer being asked for. This includes any pane
|
||||
// currently BOOTING for the old model — its tmux session already exists, so leaving it to
|
||||
// die on resolve would both hold a useless process and block the next refill (one boot at
|
||||
// a time) for up to POOL_BOOT_MS.
|
||||
if (model !== this.warmModel) {
|
||||
for (const p of this._panes) { this._drop(p, "model_switch"); }
|
||||
this._panes = [];
|
||||
this._cancelBooting("model_switch");
|
||||
this.warmModel = model;
|
||||
}
|
||||
|
||||
while (this._panes.length) {
|
||||
const p = this._panes.shift();
|
||||
if (this._now() - p.bootedAt > this.maxAgeMs) { this._drop(p, "expired"); continue; }
|
||||
if (!this._paneHealthy(p.name)) { this._drop(p, "unhealthy"); continue; }
|
||||
this.hits++;
|
||||
return p; // caller OWNS it now: it is out of the registry (so out of the spare set),
|
||||
// and the caller's finally MUST kill it. Single-use — never returned here.
|
||||
}
|
||||
this.misses++;
|
||||
return null;
|
||||
}
|
||||
|
||||
// Bring the pool back up to `size` warm panes for `warmModel`. Fire-and-forget: never
|
||||
// awaited on the request path and never throws into it.
|
||||
//
|
||||
// SLOT ACCOUNTING: a refill boot deliberately does NOT take a TuiSemaphore slot. Those
|
||||
// slots bound concurrent *turns* (each up to the 120 s wallclock) and belong to real
|
||||
// requests; charging a background pre-boot against them would let the pool starve the
|
||||
// traffic it exists to speed up. It cannot leak a slot either, because it never holds one.
|
||||
//
|
||||
// SERIALIZED, ONE BOOT AT A TIME (and re-kicked on success until the pool is at target).
|
||||
// An earlier version launched all `want` boots at once; live at size=2 that put two cold
|
||||
// `claude` boots plus an in-flight turn on the CPU together, and a refill overran even the
|
||||
// generous pool readiness cap (tui_pool_boot_failed). Booting sequentially keeps each boot
|
||||
// near its uncontended ~1.2 s, bounds the CPU burst the pool can cause, and still has the
|
||||
// replacement pane warm long before the next request arrives.
|
||||
//
|
||||
// A genuinely FAILED boot deliberately does NOT re-kick the chain — that is the backoff. A
|
||||
// persistently failing boot (bad claude binary, no auth) would otherwise spin, respawning
|
||||
// forever. The next natural trigger (the following request's refill, or the reap tick's
|
||||
// resume) retries it. A CANCELLED boot is different: we killed it on purpose, nothing is
|
||||
// wrong, and resume() is expected to start a fresh one immediately.
|
||||
refill() {
|
||||
if (!this.enabled || this._paused || !this.warmModel) return;
|
||||
if (this._bootingPane) return; // one boot in flight at a time
|
||||
if (this._panes.length >= this.size) return; // already at target
|
||||
|
||||
const model = this.warmModel;
|
||||
const gen = this._gen;
|
||||
// Mint the identity BEFORE booting: bootPane creates the tmux session synchronously, so
|
||||
// the pool must be able to name (and therefore spare, and kill) it from this moment on.
|
||||
const ident = this._mintPane();
|
||||
this._bootingPane = { ...ident, model };
|
||||
let enlisted = false;
|
||||
Promise.resolve()
|
||||
.then(() => this._bootPane(model, ident))
|
||||
.then((pane) => {
|
||||
// The world may have moved while we booted. If our generation was cancelled, kill the
|
||||
// pane here rather than ASSUMING _cancelBooting already did.
|
||||
//
|
||||
// Why not just `return`: _cancelBooting kills by name, but the tmux session only EXISTS
|
||||
// once _bootPane has actually run — and _bootPane is queued on a microtask (above). A
|
||||
// caller that does refill() and then drain() in the SAME synchronous block would have
|
||||
// _cancelBooting find nothing to kill (a no-op), bump the generation, and then this
|
||||
// microtask would create the session, boot it fine, and — under a bare `return` — walk
|
||||
// away from a LIVE authenticated `claude` that nothing owns. That is M1b in a new costume.
|
||||
// No current call site does that, so this is defense-in-depth, not a live bug — but ADR
|
||||
// 0008 and the reap-tick comment in server.mjs both explicitly contemplate a boot-time
|
||||
// pre-warm, which is exactly the shape that would reach it.
|
||||
//
|
||||
// Killing an already-dead session is a harmless no-op (_drop swallows it), so this is
|
||||
// idempotent whether or not _cancelBooting got there first.
|
||||
if (gen !== this._gen) { this._drop(pane, "cancelled_late"); return; }
|
||||
// Otherwise: still possible the pool filled or retargeted without a cancellation.
|
||||
if (this._paused || model !== this.warmModel || this._panes.length >= this.size) {
|
||||
this._drop(pane, "stale_boot");
|
||||
return;
|
||||
}
|
||||
this._panes.push(pane);
|
||||
this.boots++;
|
||||
enlisted = true;
|
||||
})
|
||||
.catch((e) => {
|
||||
// A rejection from a CANCELLED generation is not a fault: it is almost always
|
||||
// "tui_pane_not_ready", thrown because WE killed the pane out from under the boot.
|
||||
// Counting it as a bootFailure would fire the exact WARN operators are told to alert
|
||||
// on, for a completely healthy drain. Stay silent — _cancelBooting already counted
|
||||
// this as a cancellation, so do NOT count it again here.
|
||||
if (gen !== this._gen) return;
|
||||
this.bootFailures++;
|
||||
this._log("warn", "tui_pool_boot_failed", { model, error: e && e.message });
|
||||
})
|
||||
.finally(() => {
|
||||
// ONLY the current generation's boot owns the booting slot. A stale settle must not
|
||||
// clear a slot that a newer boot (started by resume()) already holds.
|
||||
if (gen === this._gen) this._bootingPane = null;
|
||||
if (enlisted) this.refill(); // continue toward target, still one at a time
|
||||
});
|
||||
}
|
||||
|
||||
// Kill the in-flight boot's pane, SYNCHRONOUSLY, and invalidate its generation. Returns 1
|
||||
// if there was one, else 0. The tmux session already exists (bootPane created it before it
|
||||
// started waiting for readiness), so this is a real kill, not a cancellation flag.
|
||||
_cancelBooting(reason) {
|
||||
if (!this._bootingPane) return 0;
|
||||
this._gen++; // the in-flight boot's settle is now inert
|
||||
this._drop(this._bootingPane, reason); // synchronous kill-session
|
||||
this._bootingPane = null;
|
||||
this.cancelled++;
|
||||
return 1;
|
||||
}
|
||||
|
||||
// Kill every pane the pool owns — warm AND currently booting — and stop refilling. Returns
|
||||
// how many were killed.
|
||||
//
|
||||
// Called (a) before the periodic reap sweep — reapStaleTuiSessions can only reap defunct
|
||||
// `claude` zombies via kill-server, and kill-server is suppressed while any live pooled pane
|
||||
// exists (including a booting one), so without this drain the pool would permanently disable
|
||||
// zombie reaping; and (b) on graceful shutdown, so no pane outlives the process as an orphan.
|
||||
//
|
||||
// EVERY KILL HERE IS SYNCHRONOUS, and that is load-bearing. It is NOT safe to leave the
|
||||
// booting pane to clean itself up on resolve: gracefulShutdown calls process.exit() in the
|
||||
// same tick as this drain (TUI panes are children of the tmux SERVER, not of node, so
|
||||
// node's activeProcesses set is empty on a TUI host and the "wait for children" path exits
|
||||
// immediately). A .then()/.catch() scheduled here would never run, and the pane would
|
||||
// survive as an orphaned, authenticated, idle `claude`.
|
||||
drain() {
|
||||
this._paused = true;
|
||||
let n = this._panes.length;
|
||||
for (const p of this._panes) this._drop(p, "drain");
|
||||
this._panes = [];
|
||||
n += this._cancelBooting("drain_booting");
|
||||
return n;
|
||||
}
|
||||
|
||||
// Undo drain() and start refilling again. Because drain() CANCELLED the in-flight boot
|
||||
// (rather than leaving it pending), the booting slot is free and this really does start a
|
||||
// fresh boot — the pool is never left empty with nothing scheduled.
|
||||
resume() {
|
||||
this._paused = false;
|
||||
this.refill();
|
||||
}
|
||||
|
||||
// /health surface (additive).
|
||||
stats() {
|
||||
return {
|
||||
size: this.size,
|
||||
warm: this._panes.length,
|
||||
booting: this.booting,
|
||||
model: this.warmModel,
|
||||
hits: this.hits,
|
||||
misses: this.misses,
|
||||
boots: this.boots,
|
||||
bootFailures: this.bootFailures,
|
||||
cancelled: this.cancelled,
|
||||
dropped: this.dropped,
|
||||
};
|
||||
}
|
||||
|
||||
_drop(pane, reason) {
|
||||
this.dropped++;
|
||||
try { this._killPane(pane.name); } catch { /* already gone */ }
|
||||
this._log("info", "tui_pool_pane_dropped", { name: pane.name, reason });
|
||||
}
|
||||
}
|
||||
@@ -139,7 +139,12 @@ export function recordTuiEntrypoint(tuiStats, observed, expectedMode = "cli") {
|
||||
// Build the additive /health `tui` block (ADR 0007 PR-B amendment). Pure: given the
|
||||
// config + live counters, returns the exact object embedded in /health. New fields only —
|
||||
// behaviour-preserving for existing /health consumers (grandfathered B.2 under ADR 0006).
|
||||
export function buildTuiHealthBlock({ enabled, entrypointMode, maxConcurrent }, tuiStats, semaphore) {
|
||||
//
|
||||
// `pool` (optional, warm pane pool — lib/tui/pool.mjs): a TuiPanePool, or null/undefined
|
||||
// when the pool is off (the default). Reported as `pool: null` when off so the block's
|
||||
// shape stays stable, and as the pool's stats (size / warm / hits / misses / …) when on —
|
||||
// the operator's window onto both the hit rate and the standing idle-process cost.
|
||||
export function buildTuiHealthBlock({ enabled, entrypointMode, maxConcurrent }, tuiStats, semaphore, pool = null) {
|
||||
return {
|
||||
enabled,
|
||||
entrypointMode, // cli | auto | off
|
||||
@@ -148,5 +153,6 @@ export function buildTuiHealthBlock({ enabled, entrypointMode, maxConcurrent },
|
||||
inflight: semaphore.inflight, // current concurrent TUI turns
|
||||
queued: semaphore.queued, // turns waiting for a slot
|
||||
maxConcurrent,
|
||||
pool: pool ? pool.stats() : null, // warm pane pool, or null when disabled
|
||||
};
|
||||
}
|
||||
|
||||
+186
-62
@@ -73,6 +73,36 @@ const defaultTmux = (args, opts = {}) =>
|
||||
// `port` (required) is this instance's own listen port (server.mjs's PORT / lib/constants.mjs
|
||||
// DEFAULT_PORT resolution) — the SPOT for "which sessions are ours."
|
||||
//
|
||||
// ── POOL/REAPER INVARIANT (warm pane pool — lib/tui/pool.mjs) ───────────────────────────
|
||||
// A warm pooled pane is one of OUR OWN `ocp-tui-<port>-*` sessions that is ALIVE AND IDLE
|
||||
// BY DESIGN — and the periodic sweep runs precisely when the instance is idle, i.e. exactly
|
||||
// when the pool is full. Without an exemption the sweep would kill every warm pane on every
|
||||
// tick (and kill-server on top). The exemption is `spare`: a set of EXACT session names the
|
||||
// caller declares live. Three properties, all load-bearing:
|
||||
//
|
||||
// 1. A LIVE POOLED PANE IS NEVER REAPED — INCLUDING ONE THAT IS STILL BOOTING. It is in
|
||||
// `spare` (the pool's live registry), so it is skipped by name. The booting case is not
|
||||
// a footnote, it is the one that bit us: bootTuiPane creates the tmux session
|
||||
// SYNCHRONOUSLY and only then waits up to POOL_BOOT_MS for the input bar, so a pooled
|
||||
// session can be live for ~20 s before its boot resolves. The pool therefore mints the
|
||||
// pane's NAME up front and holds it in `_bootingPane`, so liveNames() can name — and
|
||||
// spare — a session whose boot has not finished. (An earlier version tracked only a
|
||||
// COUNT of in-flight boots; the sweep could not name that session and killed it.)
|
||||
// 2. A LEAKED/ORPHANED POOLED PANE IS STILL REAPED. Membership is by EXACT NAME from a
|
||||
// live in-memory registry — NOT by "looks pooled" (name shape). A pane the pool no
|
||||
// longer owns (handed out, dropped, cancelled, or left behind by a previous process
|
||||
// generation — whose registry died with it) is absent from `spare` and is killed like
|
||||
// any other stale session. Fail-safe: forgetting to pass `spare` reaps MORE, never less.
|
||||
// 3. KILL-SERVER NEVER KILLS A LIVE POOL PANE. A spared session suppresses kill-server
|
||||
// exactly as a foreign session does (it is a live child of the tmux server). The
|
||||
// consequence — that a permanently-full pool would permanently disable the defunct-
|
||||
// zombie reaping that ONLY kill-server can do — is resolved in server.mjs by DRAINING
|
||||
// the pool immediately before the sweep, so `spare` is empty on the normal tick and
|
||||
// kill-server still fires. `spare` is the belt-and-braces: a reap call site that
|
||||
// forgets to drain still cannot kill a live pane.
|
||||
//
|
||||
// `spare` (default: none) — iterable of session names, or a Set. Ignored when the pool is off.
|
||||
//
|
||||
// `includeLegacy` (default false): when true, sessions matching the exact OLD bare-prefix
|
||||
// shape (LEGACY_SESSION_NAME_RE) are ALSO treated as ours for kill-session purposes. This is
|
||||
// the boot-time legacy migration: an operator upgrading past this fix could otherwise be left
|
||||
@@ -88,14 +118,20 @@ const defaultTmux = (args, opts = {}) =>
|
||||
// same class of residual risk the audit finding itself accepts ("no live instance of the new
|
||||
// version creates them"); this PR does not regress that scenario, it only removes the far
|
||||
// more common same-version collision (the actual F7 finding).
|
||||
export function reapStaleTuiSessions({ tmux = defaultTmux, port, includeLegacy = false } = {}) {
|
||||
export function reapStaleTuiSessions({ tmux = defaultTmux, port, includeLegacy = false, spare = null } = {}) {
|
||||
const r = tmux(["list-sessions", "-F", "#{session_name}"]);
|
||||
if (!r || r.status !== 0) return 0; // no tmux server / no sessions
|
||||
const names = String(r.stdout || "").split("\n").map((s) => s.trim()).filter(Boolean);
|
||||
const ownPrefix = sessionPrefixForPort(port);
|
||||
const spared = spare instanceof Set ? spare : new Set(spare || []);
|
||||
let killed = 0;
|
||||
let othersRemain = false;
|
||||
let sparedLive = 0;
|
||||
for (const name of names) {
|
||||
// Property 1+2: exemption is by EXACT NAME from the pool's live registry. A pooled-
|
||||
// LOOKING name that is not in the registry is an orphan and falls through to the
|
||||
// normal kill path below.
|
||||
if (spared.has(name)) { sparedLive++; continue; }
|
||||
const isOwn = name.startsWith(ownPrefix);
|
||||
const isLegacyOwn = includeLegacy && LEGACY_SESSION_NAME_RE.test(name);
|
||||
if (isOwn || isLegacyOwn) {
|
||||
@@ -109,7 +145,11 @@ export function reapStaleTuiSessions({ tmux = defaultTmux, port, includeLegacy =
|
||||
// Reap defunct `claude` zombies: safe ONLY when the server is now ours-only/empty.
|
||||
// kill-server is what actually reaps (server exit reparents survivors to init); a
|
||||
// per-session kill cannot, since node is not the zombies' parent.
|
||||
if (!othersRemain) {
|
||||
//
|
||||
// Property 3: a SPARED session is a live child of this tmux server, so kill-server would
|
||||
// kill it — it therefore suppresses kill-server exactly as a foreign session does. On the
|
||||
// normal sweep the pool is drained first, so sparedLive is 0 and kill-server still fires.
|
||||
if (!othersRemain && sparedLive === 0) {
|
||||
tmux(["kill-server"]);
|
||||
}
|
||||
return killed;
|
||||
@@ -119,6 +159,12 @@ export function reapStaleTuiSessions({ tmux = defaultTmux, port, includeLegacy =
|
||||
|
||||
// Boot + paste-settle timing. Conservative defaults validated on PI231; env-tunable.
|
||||
const BOOT_MS = parseInt(process.env.OCP_TUI_BOOT_MS || "4000", 10); // max wait for input-ready
|
||||
// Readiness cap for a POOL pre-boot. Deliberately far more generous than BOOT_MS: BOOT_MS is
|
||||
// tight because a client is blocked on it, whereas a warm-pane boot happens in the background
|
||||
// with nobody waiting. Observed live at size=2: a refill booting alongside an in-flight turn
|
||||
// exceeded 4000 ms and was discarded (tui_pool_boot_failed), quietly costing hit rate for a
|
||||
// pane that was merely slow, not broken. Scales with OCP_TUI_BOOT_MS if an operator raises it.
|
||||
export const POOL_BOOT_MS = BOOT_MS * 5;
|
||||
const READY_POLL_MS = parseInt(process.env.OCP_TUI_READY_POLL_MS || "400", 10); // readiness / paste-verify poll interval
|
||||
const PASTE_VERIFY_MS = parseInt(process.env.OCP_TUI_PASTE_VERIFY_MS || "5000", 10); // max wait for pasted prompt to render
|
||||
|
||||
@@ -412,38 +458,60 @@ export function buildTuiCmd(claudeBin, model, sessionId, ehome, entrypointMode)
|
||||
].join(" ");
|
||||
}
|
||||
|
||||
// Full per-request TUI lifecycle:
|
||||
// 1. Pre-trust the scratch cwd (no trust dialog will appear).
|
||||
// 2. Write prompt to a 0600 temp file (no shell injection from prompt content).
|
||||
// 3. Boot an interactive `claude` in a fresh tmux session in the scratch cwd; poll
|
||||
// capture-pane until the `? for shortcuts` input bar appears (readiness-poll
|
||||
// replaces the old blind boot sleep). BOOT_MS is the max wait, not a fixed delay.
|
||||
// 4. Paste the prompt via tmux load-buffer + paste-buffer -p (bracketed paste) —
|
||||
// reliable for large multi-line prompts where send-keys -l is not (issue #130).
|
||||
// Poll-verify the prompt landed in the input (placeholder gone / [Pasted text]);
|
||||
// fast-fail with tui_paste_not_landed if it never lands (prevents the 120s
|
||||
// wallclock "stuck typing" hang). Then submit with a SEPARATE Enter key event.
|
||||
// 5. Block on the native JSONL transcript (located by session-id) until terminal
|
||||
// marker or wall-clock cap.
|
||||
// 6. Always teardown: kill session + rm temp dir (even on throw).
|
||||
// Returns { text, entrypoint } from readTuiTranscript (entrypoint is the billing-pool
|
||||
// classifier, e.g. "cli", or null if the transcript did not include a turn_duration).
|
||||
export async function runTuiTurn({
|
||||
prompt,
|
||||
model,
|
||||
claudeBin,
|
||||
home,
|
||||
realHome,
|
||||
cwd,
|
||||
port,
|
||||
wallclockMs = 120000,
|
||||
entrypointMode = "cli",
|
||||
tmux = defaultTmux,
|
||||
// Is a pane alive AND still sitting at its input bar? Used by the warm pool to decide,
|
||||
// at hand-out time, whether a pre-booted pane is still usable (a dead/degraded pane must
|
||||
// become a MISS → cold path, never a hung turn). capture-pane exits non-zero when the
|
||||
// session no longer exists, so this covers "pane gone" and "pane not ready" in one call.
|
||||
export function tuiPaneHealthy(tmux, tmuxName) {
|
||||
const r = tmux(["capture-pane", "-p", "-t", tmuxName]);
|
||||
if (!r || r.status !== 0 || typeof r.stdout !== "string") return false;
|
||||
return tuiInputReady(r.stdout);
|
||||
}
|
||||
|
||||
// Pool pane names carry a "p" marker after the port-scoped prefix:
|
||||
// turn pane: ocp-tui-<port>-<8hex> (unchanged)
|
||||
// pool pane: ocp-tui-<port>-p<8hex>
|
||||
// Purely for operator legibility (`tmux ls` shows which panes are warm). It is NOT the
|
||||
// reaper's exemption mechanism — that is the exact-name spare set (see the POOL/REAPER
|
||||
// INVARIANT above), so a pooled-LOOKING orphan is still reaped. Both shapes start with
|
||||
// sessionPrefixForPort(port), so both remain reapable as "ours", and neither can match
|
||||
// LEGACY_SESSION_NAME_RE.
|
||||
export function poolPaneName(port, sessionId) {
|
||||
return sessionPrefixForPort(port) + "p" + sessionId.slice(0, 8);
|
||||
}
|
||||
|
||||
// Boot ONE interactive `claude` pane and wait for its input bar. Shared by the cold
|
||||
// request path (runTuiTurn) and the warm pool (lib/tui/pool.mjs) so a pooled pane is
|
||||
// spawned with byte-for-byte the same argv, HOME, cwd and trust preparation as a
|
||||
// cold-booted one — the pool must not become a second, drifting spawn path.
|
||||
//
|
||||
// Each pane gets its OWN fresh randomUUID() --session-id, fixed at boot. That is what
|
||||
// keeps a pooled pane single-use-safe: its transcript holds exactly one exchange.
|
||||
//
|
||||
// requireReady: the cold path tolerates a readiness timeout (it falls through and lets
|
||||
// the paste-verify decide — pre-existing behaviour, unchanged). The POOL sets it, because
|
||||
// a pane that never reached its input bar is worthless as a warm pane and must not be
|
||||
// enlisted: throw, let the pool count a bootFailure, and leave the request path to
|
||||
// cold-boot as usual.
|
||||
// bootMs: max wait for the input bar. Defaults to BOOT_MS (the REQUEST path's cap, which is
|
||||
// deliberately tight — a client is blocked on it). The POOL passes POOL_BOOT_MS instead: a
|
||||
// background pre-boot has nobody waiting on it, and capping it at the request-path's 4 s
|
||||
// made real refills fail (observed live: a refill booting alongside an in-flight turn took
|
||||
// >4 s and was discarded, silently lowering the hit rate). Slow != broken for a pre-boot.
|
||||
// `sessionId` / `name` (both optional): the caller may supply the pane's identity instead of
|
||||
// letting bootTuiPane mint it. The POOL does, because it must know the tmux session's NAME
|
||||
// before this function runs — the session is created synchronously below, well before the
|
||||
// readiness wait returns, so a pool that only learned the name on resolve could neither spare
|
||||
// the session from the reaper nor kill it on shutdown. Supplying BOTH also keeps the name's
|
||||
// hex suffix equal to the session-id's, so `tmux ls` correlates to the transcript file.
|
||||
export async function bootTuiPane({
|
||||
model, claudeBin, home, realHome, cwd, port, entrypointMode = "cli",
|
||||
tmux = defaultTmux, sessionId = null, name = null, requireReady = false, bootMs = BOOT_MS,
|
||||
}) {
|
||||
const sessionId = randomUUID();
|
||||
const sid = sessionId || randomUUID();
|
||||
// Port-scoped session name (F7 fix) — see sessionPrefixForPort / reapStaleTuiSessions
|
||||
// for why this instance's own listen port is the namespace discriminator.
|
||||
const tmuxName = sessionPrefixForPort(port) + sessionId.slice(0, 8);
|
||||
const tmuxName = name || (sessionPrefixForPort(port) + sid.slice(0, 8));
|
||||
const ehome = home || process.env.HOME; // HOME claude runs under (scratch or real)
|
||||
const rhome = realHome || process.env.HOME; // real home (OAuth + onboarded config source)
|
||||
|
||||
@@ -460,42 +528,97 @@ export async function runTuiTurn({
|
||||
if (!existsSync(cwd)) mkdirSync(cwd, { recursive: true });
|
||||
prepareTuiHome(rhome, ehome, cwd, { envTokenMode });
|
||||
|
||||
// Write prompt to a temp file (mode 0600) so the content never touches argv.
|
||||
const tmpDir = mkdtempSync(`${tmpdir()}/ocp-tui-`);
|
||||
const promptFile = `${tmpDir}/prompt.txt`;
|
||||
writeFileSync(promptFile, prompt, { mode: 0o600 });
|
||||
|
||||
// Minimal env for spawnSync (tmux itself). The pane's claude env comes exclusively
|
||||
// from the `env` prefix string built inside buildTuiCmd — tmux does NOT forward the
|
||||
// spawning process's env to the pane, so the {env} here is intentionally minimal.
|
||||
const env = { ...process.env };
|
||||
env.HOME = ehome; // tmux needs HOME; all claude-specific vars go via buildTuiCmd prefix
|
||||
|
||||
// Boot the interactive session inside tmux, rooted at the scratch cwd.
|
||||
// Capture the result: if tmux new-session fails (status !== 0) there is no PTY, no
|
||||
// interactive spawn — abort BEFORE the boot wait rather than paste into a non-existent
|
||||
// session or issue a billing request without a verified interactive context.
|
||||
const spawnResult = tmux(
|
||||
["new-session", "-d", "-s", tmuxName, "-x", "220", "-y", "50", "-c", cwd,
|
||||
buildTuiCmd(claudeBin, model, sid, ehome, entrypointMode)],
|
||||
{ env },
|
||||
);
|
||||
if (!spawnResult || spawnResult.status !== 0) {
|
||||
throw new Error("tui_spawn_failed: tmux session not created");
|
||||
}
|
||||
|
||||
// Wait until claude's input bar is actually ready (not a blind sleep).
|
||||
// bootMs is the MAX readiness wait, not a fixed delay.
|
||||
const ready = await pollUntil(() => tuiInputReady(tuiCapturePane(tmux, tmuxName)),
|
||||
{ timeoutMs: bootMs, intervalMs: READY_POLL_MS });
|
||||
if (!ready) {
|
||||
if (requireReady) {
|
||||
try { tmux(["kill-session", "-t", tmuxName]); } catch { /* already gone */ }
|
||||
throw new Error("tui_pane_not_ready: input bar did not appear within " + bootMs + "ms");
|
||||
}
|
||||
// Cold path (pre-existing behaviour): readiness timed out; rely on paste-verify.
|
||||
console.error("[tui] input_not_ready", tmuxName);
|
||||
}
|
||||
return { name: tmuxName, sessionId: sid, model, ehome, bootedAt: Date.now() };
|
||||
}
|
||||
|
||||
// Full per-request TUI lifecycle:
|
||||
// 1. Take a WARM pane from the pool if one is available for this model (opt-in;
|
||||
// OCP_TUI_POOL_SIZE=0 => always null => steps 2-3 below are exactly today's path).
|
||||
// A pooled pane is SINGLE-USE: it already carries its own fresh --session-id, it
|
||||
// serves this one turn, and it is killed in the finally like any other pane.
|
||||
// 2. On a MISS: pre-trust the scratch cwd, boot an interactive `claude` in a fresh tmux
|
||||
// session in the scratch cwd, poll capture-pane until the `? for shortcuts` input bar
|
||||
// appears (bootTuiPane). BOOT_MS is the max wait, not a fixed delay.
|
||||
// 3. Write prompt to a 0600 temp file (no shell injection from prompt content).
|
||||
// 4. Paste the prompt via tmux load-buffer + paste-buffer -p (bracketed paste) —
|
||||
// reliable for large multi-line prompts where send-keys -l is not (issue #130).
|
||||
// Poll-verify the prompt landed in the input (placeholder gone / [Pasted text]);
|
||||
// fast-fail with tui_paste_not_landed if it never lands (prevents the 120s
|
||||
// wallclock "stuck typing" hang). Then submit with a SEPARATE Enter key event.
|
||||
// 5. Block on the native JSONL transcript (located by THIS pane's session-id) until
|
||||
// terminal marker or wall-clock cap.
|
||||
// 6. Always teardown: kill session + rm temp dir (even on throw), and kick a background
|
||||
// pool refill so the next request finds a warm pane.
|
||||
// Returns { text, entrypoint } from readTuiTranscript (entrypoint is the billing-pool
|
||||
// classifier, e.g. "cli", or null if the transcript did not include a turn_duration).
|
||||
export async function runTuiTurn({
|
||||
prompt,
|
||||
model,
|
||||
claudeBin,
|
||||
home,
|
||||
realHome,
|
||||
cwd,
|
||||
port,
|
||||
wallclockMs = 120000,
|
||||
entrypointMode = "cli",
|
||||
tmux = defaultTmux,
|
||||
pool = null, // TuiPanePool | null — null (default) === today's cold-boot-only path
|
||||
onPane = null, // optional observer: ({ warm }) => void, for logging/metrics
|
||||
}) {
|
||||
// 1. Warm pane, or cold boot. A MISS is never an error — it is exactly today's path.
|
||||
let pane = pool ? pool.acquire(model) : null;
|
||||
const warm = !!pane;
|
||||
// Kick the refill IMMEDIATELY (not after the turn): the replacement pane then boots
|
||||
// CONCURRENTLY with this turn and is warm by the time the next request arrives. Also
|
||||
// runs on a MISS — acquire() has just retargeted the pool to this model, so the miss
|
||||
// that cold-boots today warms the pool for the next caller. Fire-and-forget; it takes
|
||||
// no TuiSemaphore slot (see pool.refill's SLOT ACCOUNTING note).
|
||||
if (pool) pool.refill();
|
||||
if (onPane) { try { onPane({ warm }); } catch { /* observer must never break a turn */ } }
|
||||
if (!pane) {
|
||||
pane = await bootTuiPane({ model, claudeBin, home, realHome, cwd, port, entrypointMode, tmux });
|
||||
}
|
||||
const tmuxName = pane.name;
|
||||
const sessionId = pane.sessionId; // THIS pane's own session-id — one session, one turn
|
||||
const ehome = pane.ehome || home || process.env.HOME;
|
||||
|
||||
// Write prompt to a temp file (mode 0600) so the content never touches argv.
|
||||
const tmpDir = mkdtempSync(`${tmpdir()}/ocp-tui-`);
|
||||
const promptFile = `${tmpDir}/prompt.txt`;
|
||||
writeFileSync(promptFile, prompt, { mode: 0o600 });
|
||||
|
||||
try {
|
||||
// 1. Boot the interactive session inside tmux, rooted at the scratch cwd.
|
||||
// Capture the result: if tmux new-session fails (status !== 0) there is no
|
||||
// PTY, no interactive spawn — abort BEFORE the boot sleep rather than paste
|
||||
// into a non-existent session or issue a billing request without a verified
|
||||
// interactive context. The finally teardown is still harmless (kill-session
|
||||
// is a no-op when the session never existed).
|
||||
const spawnResult = tmux(
|
||||
["new-session", "-d", "-s", tmuxName, "-x", "220", "-y", "50", "-c", cwd,
|
||||
buildTuiCmd(claudeBin, model, sessionId, ehome, entrypointMode)],
|
||||
{ env },
|
||||
);
|
||||
if (!spawnResult || spawnResult.status !== 0) {
|
||||
throw new Error("tui_spawn_failed: tmux session not created");
|
||||
}
|
||||
|
||||
// 2. Wait until claude's input bar is actually ready (was: blind sleep(BOOT_MS)).
|
||||
// BOOT_MS is now the MAX readiness wait, not a fixed delay.
|
||||
const ready = await pollUntil(() => tuiInputReady(tuiCapturePane(tmux, tmuxName)),
|
||||
{ timeoutMs: BOOT_MS, intervalMs: READY_POLL_MS });
|
||||
if (!ready) {
|
||||
// (readiness timed out; relying on paste-verify)
|
||||
console.error("[tui] input_not_ready", tmuxName);
|
||||
}
|
||||
|
||||
// 3. Paste the prompt via a tmux PASTE BUFFER with bracketed paste (-p), NOT
|
||||
// `send-keys -l`. send-keys of a large multi-line prompt is unreliable: the
|
||||
// embedded newlines arrive as separate key events (effectively repeated Enter),
|
||||
@@ -521,11 +644,12 @@ export async function runTuiTurn({
|
||||
// Submit (separate Enter key event).
|
||||
tmux(["send-keys", "-t", tmuxName, "Enter"]);
|
||||
|
||||
// 4. Block on the native transcript (resolved by session-id) until terminal.
|
||||
// 5. Block on the native transcript (resolved by THIS pane's session-id) until terminal.
|
||||
// Returns { text, entrypoint } from readTuiTranscript.
|
||||
return await readTuiTranscript({ home: ehome, sessionId, wallclockMs });
|
||||
} finally {
|
||||
// 5. Teardown — always, even on throw.
|
||||
// 6. Teardown — always, even on throw. A pooled pane is torn down here exactly like a
|
||||
// cold-booted one: SINGLE-USE, never returned to the pool (see pool.mjs).
|
||||
try { tmux(["kill-session", "-t", tmuxName]); } catch { /* already gone */ }
|
||||
try { rmSync(tmpDir, { recursive: true, force: true }); } catch { /* best effort */ }
|
||||
}
|
||||
|
||||
@@ -73,6 +73,17 @@ export function isTerminalLine(obj) {
|
||||
// transcript holding one logical exchange). If a future warm-pool ever reuses a
|
||||
// session WITHOUT a fresh session-id / clear, earlier-turn text could leak — that
|
||||
// author must add user-line scoping here. See spec §7.2.
|
||||
//
|
||||
// STATUS (warm pool, lib/tui/pool.mjs — the "future warm-pool" this note anticipated):
|
||||
// the pool does NOT reuse sessions, so the precondition above still holds and no
|
||||
// user-line scoping was added. Each pooled pane is booted with its OWN fresh
|
||||
// randomUUID() --session-id (bootTuiPane) and is SINGLE-USE: it serves exactly one turn
|
||||
// and is then killed and replaced. One session still means one logical exchange, so the
|
||||
// last assistant entry is still that request's answer.
|
||||
// The warning therefore stands UNCHANGED for anyone who later wants a pane to serve a
|
||||
// SECOND turn (or to reset one with /clear and reuse it): that is a leak, and it needs
|
||||
// user-line scoping HERE before it can be safe. Do not relax pool.mjs's single-use rule
|
||||
// without doing that work first.
|
||||
export function extractLatestAssistantText(events) {
|
||||
let text = "";
|
||||
for (const ev of events) {
|
||||
|
||||
+117
-5
@@ -22,6 +22,8 @@
|
||||
* CLAUDE_MAX_CONCURRENT — max concurrent claude processes, -p/stream-json path (default: 8)
|
||||
* CLAUDE_MAX_QUEUE — max requests waiting for a -p slot before HTTP 429 (default: 16)
|
||||
* OCP_TUI_MAX_CONCURRENT — max concurrent interactive TUI turns, TUI-mode path (default: 2)
|
||||
* OCP_TUI_POOL_SIZE — pre-booted warm `claude` panes held for TUI-mode (default: 0 = off;
|
||||
* max 4). Each is a live idle process; cuts ~3-4s per request.
|
||||
* OCP_SPAWN_REAL_HOME — "1" forces the -p spawn to use the real HOME (disables the
|
||||
* latency spawn-home isolation; default: isolated when a token exists)
|
||||
* CLAUDE_BREAKER_THRESHOLD — failures in window before circuit opens (default: 6)
|
||||
@@ -32,7 +34,7 @@
|
||||
* CLAUDE_HEARTBEAT_INTERVAL — SSE heartbeat interval in ms on streaming path (default: 0 = disabled)
|
||||
*/
|
||||
import { createServer } from "node:http";
|
||||
import { spawn, execFileSync } from "node:child_process";
|
||||
import { spawn, execFileSync, spawnSync } from "node:child_process";
|
||||
import { randomUUID, timingSafeEqual } from "node:crypto";
|
||||
import { readFileSync, readdirSync, accessSync, existsSync, constants, chmodSync, statSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
@@ -41,9 +43,10 @@ import { homedir } from "node:os";
|
||||
import { validateKey, recordUsage, getUsageByKey, getUsageTimeline, getRecentUsage, createKey, listKeys, revokeKey, closeDb, checkQuota, updateKeyQuota, getKeyQuota, findKey, cacheHash, getCachedResponse, setCachedResponse, clearCache, getCacheStats, hasCacheControl, singleflight, getInflightStats } from "./keys.mjs";
|
||||
import { DEFAULT_PORT } from "./lib/constants.mjs";
|
||||
import { isLoopbackBind } from "./lib/net.mjs";
|
||||
import { runTuiTurn, reapStaleTuiSessions, resolveTuiHome } from "./lib/tui/session.mjs";
|
||||
import { runTuiTurn, reapStaleTuiSessions, resolveTuiHome, bootTuiPane, tuiPaneHealthy, poolPaneName, POOL_BOOT_MS } from "./lib/tui/session.mjs";
|
||||
import { detectTuiUpstreamError } from "./lib/tui/transcript.mjs";
|
||||
import { TuiSemaphore, SemaphoreAbortError, recordTuiEntrypoint, buildTuiHealthBlock } from "./lib/tui/semaphore.mjs";
|
||||
import { TuiPanePool, resolvePoolSize, POOL_MAX_SIZE } from "./lib/tui/pool.mjs";
|
||||
import { createSerialMutex, createTtlCache, isTokenExpiring, orderLabelsLastGoodFirst } from "./lib/spawn-auth.mjs";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
@@ -351,6 +354,48 @@ const tuiStats = {
|
||||
entrypointMismatches: 0, // count of cli-expected-but-got-other turns
|
||||
};
|
||||
|
||||
// ── Warm pane pool (docs/plans/2026-07-13-tui-latency #3) — opt-in; default OFF ─────────
|
||||
// OCP_TUI_POOL_SIZE=0 (default) => tuiPool is null => runTuiTurn's cold-boot path is
|
||||
// byte-for-byte unchanged. Set it to N (clamped to POOL_MAX_SIZE) to keep N pre-booted
|
||||
// `claude` panes warm, each SINGLE-USE (see lib/tui/pool.mjs for why single-use is the
|
||||
// load-bearing rule, and lib/tui/session.mjs for the POOL/REAPER INVARIANT).
|
||||
//
|
||||
// Default-off is deliberate on a stable production path: a warm pane is a LIVE idle
|
||||
// `claude` process held whether or not a request ever arrives, so the operator must opt
|
||||
// in to that standing cost. Measured saving when on (this host, Sonnet 4.6, --effort low):
|
||||
// end-to-end p50 10.17 s (n=6, pool off) -> 6.00 s (n=12 warm hits), i.e. -41%.
|
||||
// cli.js does NOT perform this operation (Class B, OCP-owned TUI spawn) — see ADR 0007.
|
||||
const TUI_POOL_SIZE = TUI_MODE ? resolvePoolSize(process.env.OCP_TUI_POOL_SIZE) : 0;
|
||||
const tuiPool = TUI_POOL_SIZE > 0
|
||||
? new TuiPanePool({
|
||||
size: TUI_POOL_SIZE,
|
||||
// The POOL mints the pane's identity, not bootTuiPane: the tmux session exists the
|
||||
// instant the boot starts, so the pool must be able to name (hence spare, hence kill)
|
||||
// it before then. Name is derived from the session-id, so `tmux ls` correlates to the
|
||||
// transcript file <HOME>/.claude/projects/*/<sessionId>.jsonl.
|
||||
mintPane: () => {
|
||||
const sessionId = randomUUID();
|
||||
return { sessionId, name: poolPaneName(PORT, sessionId) };
|
||||
},
|
||||
bootPane: (model, ident) => bootTuiPane({
|
||||
model,
|
||||
claudeBin: CLAUDE,
|
||||
home: TUI_HOME,
|
||||
realHome: process.env.HOME,
|
||||
cwd: TUI_CWD,
|
||||
port: PORT,
|
||||
entrypointMode: TUI_ENTRYPOINT,
|
||||
sessionId: ident.sessionId,
|
||||
name: ident.name,
|
||||
requireReady: true, // a pane that never reached its input bar must not be enlisted
|
||||
bootMs: POOL_BOOT_MS, // background pre-boot — no client is blocked, so be patient
|
||||
}),
|
||||
killPane: (name) => { try { spawnSync(process.env.OCP_TUI_TMUX_BIN || "tmux", ["kill-session", "-t", name]); } catch { /* already gone */ } },
|
||||
paneHealthy: (name) => tuiPaneHealthy((args) => spawnSync(process.env.OCP_TUI_TMUX_BIN || "tmux", args, { encoding: "utf8" }), name),
|
||||
log: (level, event, data) => logEvent(level, event, data),
|
||||
})
|
||||
: null;
|
||||
|
||||
// ── FIX ③ (latency): default-path (-p / stream-json) spawn-home isolation ──────────────
|
||||
// PROBLEM (measured, not theoretical): OCP's default spawn inherits the operator's real HOME
|
||||
// (loading the global ~/.claude — plugins, skills, hooks) and runs with cwd=~/ocp (loading the
|
||||
@@ -770,19 +815,45 @@ const cacheCleanupInterval = setInterval(() => {
|
||||
// mechanism and the 15-min cadence makes the window negligible).
|
||||
// Gated on TUI_MODE — zero effect (no kill-server, no list-sessions) when TUI is off.
|
||||
// cli.js does NOT perform this operation (Class B, OCP-owned TUI spawn) — see ADR 0007.
|
||||
//
|
||||
// WARM POOL INTERACTION (the crux — see the POOL/REAPER INVARIANT in lib/tui/session.mjs).
|
||||
// A warm pooled pane is one of OUR OWN ocp-tui-<port>-* sessions that is alive and idle BY
|
||||
// DESIGN, and this sweep fires precisely when the instance is idle — i.e. exactly when the
|
||||
// pool is full. Two things are therefore required, and both are done here:
|
||||
// (a) DRAIN the pool BEFORE the sweep. Zombie reaping is possible ONLY via kill-server,
|
||||
// and a live pooled pane suppresses kill-server (it is a live child of the tmux
|
||||
// server). A permanently-full pool would otherwise permanently disable the very
|
||||
// thing this tick exists to do. Draining costs one pane re-boot per tick (~1.2 s of
|
||||
// background work every 15 min) and is invisible to callers: a request landing in the
|
||||
// drain→refill gap simply MISSES the pool and takes today's cold path.
|
||||
// (b) Pass the pool's live registry as `spare` anyway. After (a) it is empty, so this is
|
||||
// belt-and-braces — it makes it impossible for THIS call site (or a future one) to
|
||||
// kill a live pooled pane even if the drain were ever removed or reordered.
|
||||
// RESIDUAL (unchanged in kind from the pre-pool code, and explicitly accepted there): a
|
||||
// request arriving in the narrow window between the idle-check and kill-server has its pane
|
||||
// torn down and fails cleanly via runTuiTurn's honesty gates. The drain widens that window
|
||||
// by the cost of N kill-session calls (single-digit ms), not materially.
|
||||
const TUI_REAP_INTERVAL_MS = 15 * 60 * 1000;
|
||||
const tuiReapInterval = TUI_MODE ? setInterval(() => {
|
||||
if (tuiSemaphore.inflight > 0 || tuiSemaphore.queued > 0) return; // a turn is live — defer
|
||||
try {
|
||||
const drained = tuiPool ? tuiPool.drain() : 0;
|
||||
// F7 fix: scope to THIS instance's own port; a sibling ocp-tui-<otherPort>-* session
|
||||
// (a second OCP instance on the same host) is treated as foreign, same as olp-tui-*.
|
||||
// includeLegacy is NOT set here — see reapStaleTuiSessions' comment: the periodic sweep
|
||||
// conservatively treats any lingering bare-prefix legacy session as foreign so it can
|
||||
// never trigger kill-server on a steady-state tick; only the one-time boot reap below
|
||||
// claims legacy-shaped zombies.
|
||||
const n = reapStaleTuiSessions({ port: PORT });
|
||||
if (n) logEvent("info", "tui_reaped_stale_sessions", { count: n, trigger: "periodic" });
|
||||
const n = reapStaleTuiSessions({ port: PORT, spare: tuiPool ? tuiPool.liveNames() : null });
|
||||
if (n || drained) {
|
||||
logEvent("info", "tui_reaped_stale_sessions", { count: n, poolDrained: drained, trigger: "periodic" });
|
||||
}
|
||||
} catch (e) { logEvent("error", "tui_periodic_reap_failed", { error: e.message }); }
|
||||
finally {
|
||||
// Refill in the background regardless of how the sweep went — a throw mid-sweep must not
|
||||
// leave the pool permanently paused (it would silently degrade to the cold path forever).
|
||||
if (tuiPool) { try { tuiPool.resume(); } catch { /* best effort */ } }
|
||||
}
|
||||
}, TUI_REAP_INTERVAL_MS) : null;
|
||||
if (tuiReapInterval && typeof tuiReapInterval.unref === "function") tuiReapInterval.unref();
|
||||
|
||||
@@ -1323,6 +1394,15 @@ async function callClaudeTui(model, messages, _conversationId, _keyName, res) {
|
||||
// different port never collides with this instance's reap/kill-server logic.
|
||||
wallclockMs: TUI_WALLCLOCK_MS,
|
||||
entrypointMode: TUI_ENTRYPOINT,
|
||||
// Warm pane pool (null unless OCP_TUI_POOL_SIZE > 0 → today's cold path exactly).
|
||||
// A pooled pane is single-use: runTuiTurn kills it in its finally like any other.
|
||||
pool: tuiPool,
|
||||
// Only observe when the pool is ON — with it off (the default) no new log line is
|
||||
// emitted, so the disabled path stays byte-for-byte today's, logs included.
|
||||
onPane: tuiPool
|
||||
? ({ warm }) => logEvent("info", warm ? "tui_pool_hit" : "tui_pool_miss",
|
||||
{ model: cliModel, warmRemaining: tuiPool.warm })
|
||||
: null,
|
||||
});
|
||||
// ── Honesty gates (issue #133) ─ run BEFORE recordModelSuccess / cache write-back.
|
||||
// A throw here propagates to the catch below (recordModelError + reject), so the
|
||||
@@ -2558,9 +2638,12 @@ const server = createServer(async (req, res) => {
|
||||
// still appears with enabled:false (cheap, harmless) so the shape is stable.
|
||||
// entrypointMismatches/lastEntrypoint exist so an operator can poll /health to catch a
|
||||
// silent metered-pool drift (the audit's top risk after the 6/15 billing flip).
|
||||
// `pool` is a NEW nested field inside the (already additive) tui block: null when the
|
||||
// warm pool is off (the default), so the disabled shape is unchanged apart from one
|
||||
// explicit null. Lets the operator confirm hit rate + standing process cost.
|
||||
tui: buildTuiHealthBlock(
|
||||
{ enabled: TUI_MODE, entrypointMode: TUI_ENTRYPOINT, maxConcurrent: TUI_MAX_CONCURRENT },
|
||||
tuiStats, tuiSemaphore,
|
||||
tuiStats, tuiSemaphore, tuiPool,
|
||||
),
|
||||
});
|
||||
}
|
||||
@@ -2797,6 +2880,27 @@ function gracefulShutdown(signal) {
|
||||
if (tuiReapInterval) clearInterval(tuiReapInterval);
|
||||
closeDb();
|
||||
|
||||
// 2b. Drain the warm pane pool. A pooled `claude` is a child of the tmux SERVER, not of
|
||||
// this node process, so it is NOT in activeProcesses and step 3 below cannot reach it —
|
||||
// without this explicit drain every warm pane would outlive OCP as an orphan (and the
|
||||
// pool's in-memory registry dies with the process, so nothing would remember it owned them).
|
||||
//
|
||||
// drain() kills the pane that is currently BOOTING too, and it does so SYNCHRONOUSLY. That
|
||||
// is required, not incidental: step 4 below calls process.exit(0) in THIS SAME TICK whenever
|
||||
// activeProcesses is empty — which on a TUI host it always is — so any cleanup a boot
|
||||
// deferred to a .then()/.catch() would simply never run. (That was a real bug: the pool used
|
||||
// to track in-flight boots as a count, could not name the booting session, and orphaned a
|
||||
// live authenticated `claude` on every shutdown that landed mid-boot.)
|
||||
//
|
||||
// Orphans that survive anyway (SIGKILL, power loss) are still caught by the next instance's
|
||||
// boot reap — this makes the graceful path clean, it is not the only safety net.
|
||||
if (tuiPool) {
|
||||
try {
|
||||
const drained = tuiPool.drain();
|
||||
if (drained) logEvent("info", "tui_pool_drained", { count: drained, trigger: "shutdown" });
|
||||
} catch (e) { logEvent("error", "tui_pool_drain_failed", { error: e.message }); }
|
||||
}
|
||||
|
||||
// 3. Kill all active child processes
|
||||
for (const proc of activeProcesses) {
|
||||
try { proc.kill("SIGTERM"); } catch {}
|
||||
@@ -2866,11 +2970,19 @@ server.listen(PORT, BIND_ADDRESS, () => {
|
||||
? (TUI_HOME === process.env.HOME ? "env-token (real home — unset OCP_TUI_HOME for credential isolation)" : "env-token (credential-isolated home — no credentials.json)")
|
||||
: "credentials.json (no CLAUDE_CODE_OAUTH_TOKEN — see Troubleshooting #401)";
|
||||
console.log(` TUI-mode: ON home=${TUI_HOME} cwd=${TUI_CWD} auth=${tuiAuth} wallclock=${TUI_WALLCLOCK_MS}ms maxConcurrent=${TUI_MAX_CONCURRENT}`);
|
||||
console.log(TUI_POOL_SIZE > 0
|
||||
? ` TUI warm pool: ON size=${TUI_POOL_SIZE} — ${TUI_POOL_SIZE} idle \`claude\` process(es) held warm; first request per model is still a cold MISS`
|
||||
: ` TUI warm pool: OFF (set OCP_TUI_POOL_SIZE=1..${POOL_MAX_SIZE} to pre-boot panes and cut ~3-4s per request)`);
|
||||
try {
|
||||
// F7 fix: scope to THIS instance's own port (see reapStaleTuiSessions). includeLegacy:
|
||||
// true ONLY here — the one-time boot reap is the designated point to claim orphaned
|
||||
// bare-prefix ("ocp-tui-<uuid8>") zombie sessions left by a PRE-fix process generation
|
||||
// of this same instance (no live post-fix instance ever creates that shape again).
|
||||
// No `spare`: the warm pool is EMPTY at boot (there is no boot-time pre-warm — the pool
|
||||
// learns its model from the first request), so this reap has no live pane to protect and
|
||||
// it is exactly what SHOULD claim any ocp-tui-<port>-p* pool orphans left by a previous
|
||||
// process generation of this instance (POOL/REAPER INVARIANT property 2). If a future
|
||||
// change ever pre-warms at boot, this call MUST start passing tuiPool.liveNames().
|
||||
const n = reapStaleTuiSessions({ port: PORT, includeLegacy: true });
|
||||
if (n) logEvent("info", "tui_reaped_stale_sessions", { count: n });
|
||||
} catch {}
|
||||
|
||||
+514
-3
@@ -23,9 +23,30 @@ process.env.HOME = homedir(); // ensure consistent
|
||||
let passed = 0;
|
||||
let failed = 0;
|
||||
|
||||
// Pending promises from tests declared `async` but registered through the SYNC `test()` helper.
|
||||
// 44 tests in this file are written that way. Before this, `test()` called fn(), got a promise back,
|
||||
// and immediately printed ✓ and incremented `passed` — WITHOUT AWAITING IT. So for every async test:
|
||||
// - ✓ meant "did not throw synchronously", NOT "passed";
|
||||
// - a failed assertion escaped as an unhandled rejection, which crashes the process (CI still goes
|
||||
// red on the non-zero exit) but is NOT counted, so the summary could print "N passed, 0 failed"
|
||||
// and be wrong.
|
||||
// The suite's own headline number was therefore not evidence for any async test — including the
|
||||
// regression guards in this PR. Collected here and awaited before the summary prints.
|
||||
const pendingAsync = [];
|
||||
|
||||
function test(name, fn) {
|
||||
try {
|
||||
fn();
|
||||
const r = fn();
|
||||
if (r && typeof r.then === "function") {
|
||||
// Async body: settle it before counting. Do NOT print ✓ yet.
|
||||
pendingAsync.push(
|
||||
r.then(
|
||||
() => { passed++; console.log(` ✓ ${name}`); },
|
||||
(e) => { failed++; console.log(` ✗ ${name}: ${e.message}`); },
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
passed++;
|
||||
console.log(` ✓ ${name}`);
|
||||
} catch (e) {
|
||||
@@ -2027,6 +2048,489 @@ test("reaper with includeLegacy=true still spares a sibling instance's port-scop
|
||||
assert.ok(!calls.includes("kill-server"), "sibling instance's live session still blocks kill-server");
|
||||
});
|
||||
|
||||
// ── TUI warm pane pool (docs/plans/2026-07-13-tui-latency #3) ────────────
|
||||
import { TuiPanePool, resolvePoolSize, POOL_MAX_SIZE, POOL_MAX_AGE_MS } from "./lib/tui/pool.mjs";
|
||||
import { poolPaneName as poolName } from "./lib/tui/session.mjs";
|
||||
|
||||
// A pool wired to fakes: no tmux, no claude. bootPane resolves on the microtask queue; use
|
||||
// `await settle()` after a refill() to let the SERIALIZED boot chain run to target.
|
||||
// `live` models the real tmux server: bootTuiPane creates the session SYNCHRONOUSLY and only
|
||||
// THEN waits (up to POOL_BOOT_MS) for the input bar, so the fake boot registers the session
|
||||
// immediately and only afterwards resolves. `opts.hold` keeps a boot in that mid-flight window
|
||||
// so tests can act on a pane that is live-but-not-yet-warm — the state that hid two bugs.
|
||||
function makeFakePool(opts = {}) {
|
||||
const killed = [];
|
||||
const booted = [];
|
||||
const live = new Set(); // "tmux sessions" that currently exist
|
||||
let seq = 0;
|
||||
let clock = 1_000_000;
|
||||
const healthy = new Set();
|
||||
// FIFO gate queue — one entry per in-flight held boot. A single `release` slot would be
|
||||
// OVERWRITTEN by a later boot, so releasing "the first boot" would silently release the
|
||||
// second instead (and mask the stale-settle bug this harness exists to test).
|
||||
const gates = [];
|
||||
const pool = new TuiPanePool({
|
||||
size: opts.size ?? 2,
|
||||
maxAgeMs: opts.maxAgeMs ?? POOL_MAX_AGE_MS,
|
||||
now: () => clock,
|
||||
mintPane: () => {
|
||||
const n = ++seq;
|
||||
return { sessionId: `sid-${n}`, name: `ocp-tui-3456-p${String(n).padStart(8, "0")}` };
|
||||
},
|
||||
bootPane: async (model, { sessionId, name }) => {
|
||||
live.add(name); // session exists NOW
|
||||
booted.push({ name, model });
|
||||
if (opts.hold) {
|
||||
await new Promise((r) => gates.push(r)); // ...stuck waiting for readiness
|
||||
}
|
||||
if (opts.bootThrows) { live.delete(name); throw new Error("boom"); }
|
||||
// A pane whose session was killed while booting can never become ready — exactly what
|
||||
// the real bootTuiPane does (it throws tui_pane_not_ready).
|
||||
if (!live.has(name)) throw new Error("tui_pane_not_ready");
|
||||
healthy.add(name);
|
||||
return { name, sessionId, model, bootedAt: clock };
|
||||
},
|
||||
killPane: (name) => { killed.push(name); healthy.delete(name); live.delete(name); },
|
||||
paneHealthy: (name) => healthy.has(name),
|
||||
});
|
||||
return {
|
||||
pool, killed, booted, healthy, live,
|
||||
releaseBoot: () => { const r = gates.shift(); if (r) r(); }, // release the OLDEST held boot
|
||||
advance: (ms) => { clock += ms; }, at: () => clock,
|
||||
};
|
||||
}
|
||||
const tick = () => new Promise((r) => setImmediate(r));
|
||||
// Refills are SERIALIZED (one boot at a time, re-kicked on success), so settling the pool
|
||||
// takes a chain of microtask turns, not one. 40 is far more than POOL_MAX_SIZE needs.
|
||||
const settle = async () => { for (let i = 0; i < 40; i++) await tick(); };
|
||||
|
||||
console.log("\nTUI warm pane pool (acquire / miss / refill / TTL / reaper exemption):");
|
||||
|
||||
test("resolvePoolSize: default/garbage/negative disable the pool; size is clamped to POOL_MAX_SIZE", () => {
|
||||
assert.equal(resolvePoolSize(undefined), 0, "unset => off (byte-for-byte today's cold path)");
|
||||
assert.equal(resolvePoolSize("0"), 0);
|
||||
assert.equal(resolvePoolSize("-3"), 0);
|
||||
assert.equal(resolvePoolSize("banana"), 0, "garbage disables rather than guessing a size");
|
||||
assert.equal(resolvePoolSize("2"), 2);
|
||||
assert.equal(resolvePoolSize("99"), POOL_MAX_SIZE, "clamped — never boot an unbounded number of idle claudes");
|
||||
});
|
||||
|
||||
test("pool size 0 is inert: acquire always misses and refill never boots", async () => {
|
||||
const { pool, booted } = makeFakePool({ size: 0 });
|
||||
assert.equal(pool.enabled, false);
|
||||
assert.equal(pool.acquire("m1"), null, "disabled pool always MISSES → caller cold-boots");
|
||||
pool.refill();
|
||||
await settle();
|
||||
assert.equal(booted.length, 0, "a disabled pool must never spawn a process");
|
||||
});
|
||||
|
||||
test("acquire MISSES on an empty pool, and the miss refills for the requested model", async () => {
|
||||
const { pool, booted } = makeFakePool({ size: 2 });
|
||||
assert.equal(pool.acquire("sonnet"), null, "first request is always a MISS (no boot-time pre-warm)");
|
||||
assert.equal(pool.misses, 1);
|
||||
pool.refill();
|
||||
await settle();
|
||||
assert.equal(pool.warm, 2, "refilled to target");
|
||||
assert.deepEqual(booted.map((b) => b.model), ["sonnet", "sonnet"], "warmed for the model that missed");
|
||||
});
|
||||
|
||||
test("acquire HITS a warm pane, hands it out ONCE, and never returns it (single-use)", async () => {
|
||||
const { pool } = makeFakePool({ size: 2 });
|
||||
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||
assert.equal(pool.warm, 2);
|
||||
|
||||
const a = pool.acquire("sonnet");
|
||||
assert.ok(a && a.name && a.sessionId, "warm pane handed out");
|
||||
assert.equal(pool.hits, 1);
|
||||
assert.equal(pool.warm, 1, "the pane LEAVES the registry when acquired");
|
||||
|
||||
const b = pool.acquire("sonnet");
|
||||
assert.notEqual(b.name, a.name, "a pane is NEVER handed out twice — single-use");
|
||||
assert.notEqual(b.sessionId, a.sessionId, "each pane carries its OWN fresh session-id (transcript.mjs scoping)");
|
||||
assert.equal(pool.warm, 0);
|
||||
assert.equal(pool.acquire("sonnet"), null, "exhausted pool MISSES rather than reusing a pane");
|
||||
});
|
||||
|
||||
test("refill is bounded: never more than `size` panes, and concurrent refills do not overshoot", async () => {
|
||||
const { pool, booted } = makeFakePool({ size: 2 });
|
||||
pool.acquire("sonnet");
|
||||
pool.refill(); pool.refill(); pool.refill(); // hammer it
|
||||
await settle();
|
||||
assert.equal(pool.warm, 2, "still exactly `size` warm panes");
|
||||
assert.equal(booted.length, 2, "the _booting guard prevented duplicate boots");
|
||||
});
|
||||
|
||||
// Live finding at size=2: two cold `claude` boots racing an in-flight turn made a refill
|
||||
// overrun even the generous pool readiness cap. Boots are therefore SERIALIZED.
|
||||
test("refill boots panes ONE AT A TIME (never two claude cold-boots racing each other)", async () => {
|
||||
let concurrent = 0, peak = 0;
|
||||
let seq = 0;
|
||||
const pool = new TuiPanePool({
|
||||
size: 3,
|
||||
mintPane: () => { const n = ++seq; return { sessionId: `s${n}`, name: `p${n}` }; },
|
||||
bootPane: async (model, { sessionId, name }) => {
|
||||
concurrent++; peak = Math.max(peak, concurrent);
|
||||
await new Promise((r) => setImmediate(r)); // simulate boot latency
|
||||
concurrent--;
|
||||
return { name, sessionId, model, bootedAt: Date.now() };
|
||||
},
|
||||
killPane: () => {},
|
||||
paneHealthy: () => true,
|
||||
});
|
||||
pool.acquire("sonnet");
|
||||
pool.refill();
|
||||
await settle();
|
||||
assert.equal(pool.warm, 3, "chain still reaches the target size");
|
||||
assert.equal(peak, 1, "at most ONE boot in flight at any moment");
|
||||
});
|
||||
|
||||
test("a FAILED boot does not re-kick the chain (backoff — a broken claude must not spin)", async () => {
|
||||
const { pool, booted } = makeFakePool({ size: 3, bootThrows: true });
|
||||
pool.acquire("sonnet");
|
||||
pool.refill();
|
||||
await settle();
|
||||
assert.equal(pool.bootFailures, 1, "counted as a genuine failure (nobody cancelled it)");
|
||||
assert.equal(booted.length, 1, "exactly ONE attempt — a failure stops the chain, it does not respawn forever");
|
||||
assert.equal(pool.warm, 0);
|
||||
assert.equal(pool.booting, 0, "and the booting slot is released, so the next trigger can retry");
|
||||
});
|
||||
|
||||
test("acquire drops an UNHEALTHY warm pane (kills it) and falls through to a MISS", async () => {
|
||||
const { pool, killed, healthy, booted } = makeFakePool({ size: 1 });
|
||||
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||
const dead = booted[0].name;
|
||||
healthy.delete(dead); // pane died / stopped being input-ready while idle
|
||||
|
||||
assert.equal(pool.acquire("sonnet"), null, "a dead pane must MISS, never hang a turn");
|
||||
assert.ok(killed.includes(dead), "the dead pane is killed, not leaked");
|
||||
assert.equal(pool.misses, 2);
|
||||
});
|
||||
|
||||
test("acquire drops an EXPIRED warm pane (older than maxAgeMs)", async () => {
|
||||
const { pool, killed, booted, advance } = makeFakePool({ size: 1, maxAgeMs: 60_000 });
|
||||
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||
advance(60_001);
|
||||
assert.equal(pool.acquire("sonnet"), null, "a pane past its TTL is not handed out");
|
||||
assert.ok(killed.includes(booted[0].name), "expired pane is killed");
|
||||
});
|
||||
|
||||
test("a model switch drops the wrong-model panes and retargets the pool (--model is fixed at spawn)", async () => {
|
||||
const { pool, killed, booted } = makeFakePool({ size: 2 });
|
||||
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||
const sonnetPanes = booted.map((b) => b.name);
|
||||
|
||||
assert.equal(pool.acquire("opus"), null, "different model => MISS (a sonnet pane cannot serve opus)");
|
||||
assert.equal(pool.warm, 0, "sonnet panes dropped");
|
||||
for (const p of sonnetPanes) assert.ok(killed.includes(p), "wrong-model pane killed, not leaked");
|
||||
assert.equal(pool.warmModel, "opus", "pool retargeted to the model actually being asked for");
|
||||
|
||||
pool.refill(); await settle();
|
||||
assert.deepEqual(booted.slice(2).map((b) => b.model), ["opus", "opus"], "refilled for the NEW model");
|
||||
});
|
||||
|
||||
test("a boot that resolves AFTER a drain kills its own pane instead of enlisting it", async () => {
|
||||
const { pool, live } = makeFakePool({ size: 1 });
|
||||
pool.acquire("sonnet");
|
||||
pool.refill(); // boot is in flight...
|
||||
pool.drain(); // ...pool drained before it resolves (shutdown / reap sweep)
|
||||
await settle();
|
||||
assert.equal(pool.warm, 0, "the late pane must NOT be enlisted into a drained pool");
|
||||
// Assert LIVENESS, not the kill-call COUNT. This assertion used to read
|
||||
// `assert.equal(killed.length, 1)` — and it PASSED while the orphan it is named after was
|
||||
// actually present: _cancelBooting kills BY NAME, and at drain time the tmux session does not
|
||||
// exist yet (bootPane runs on a microtask), so that kill is a NO-OP which still increments the
|
||||
// counter. "kill was called once" and "a live session is orphaned" were both true at the same
|
||||
// time. The only honest question is whether the session is dead.
|
||||
assert.equal(live.size, 0, "it kills itself — no orphan process left behind");
|
||||
});
|
||||
|
||||
test("bootPane failure is counted, never thrown into the request path, and does not wedge refill", async () => {
|
||||
const { pool } = makeFakePool({ size: 1, bootThrows: true });
|
||||
pool.acquire("sonnet");
|
||||
pool.refill();
|
||||
await settle();
|
||||
assert.equal(pool.warm, 0);
|
||||
assert.equal(pool.bootFailures, 1);
|
||||
assert.equal(pool.booting, 0, "the _booting counter is released on failure (else refill wedges forever)");
|
||||
assert.equal(pool.acquire("sonnet"), null, "and the caller just MISSES → cold path");
|
||||
});
|
||||
|
||||
test("drain kills every warm pane and pauses refills; resume restarts them", async () => {
|
||||
const { pool, killed } = makeFakePool({ size: 2 });
|
||||
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||
assert.equal(pool.warm, 2);
|
||||
|
||||
assert.equal(pool.drain(), 2, "drain reports how many it killed");
|
||||
assert.equal(pool.warm, 0);
|
||||
assert.equal(killed.length, 2, "both panes killed — none outlive the drain");
|
||||
|
||||
pool.refill(); await settle();
|
||||
assert.equal(pool.warm, 0, "refill is a NO-OP while drained (paused)");
|
||||
|
||||
pool.resume(); await settle();
|
||||
assert.equal(pool.warm, 2, "resume refills");
|
||||
});
|
||||
|
||||
// ── The crux: pool ↔ reaper coexistence (POOL/REAPER INVARIANT, lib/tui/session.mjs) ──
|
||||
console.log("\nTUI warm pool ↔ session reaper coexistence:");
|
||||
|
||||
test("INVARIANT 1: a LIVE pooled pane is NEVER reaped (it is in the spare set)", () => {
|
||||
const killed = [];
|
||||
const live = "ocp-tui-3456-pdeadbeef";
|
||||
const fakeTmux = (args) => {
|
||||
if (args[0] === "list-sessions") return { status: 0, stdout: `${live}\nocp-tui-3456-aaaa\n` };
|
||||
if (args[0] === "kill-session") { killed.push(args[args.indexOf("-t") + 1]); return { status: 0 }; }
|
||||
return { status: 0, stdout: "" };
|
||||
};
|
||||
const n = reapStaleTuiSessions({ tmux: fakeTmux, port: 3456, spare: new Set([live]) });
|
||||
assert.equal(n, 1, "only the stale turn session was reaped");
|
||||
assert.ok(!killed.includes(live), "the live warm pane must survive the sweep");
|
||||
assert.ok(killed.includes("ocp-tui-3456-aaaa"), "a genuinely stale own session is still reaped");
|
||||
});
|
||||
|
||||
test("INVARIANT 2: an ORPHANED pooled pane (pool-shaped but NOT in the spare set) IS reaped", () => {
|
||||
const killed = [];
|
||||
// ocp-tui-3456-porphan01 LOOKS pooled but the live registry does not claim it — e.g. left
|
||||
// behind by a previous process generation, whose in-memory registry died with it.
|
||||
const fakeTmux = (args) => {
|
||||
if (args[0] === "list-sessions") return { status: 0, stdout: "ocp-tui-3456-porphan01\nocp-tui-3456-plive0001\n" };
|
||||
if (args[0] === "kill-session") { killed.push(args[args.indexOf("-t") + 1]); return { status: 0 }; }
|
||||
return { status: 0, stdout: "" };
|
||||
};
|
||||
const n = reapStaleTuiSessions({ tmux: fakeTmux, port: 3456, spare: new Set(["ocp-tui-3456-plive0001"]) });
|
||||
assert.equal(n, 1);
|
||||
assert.deepEqual(killed, ["ocp-tui-3456-porphan01"], "exemption is by EXACT NAME, never by name shape");
|
||||
});
|
||||
|
||||
test("INVARIANT 2b: with NO spare set (the pre-pool call shape) pool-shaped panes are reaped — fail-safe", () => {
|
||||
const killed = [];
|
||||
const fakeTmux = (args) => {
|
||||
if (args[0] === "list-sessions") return { status: 0, stdout: "ocp-tui-3456-pdeadbeef\n" };
|
||||
if (args[0] === "kill-session") { killed.push(args[args.indexOf("-t") + 1]); return { status: 0 }; }
|
||||
return { status: 0, stdout: "" };
|
||||
};
|
||||
const n = reapStaleTuiSessions({ tmux: fakeTmux, port: 3456 });
|
||||
assert.equal(n, 1, "omitting `spare` reaps MORE, never less — forgetting it can't leak panes");
|
||||
assert.deepEqual(killed, ["ocp-tui-3456-pdeadbeef"]);
|
||||
});
|
||||
|
||||
test("INVARIANT 3: kill-server is SUPPRESSED while a live pooled pane is spared", () => {
|
||||
const calls = [];
|
||||
const live = "ocp-tui-3456-plive0001";
|
||||
const fakeTmux = (args) => {
|
||||
calls.push(args.join(" "));
|
||||
if (args[0] === "list-sessions") return { status: 0, stdout: `${live}\nocp-tui-3456-aaaa\n` };
|
||||
return { status: 0, stdout: "" };
|
||||
};
|
||||
reapStaleTuiSessions({ tmux: fakeTmux, port: 3456, spare: new Set([live]) });
|
||||
assert.ok(!calls.includes("kill-server"), "kill-server would kill the live pane (a child of the tmux server)");
|
||||
});
|
||||
|
||||
test("INVARIANT 3b: after a DRAIN the spare set is empty, so kill-server fires again (zombie reaping preserved)", async () => {
|
||||
// This is the whole reason server.mjs drains BEFORE the periodic sweep: a permanently-full
|
||||
// pool would otherwise permanently suppress the only mechanism that reaps defunct claudes.
|
||||
const { pool } = makeFakePool({ size: 2 });
|
||||
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||
assert.equal(pool.liveNames().size, 2, "pool is full → the sweep would be suppressed");
|
||||
|
||||
pool.drain();
|
||||
assert.equal(pool.liveNames().size, 0, "drain empties the live registry");
|
||||
|
||||
const calls = [];
|
||||
const fakeTmux = (args) => {
|
||||
calls.push(args.join(" "));
|
||||
if (args[0] === "list-sessions") return { status: 0, stdout: "ocp-tui-3456-aaaa\n" };
|
||||
return { status: 0, stdout: "" };
|
||||
};
|
||||
reapStaleTuiSessions({ tmux: fakeTmux, port: 3456, spare: pool.liveNames() });
|
||||
assert.ok(calls.includes("kill-server"), "kill-server fires post-drain — defunct zombies still get reaped");
|
||||
});
|
||||
|
||||
// ── MID-BOOT: the state that hid M1a + M1b ────────────────────────────────────────────
|
||||
// bootTuiPane creates the tmux session SYNCHRONOUSLY, then waits up to POOL_BOOT_MS (20s)
|
||||
// for the input bar. So a pooled session can be LIVE for ~20s before its boot resolves.
|
||||
// Every reaper test above uses a pool that is either full or drained — never mid-boot.
|
||||
// That gap is exactly why both bugs shipped past the first round of tests.
|
||||
|
||||
test("M1a: a reap tick during an IN-FLIGHT BOOT must not orphan-kill the booting pane", async () => {
|
||||
const { pool, live } = makeFakePool({ size: 1, hold: true });
|
||||
pool.acquire("sonnet");
|
||||
pool.refill();
|
||||
await tick(); // boot started; session live; NOT yet warm
|
||||
assert.equal(pool.warm, 0, "not warm yet");
|
||||
assert.equal(pool.booting, 1, "a boot is in flight");
|
||||
assert.equal(live.size, 1, "...and its tmux session ALREADY EXISTS");
|
||||
|
||||
const bootingName = [...live][0];
|
||||
assert.ok(pool.liveNames().has(bootingName),
|
||||
"REGRESSION GUARD: the booting pane MUST be nameable, or the sweep cannot spare it " +
|
||||
"(the pool used to track in-flight boots as a COUNT and this was empty)");
|
||||
});
|
||||
|
||||
test("M1a: the reap tick's drain kills the booting pane, and resume() starts a FRESH boot", async () => {
|
||||
const warns = [];
|
||||
const { pool, live, releaseBoot } = makeFakePool({ size: 1, hold: true });
|
||||
pool._log = (lvl, ev) => { if (lvl === "warn") warns.push(ev); };
|
||||
pool.acquire("sonnet");
|
||||
pool.refill();
|
||||
await tick();
|
||||
const first = [...live][0];
|
||||
|
||||
// The reap tick, as server.mjs runs it: drain -> reap -> resume.
|
||||
const drained = pool.drain();
|
||||
assert.equal(drained, 1, "drain accounts for the booting pane");
|
||||
assert.equal(live.size, 0, "its tmux session is killed — kill-server can now flush zombies");
|
||||
assert.equal(pool.liveNames().size, 0, "nothing left to spare, so kill-server is not suppressed");
|
||||
|
||||
pool.resume();
|
||||
await tick();
|
||||
assert.equal(pool.booting, 1, "resume() started a FRESH boot — the pool is not left empty with nothing scheduled");
|
||||
assert.notEqual([...live][0], first, "and it is a NEW pane, not the killed one");
|
||||
|
||||
// Now let the ORIGINAL (cancelled) boot settle. It rejects with tui_pane_not_ready because
|
||||
// we killed its session — but that is OUR doing, not a fault.
|
||||
releaseBoot();
|
||||
await settle();
|
||||
assert.equal(pool.bootFailures, 0,
|
||||
"a cancelled boot must NOT be counted as a bootFailure — that is the WARN operators alert on");
|
||||
assert.deepEqual(warns, [], "and it must not log tui_pool_boot_failed for a healthy drain");
|
||||
assert.equal(pool.cancelled, 1, "it is counted as a cancellation instead (counted exactly once)");
|
||||
});
|
||||
|
||||
test("M1b: shutdown drain kills the booting pane SYNCHRONOUSLY — no orphaned claude", async () => {
|
||||
const { pool, live } = makeFakePool({ size: 1, hold: true });
|
||||
pool.acquire("sonnet");
|
||||
pool.refill();
|
||||
await tick();
|
||||
assert.equal(live.size, 1, "a live pooled session exists");
|
||||
|
||||
// gracefulShutdown: drain() then process.exit(0) IN THE SAME TICK (TUI panes are tmux
|
||||
// children, not node children, so activeProcesses is empty and the exit is immediate).
|
||||
// Nothing scheduled on the microtask queue can run. So we assert WITHOUT awaiting.
|
||||
pool.drain();
|
||||
assert.equal(live.size, 0,
|
||||
"REGRESSION GUARD: the pane must be dead BEFORE any await. A .then()-based cleanup would " +
|
||||
"never run before process.exit and would orphan a live authenticated `claude`.");
|
||||
});
|
||||
|
||||
// M1b, second costume: drain() in the SAME synchronous block as refill(). The tmux session does
|
||||
// not exist yet at drain time (bootPane runs on a microtask), so _cancelBooting's kill-by-name is
|
||||
// a no-op — and a `.then` that merely `return`s on a stale generation would then let the boot
|
||||
// CREATE the session and walk away from it. Not reachable from any current call site, but ADR 0008
|
||||
// and the reap-tick comment both contemplate a boot-time pre-warm, which is exactly this shape.
|
||||
// NOTE: deliberately NOT `hold: true`. A held boot never settles, so its `.then` never runs and
|
||||
// the guard would vacuously pass — the test must let the boot actually SUCCEED, because the bug is
|
||||
// precisely that a SUCCESSFUL boot on a cancelled generation walks away from its live session.
|
||||
test("M1b': a boot cancelled BEFORE its session existed is still killed when it settles", async () => {
|
||||
const { pool, live } = makeFakePool({ size: 1 });
|
||||
pool.acquire("sonnet"); // miss → learns the model
|
||||
|
||||
pool.refill(); // mints the identity; bootPane is queued on a microtask — no session YET
|
||||
pool.drain(); // SAME sync block: kill-by-name finds nothing to kill (no-op), bumps the gen
|
||||
assert.equal(live.size, 0, "precondition: the session genuinely did not exist at cancel time");
|
||||
|
||||
await tick(); // NOW the boot microtask runs, CREATES the session, and settles on a stale gen
|
||||
await tick();
|
||||
|
||||
assert.equal(live.size, 0,
|
||||
"REGRESSION GUARD: a stale-generation boot must KILL its pane, not assume _cancelBooting " +
|
||||
"already did. _cancelBooting kills BY NAME, and the tmux session does not exist until the " +
|
||||
"boot microtask runs — so a cancellation landing first is a no-op, and a bare `return` here " +
|
||||
"orphans a live authenticated `claude` that nothing owns.");
|
||||
});
|
||||
|
||||
test("a stale boot settling after drain+resume must not clear the NEW boot's slot", async () => {
|
||||
const { pool, releaseBoot } = makeFakePool({ size: 1, hold: true });
|
||||
pool.acquire("sonnet");
|
||||
pool.refill();
|
||||
await tick();
|
||||
pool.drain(); // cancels boot #1 (generation bumped)
|
||||
pool.resume();
|
||||
await tick();
|
||||
assert.equal(pool.booting, 1, "boot #2 owns the slot");
|
||||
releaseBoot(); // boot #1 finally settles (rejects)
|
||||
await settle();
|
||||
assert.equal(pool.booting, 1, "boot #2 STILL owns the slot — a stale settle must not free it");
|
||||
});
|
||||
|
||||
test("a model switch cancels an in-flight boot for the OLD model (kills it, frees the slot)", async () => {
|
||||
const { pool, live, killed } = makeFakePool({ size: 1, hold: true });
|
||||
pool.acquire("sonnet");
|
||||
pool.refill();
|
||||
await tick();
|
||||
const sonnetPane = [...live][0];
|
||||
|
||||
pool.acquire("opus"); // retarget mid-boot
|
||||
assert.ok(killed.includes(sonnetPane), "the old model's booting pane is killed, not left to linger");
|
||||
assert.equal(pool.booting, 0, "and its slot is freed immediately, so the new model can boot now");
|
||||
assert.equal(pool.warmModel, "opus");
|
||||
|
||||
pool.refill();
|
||||
await tick();
|
||||
assert.equal(pool.booting, 1, "a boot for the NEW model starts without waiting out the old one");
|
||||
});
|
||||
|
||||
test("a pane handed out for a turn leaves the spare set immediately (so its teardown is authoritative)", async () => {
|
||||
const { pool } = makeFakePool({ size: 2 });
|
||||
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||
const taken = pool.acquire("sonnet");
|
||||
assert.ok(!pool.liveNames().has(taken.name),
|
||||
"an acquired pane is the CALLER's — the pool must not also claim it live, or a crashed turn's pane would be spared forever");
|
||||
});
|
||||
|
||||
test("N1: the pool mints ONE identity — the tmux name's hex is the transcript session-id's hex", async () => {
|
||||
// Without this, `tmux ls` shows a pane whose name has no relation to any transcript file,
|
||||
// so a live pane cannot be correlated to <HOME>/.claude/projects/*/<sessionId>.jsonl.
|
||||
const seen = [];
|
||||
const pool = new TuiPanePool({
|
||||
size: 1,
|
||||
mintPane: () => {
|
||||
const sessionId = "deadbeef-1111-2222-3333-444444444444";
|
||||
return { sessionId, name: poolName(3456, sessionId) };
|
||||
},
|
||||
bootPane: async (model, ident) => {
|
||||
seen.push(ident);
|
||||
return { ...ident, model, bootedAt: Date.now() };
|
||||
},
|
||||
killPane: () => {},
|
||||
paneHealthy: () => true,
|
||||
});
|
||||
pool.acquire("sonnet");
|
||||
pool.refill();
|
||||
await settle();
|
||||
assert.equal(seen.length, 1, "bootPane received the pool-minted identity");
|
||||
assert.equal(seen[0].name, "ocp-tui-3456-pdeadbeef");
|
||||
assert.ok(seen[0].name.endsWith(seen[0].sessionId.slice(0, 8)),
|
||||
"the tmux session name carries the session-id's own hex — `tmux ls` correlates to the transcript");
|
||||
const pane = pool.acquire("sonnet");
|
||||
assert.equal(pane.sessionId, seen[0].sessionId,
|
||||
"and the turn reads the transcript under THAT session-id — one identity end to end");
|
||||
});
|
||||
|
||||
test("pool pane names are port-scoped (reapable as ours) and never match the legacy shape", () => {
|
||||
const name = poolName(3456, "deadbeef-1111-2222-3333-444444444444");
|
||||
assert.ok(name.startsWith(sessionPrefixForPort(3456)), "pool panes are OURS → reapable when orphaned");
|
||||
assert.equal(name, "ocp-tui-3456-pdeadbeef");
|
||||
assert.ok(!LEGACY_SESSION_NAME_RE.test(name), "must never be mistaken for a legacy bare-prefix session");
|
||||
assert.ok(!poolName(9999, "aaaaaaaa-0000-0000-0000-000000000000").startsWith(sessionPrefixForPort(3456)),
|
||||
"a sibling instance's pool pane is foreign to us");
|
||||
});
|
||||
|
||||
test("buildTuiHealthBlock reports pool:null when off, and the pool's stats when on", async () => {
|
||||
const st = { lastEntrypoint: "cli", entrypointMismatches: 0 };
|
||||
const sem = { inflight: 0, queued: 0 };
|
||||
const off = buildTuiHealthBlock({ enabled: true, entrypointMode: "cli", maxConcurrent: 2 }, st, sem, null);
|
||||
assert.equal(off.pool, null, "pool disabled → explicit null (stable /health shape)");
|
||||
|
||||
const { pool } = makeFakePool({ size: 2 });
|
||||
pool.acquire("sonnet"); pool.refill(); await settle();
|
||||
const on = buildTuiHealthBlock({ enabled: true, entrypointMode: "cli", maxConcurrent: 2 }, st, sem, pool);
|
||||
assert.equal(on.pool.size, 2);
|
||||
assert.equal(on.pool.warm, 2);
|
||||
assert.equal(on.pool.misses, 1);
|
||||
assert.equal(on.pool.model, "sonnet");
|
||||
});
|
||||
|
||||
// ── TUI home preparation (scratch vs real) ───────────────────────────────
|
||||
import { prepareTuiHome, ensureTuiCwdTrusted } from "./lib/tui/session.mjs";
|
||||
import { mkdtempSync as hMkdtemp, mkdirSync as hMkdir, writeFileSync as hWrite, readFileSync as hRead, existsSync as hExists, readlinkSync as hReadlink } from "node:fs";
|
||||
@@ -2474,8 +2978,13 @@ test("buildTuiHealthBlock: shape + live counters (the additive /health tui block
|
||||
const ts = { lastEntrypoint: "cli", entrypointMismatches: 3 };
|
||||
const block = buildTuiHealthBlock(
|
||||
{ enabled: true, entrypointMode: "cli", maxConcurrent: 2 }, ts, sem);
|
||||
// `pool` joined this key set with the warm pane pool. The tui block is ADR-0007-owned (it
|
||||
// did not exist at v3.16.4, so it is outside ADR 0006's grandfather freeze), and the
|
||||
// addition is purely additive: every pre-existing key below still carries a byte-identical
|
||||
// value, and `pool` is null unless the operator opts in via OCP_TUI_POOL_SIZE.
|
||||
assert.deepEqual(Object.keys(block).sort(),
|
||||
["enabled", "entrypointMismatches", "entrypointMode", "inflight", "lastEntrypoint", "maxConcurrent", "queued"]);
|
||||
["enabled", "entrypointMismatches", "entrypointMode", "inflight", "lastEntrypoint", "maxConcurrent", "pool", "queued"]);
|
||||
assert.equal(block.pool, null, "no pool passed → null (the default, pool disabled)");
|
||||
assert.equal(block.enabled, true);
|
||||
assert.equal(block.entrypointMode, "cli");
|
||||
assert.equal(block.lastEntrypoint, "cli");
|
||||
@@ -2934,7 +3443,9 @@ async function runAsyncTests() {
|
||||
}
|
||||
|
||||
// ── Cleanup ──
|
||||
runAsyncTests().then(() => {
|
||||
// Settle the async-bodied tests registered through the sync `test()` helper BEFORE summarizing —
|
||||
// otherwise their pass/fail is not reflected in the counts (see the `pendingAsync` comment above).
|
||||
runAsyncTests().then(() => Promise.all(pendingAsync)).then(() => {
|
||||
closeDb();
|
||||
console.log(`\n=== Results: ${passed} passed, ${failed} failed ===\n`);
|
||||
process.exit(failed > 0 ? 1 : 0);
|
||||
|
||||
Reference in New Issue
Block a user