docs(plans): TUI streaming is not achievable — prereq spike result + honest README constraints

Backlog #2 of docs/plans/2026-07-13-tui-latency demanded a prereq spike before any streaming design: does the transcript JSONL grow during a turn, or only at the end? The spike was run. All three candidate sources are dead: (a) transcript JSONL — grows at EVENT granularity; the assistant's text event is written as ONE complete line, ~0.3s before the terminal turn_duration event (observed: turn_duration 7319ms; text event at t+7.0s, terminal at t+7.3s). (b) tmux capture-pane — the pane is a RENDERED view, not the text. Same turn, transcript T = '## Semaphore\n\nA **semaphore** is a synchronization…' pane = '⏺ Semaphore' / ' A semaphore is a synchronization…' '## ', '**' and ```-fences are absent from the pane entirely (rendered to ANSI, then stripped by capture-pane -p). T.startsWith(paneText) is FALSE both raw and indent-stripped — not on redraw, but on essentially every markdown answer. capture-pane -e recovers styling, never source spelling: no unique inverse. (c) --debug-file — byte-exact ('last_assistant_message':'## Title\n\n**alpha…'), but only inside end-of-turn Stop-hook payloads; zero content_block_delta / text_delta events; ~2.7MB per turn. --output-format stream-json, the only interface emitting token deltas, requires -p — the metered-billing path TUI mode exists to avoid (cc_entrypoint=sdk-cli). The constraint is structural. OCP's TUI SSE is, and remains, replay-only. Also corrects this plan's own "~20s waiting for the whole turn" decomposition, which was inferred from an external 30-32s report and never measured through OCP. Measured through a real OCP instance (TUI, claude-sonnet-4-6, n=5): median 11.30s before #156, 9.55s after, vs a native turn_duration of ~7.3s → OCP's own overhead is ~2-4s, not ~20s. The remainder is generation time, which streaming would not shorten (it moves the first byte, not the last) — so a consumer needing the COMPLETE answer, which is the JSON-card case that motivated this work, would have gained nothing from streaming. Backlog #4 measured while here: --exclude-dynamic-system-prompt-sections gives ZERO marginal benefit (TTFT median 6.39s vs 6.17s for --effort low alone, n=5, one worse outlier). Do not adopt. Banner stayed on Claude Max. README: documents the ~6s TTFT floor plainly (TUI mode cannot serve interactive-latency consumers) and states that no-token-streaming is structural rather than a missing feature. No code change. No version bump (docs-only). Not endpoint-touching: no server.mjs diff, so no cli.js citation applies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VqgWJcjxrjjL9L9SkpZyXR
2026-07-21 21:15:09 +00:00 · 2026-07-13 15:32:41 +10:00
parent 5258d5d395
commit 6848f9751c
4 changed files with 254 additions and 4 deletions
@@ -1033,7 +1033,7 @@ Then restart OCP. At boot you will see (with the env token set, isolated home au
 ### What changes / what doesn't
 - **Callers see no API change.** The response is a normal OpenAI completion object or chunked SSE — identical wire format.
- **No real token streaming.** TUI-mode buffers the full response then replays it as chunked SSE. You will see a delay then the complete response rather than real-time tokens.
+- **No real token streaming — and it cannot be added.** TUI-mode buffers the full response then replays it as chunked SSE. You will see a delay then the complete response rather than real-time tokens. This is **structural, not a missing feature**: interactive `claude` exposes no byte-faithful incremental output. Its three observables are a *rendered* terminal (the pane's text has markdown source markers — `## `, `**`, code fences — stripped by rendering, so it is not the model's text), a transcript written at *event* granularity (the whole answer lands in one line ~0.3 s before the turn ends), and a debug log also written at end-of-turn. The one interface that emits token deltas, `--output-format stream-json`, requires `-p` — the metered-billing path TUI mode exists to avoid. Evidence: [`docs/plans/2026-07-13-tui-latency/streaming-spike.md`](docs/plans/2026-07-13-tui-latency/streaming-spike.md).
 - **Cache and singleflight work normally.** TUI-mode writes the buffered response to the cache on success; cache-hits skip the interactive turn entirely.
 - **The host's `CLAUDE.md` / auto-memory is never injected.** OCP is a proxy — the proxied client (OpenClaw / your IDE) owns its own context and memory. TUI-mode always runs `claude` with `CLAUDE_CODE_DISABLE_CLAUDE_MDS` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY`, so a `CLAUDE.md` on the OCP host can never leak into proxied turns (verified live; see #4). Built-in tool schemas + the interactive system prompt remain (the inherent ~20–35K context floor of interactive mode); MCP is hard-disabled.
 - **Authenticate via `CLAUDE_CODE_OAUTH_TOKEN` in a credential-isolated home (recommended).** tmux does not forward the parent process's env to the pane, so OCP sets the token explicitly on the spawned `claude` when `CLAUDE_CODE_OAUTH_TOKEN` is present. But passing the token is **not enough on its own**: interactive `claude` *prefers* `~/.claude/.credentials.json` over the env var (unlike the `-p` path), so a stale `credentials.json` would shadow the token. With the env token set and `OCP_TUI_HOME` unset, OCP therefore runs claude in a **credential-isolated home** (`$HOME/.ocp-tui/home`) that has **no `credentials.json`** — so the env token is the only credential and is authoritative, and claude never runs the token-refresh path (so the single-use refresh token can't be corrupted by the spawn/teardown cycle). On a long-running host the credentials.json path produced a permanent `Please run /login · API Error: 401` that re-login could not fix (the next spawn re-corrupted it); the isolated home ends that at the root. Transcripts land under the same isolated home, so the answer-reader is unaffected. Without the env token, claude falls back to the real home's `credentials.json` (byte-for-byte the previous behaviour). (The token is visible in `ps` on the pane command — acceptable for the single-user A-path; the multi-user B-path is refused at boot.) See ADR 0007 PR-C / PR-D amendments.
@@ -1041,6 +1041,33 @@ Then restart OCP. At boot you will see (with the env token set, isolated home au
 - **Default path unchanged.** Unset `CLAUDE_TUI_MODE` and restart → `callClaude` / `callClaudeStreaming` are used again, byte-for-byte identical to today.
 - **Concurrency is bounded separately.** TUI turns are heavy (per-request cold-boot + long wallclock), so the TUI path has its own limiter — `OCP_TUI_MAX_CONCURRENT` (default `2`), independent of `CLAUDE_MAX_CONCURRENT`. Excess turns queue; a full queue returns a 503. Tune it up only on a host that can run more interactive `claude` sessions at once.
 ### ⚠️ Latency: TUI mode has a ~6-second floor, and it is immovable
 **TUI mode cannot serve real-time or interactive-latency consumers.** This is a hard property of the
 path, stated plainly so you can rule it out before building on it:
 | | measured |
 |---|---|
 | **TTFT floor (first token)** | **≈ 6 s** — immovable |
 | cold boot → input bar ready | ~1 s (per request; not the bottleneck) |
 | OCP's own overhead above the CLI | ~2–4 s |
 | direct Anthropic API, same prompt (for scale) | 0.84–1.64 s |
 The ~6 s floor is the `claude` CLI itself: it always injects the full Claude Code system prompt plus
 its tool definitions before your prompt, on every turn, no matter what you ask. No flag removes it
 (`--exclude-dynamic-system-prompt-sections` was measured: **no effect** on the floor). Extended
 thinking is *not* the cause — `OCP_TUI_EFFORT` already defaults to `low`, which is what cuts a
 formerly-inherited `xhigh` down to this floor and collapses its variance.
 On top of the floor you pay the model's generation time (a function of output length), and there is
 **no progressive output** — see "No real token streaming" above; it is structural and cannot be
 fixed. So a turn returns as one blob after the full generation completes.
 **Use TUI mode for**: batch, background, and latency-insensitive work where the subscription pool is
 the point. **Do not use it for**: anything a person is waiting on interactively, or any consumer with
 a sub-5-second budget. Full measurements and methodology:
 [`docs/plans/2026-07-13-tui-latency/`](docs/plans/2026-07-13-tui-latency/).
 ### Monitoring drift via `/health`
 `GET /health` includes a `tui` block so you can poll for a silent billing-pool drift (the top risk after the 6/15 flip — a lost TTY flipping `cc_entrypoint` from `cli` to the metered `sdk-cli` pool would still return answers but burn metered credits). The block is **always present** (with `enabled:false` when TUI-mode is off):
@@ -1,7 +1,9 @@
 # TUI-mode latency: measured floor, and the four things worth fixing
 **Date**: 2026-07-13
-**Status**: findings + backlog (no code changed yet)
+**Status**: findings + backlog. **Superseded in part** — see the dated update boxes below.
 Item #1 shipped ([#156](https://github.com/dtzp555-max/ocp/pull/156)); item #2 is **dead**
 ([`streaming-spike.md`](streaming-spike.md)); item #4 measured, **no effect**; item #3 stands.
 **Measured on**: Mac mini / macOS 26.5.2 / Claude Code **v2.1.207** / Sonnet 5 / Claude Max subscription / **real-home mode** (no `CLAUDE_CODE_OAUTH_TOKEN`, no `OCP_TUI_HOME` in the service env)
 **Evidence**: [`measurements.jsonl`](measurements.jsonl) — **n=15** (3 configs × 5) · banner captures [`billing-banner.txt`](billing-banner.txt) · harness [`floor.sh`](floor.sh)
@@ -47,6 +49,17 @@ All rows in [`measurements.jsonl`](measurements.jsonl); every number below is re
 before returning anything. There is no streaming path. The ~20 s delta between this harness's
 real TTFT and OCP's reported 30–32 s is exactly that.
 > **⚠️ 2026-07-13 correction — this decomposition attributes the ~20 s to the wrong thing.** It was
 > inferred from the external 30–32 s report, never measured *through* OCP. It has since been measured
 > through a real OCP instance (TUI mode, `claude-sonnet-4-6`, the same ~1850-token prompt, n=5):
 > **median 11.30 s** before [#156](https://github.com/dtzp555-max/ocp/pull/156), **9.55 s** after —
 > against a native `turn_duration` of ~7.3 s for a comparable turn. So **OCP's own overhead above the
 > CLI is ~2–4 s, not ~20 s.** The rest of any larger number is the model *generating a long answer*,
 > which the blocking wait does not cause and streaming would not shorten — it would only move the
 > first byte earlier. The 30–32 s figure therefore reflects a much longer output (and/or the
 > then-inherited `xhigh` effort), not 20 s of OCP dead time. See
 > [`streaming-spike.md`](streaming-spike.md) § "What streaming would have bought".
 ---
 ## ⚠️ Blocking constraint: `--bare` silently drops you off the subscription pool
@@ -105,7 +118,21 @@ the effort level silently changes if the operator ever switches to env-token mod
 - **Risk**: none — banner confirms it stays on `Claude Max` (see `billing-banner.txt`).
 - ⚠️ Do **not** reach for `--bare` to shave boot: see above.
-### 2. Real streaming instead of blocking on turn-terminal — **the big one (~20 s)**
+### 2. Real streaming instead of blocking on turn-terminal — ~~the big one (~20 s)~~ → **DEAD**
 > **2026-07-13 update — the prereq spike below was run, and it kills this item.** The transcript
 > grows at *event* granularity (the whole answer lands in one line, ~0.3 s before terminal), and the
 > pane is a **rendered** view whose `capture-pane` text no longer contains the answer's source bytes
 > (`## `, `**`, code fences are gone) — so no pane-derived stream can be reconciled with the
 > authoritative text. A third source (`--debug-file`) is byte-exact but also end-of-turn.
 > `--output-format stream-json`, the only interface that emits token deltas, requires `-p` — the
 > metered-billing path that TUI mode exists to avoid. **True token streaming is not achievable on the
 > TUI path**; OCP's TUI SSE is replay-only. Note also that streaming would never have shortened a
 > turn — only its first byte — so a consumer that needs the *complete* answer (the JSON-card case
 > that motivated this investigation) gains nothing from it.
 >
 > Full evidence, the value re-assessment, and the maintainer's options: **[`streaming-spike.md`](streaming-spike.md)**.
 > The original framing is preserved below for the record.
 Today `runTuiTurn` blocks on the transcript until the turn is *finished*. The pane is already
 rendering tokens incrementally the whole time — this harness proves you can observe first token
@@ -132,7 +159,14 @@ the background) amortizes it to zero for any workload below the pool refill rate
  #148 — pooled panes must not look like zombies to the sweep.
 - Lower priority than #1 and #2: it is the smallest slice.
-### 4. Trim the prefill — **probably not worth it; know the floor**
+### 4. Trim the prefill — ~~probably not worth it~~ **MEASURED: no effect. Do not adopt.**
 > **2026-07-13 update.** `--exclude-dynamic-system-prompt-sections` was measured with the same
 > harness (`floor.sh`, n=5, Sonnet 5, on top of `--effort low`): **TTFT median 6.39 s**
 > (5.87–10.54 s) vs **6.17 s** (5.87–6.44 s) for `--effort low` alone — i.e. **zero marginal
 > benefit**, within noise and with one worse outlier. The banner stayed on `· Claude Max`
 > (no billing-pool drop), but there is no win to bank. The ~6 s floor stands as stated below.
 After #1–#3, the floor is **~6 s**, and it does not go lower. `claude` always injects the full
 Claude Code system prompt + tool definitions (thousands to tens of thousands of prefill tokens)
@@ -0,0 +1,5 @@
 {"i":1,"tag":"effort-low-exclude-dynamic","model":"claude-sonnet-5","extra_args":"--effort low --exclude-dynamic-system-prompt-sections","prompt_chars":7451,"boot_ms":934,"ttft_ms":5867,"complete_ms":9953}
 {"i":2,"tag":"effort-low-exclude-dynamic","model":"claude-sonnet-5","extra_args":"--effort low --exclude-dynamic-system-prompt-sections","prompt_chars":7451,"boot_ms":1275,"ttft_ms":6388,"complete_ms":9874}
 {"i":3,"tag":"effort-low-exclude-dynamic","model":"claude-sonnet-5","extra_args":"--effort low --exclude-dynamic-system-prompt-sections","prompt_chars":7451,"boot_ms":874,"ttft_ms":10537,"complete_ms":11782}
 {"i":4,"tag":"effort-low-exclude-dynamic","model":"claude-sonnet-5","extra_args":"--effort low --exclude-dynamic-system-prompt-sections","prompt_chars":7451,"boot_ms":1170,"ttft_ms":6379,"complete_ms":9947}
 {"i":5,"tag":"effort-low-exclude-dynamic","model":"claude-sonnet-5","extra_args":"--effort low --exclude-dynamic-system-prompt-sections","prompt_chars":7451,"boot_ms":1329,"ttft_ms":6443,"complete_ms":9884}
@@ -0,0 +1,184 @@
 # Backlog #2 (real streaming): the prereq spike says **no**
 **Date**: 2026-07-13
 **Status**: negative result — evidence, not a decision. The maintainer chooses what to do with it.
 **Scope**: this answers the prereq spike that [`README.md`](README.md) § "Backlog #2" demanded *before* any streaming design:
 > **Prereq spike (do this before designing anything)**: does the transcript JSONL grow *during* a
 > turn, or only at the end? If only at the end, (a) is dead and you are stuck with (b).
 The spike was run. **Both (a) and (b) are dead**, and so is a third candidate the original backlog
 did not consider. The conclusion is stronger than "streaming is hard": **the interactive `claude`
 CLI does not expose its token stream to any observer.** It renders tokens to a terminal and records
 the *finished* text. There is no byte-faithful incremental source for a proxy to forward.
 **Measured on**: Mac mini / Claude Code **v2.1.207** / Sonnet 5 + Sonnet 4.6 / Claude Max /
 real-home mode. Every claim below is reproducible from the commands given.
 ---
 ## The three candidate sources, and how each one dies
 ### (a) Incremental transcript reads — **dead: event granularity, not token granularity**
 The transcript JSONL *does* grow during a turn, but it grows one **whole event at a time**, and the
 assistant's text event is written as **one complete line** — it appears only ~0.3 s before the
 terminal `turn_duration` event.
 Observed (session `efd5b161`, `turn_duration: 7319 ms`, polling the file every 0.5 s):
 ```
 bytes=64347 lines=13  →  64479 lines=14  →  92192 lines=21     (during the turn)
 #6  t+0.0s   type=user       (the prompt)
 #15 t+4.7s   type=assistant  blocks=thinking
 #16 t+7.0s   type=assistant  blocks=text      ← the ENTIRE answer, in one line
 #21 t+7.3s   type=system     subtype=turn_duration   ← terminal
 ```
 Reading the transcript incrementally therefore buys ~0.3 s, not a token stream. The "clean option"
 the backlog hoped for does not exist.
 ### (b) `tmux capture-pane` diffing — **dead: the pane is a RENDERED view, not the text**
 This is the one the backlog expected to fall back to, noting it is "lossy for exact text (wrapping,
 scrollback, spinner lines)". The loss is **worse than formatting noise: the pane does not contain
 the answer's source bytes at all.** claude's TUI *renders* markdown, and `capture-pane -p` strips
 the ANSI styling that rendering produced — so the source markers are gone, irrecoverably.
 Same turn, same lines — authoritative transcript text vs. what the pane holds
 (`capture-pane -p -J -S -500`, pane width 220):
 ```
 TRANSCRIPT (authoritative T, via extractLatestAssistantText):
  '## Semaphore'
  ''
  'A **semaphore** is a synchronization primitive that controls access to a shared resource by …'
  ''
  '- Tracks available "permits" and decrements the count when a thread acquires access, …'
 PANE (the same turn):
  ''
  '⏺ Semaphore'                                   ← '## ' heading became a bullet glyph
  ''
  '  A semaphore is a synchronization primitive that controls access to a shared resource by …'
  ''                                             ← '**' bold markers GONE; every line indented 2 sp
  '  - Tracks available "permits" and decrements the count when a thread acquires access, …'
 ```
 Token-presence check over the whole final frame:
 | token in the answer | in transcript `T` | in pane |
 |---|---|---|
 | `## ` (ATX heading) | yes | **no** — rendered as `⏺` |
 | `**` (bold markers) | yes | **no** — rendered as ANSI bold, then stripped by `-p` |
 | ` ```javascript ` (fence + language) | yes | **no** — fence gone, language tag gone |
 | `- ` (list item) | yes | yes |
 Consequence for any design built on pane diffing: with `S` = text streamed from the pane and `T` =
 the transcript's authoritative text, the prefix invariant **`T.startsWith(S)` is false** — both raw
 and with the 2-space indent stripped. Not "sometimes, on redraw" — **on essentially every answer
 containing markdown**, which is most answers.
 And it cannot be repaired:
 - `capture-pane -e` (keep ANSI) recovers *styling*, never the *source spelling*: you cannot tell
  `**bold**` from `__bold__`, recover a fence's language tag, or distinguish an H2 from bold text.
 - Reconstructing markdown from rendered output is a lossy inverse with no unique solution, and it
  would be pinned to one claude TUI version's theme.
 A proxy that streams pane text is streaming **something the model did not say**. For OCP that is not
 a quality trade-off, it is a correctness violation (`ALIGNMENT.md` Core Principle: forward what
 `cli.js` emits; do not invent).
 ### (c) `--debug-file` — **dead: byte-exact, but only at end-of-turn**
 Not considered in the original backlog; checked for completeness because it is the only other
 observable claude writes.
 `claude --debug-file <path>` does contain the **byte-exact raw markdown** — but only inside
 **end-of-turn hook payloads**:
 ```
 [DEBUG] Hooks: Parsed initial response: {… "hook_event_name":"Stop", …
        "last_assistant_message":"## Title\n\n**alpha bravo charlie**" …}
 ```
 Note `## ` and `**` survive here — this is the real text. But it arrives with the `Stop` hook, i.e.
 the same end-of-turn granularity as the transcript. Grepping the log for streaming events
 (`content_block_delta` / `text_delta`) finds **zero**. The log is also ~2.7 MB per turn, which rules
 it out as a production mechanism regardless.
 ---
 ## What this means
 **True token streaming is not achievable on the TUI path**, at any effort, with any flag combination
 available in claude 2.1.207. The three observables are: a rendered terminal (lossy beyond repair), a
 transcript written at event granularity (end-of-turn), and a debug log written at end-of-turn.
 `--output-format stream-json` — the one interface that *does* emit `text_delta` events — works
 **only with `--print`/`-p`**, and the `-p` path is precisely what TUI mode exists to avoid (it is
 classified `cc_entrypoint=sdk-cli` → the metered credit pool, not the subscription pool). The
 constraint is structural, not a missing feature.
 **Therefore OCP's SSE on the TUI path is, and remains, replay-only**: the answer is buffered until
 the turn is terminal and then replayed as SSE chunks (`server.mjs`, the `streamStringAsSSE` branch).
 It is wire-valid OpenAI SSE; it is not progressive.
 ### What streaming would have bought — smaller than the backlog implied
 Worth stating plainly, because it changes the value calculus:
 - **Streaming never makes the answer arrive sooner.** It makes the *first byte* arrive sooner. A
  consumer that needs the *complete* answer before it can act — e.g. a JSON-card consumer parsing a
  structured reply, which is exactly the 知音 AI use case that motivated this whole investigation —
  gains **nothing at all** from streaming. Only a progressively-rendering consumer (a chat UI) gains.
 - The backlog's "~20 s" for item #2 was inferred from an external report of 30–32 s, not measured
  through OCP. Measured through a real OCP instance (TUI mode, `claude-sonnet-4-6`, ~1850-token
  prompt, n=5): **median 11.30 s** before PR #156, **9.55 s** after — against a native
  `turn_duration` of ~7.3 s for a comparable turn. So OCP's own overhead over the CLI is roughly
  **2–4 s**, not 20 s; the rest of any large number is the model generating a long answer, which
  streaming hides but does not shorten. (A 30 s turn is a 30 s turn either way — the last token
  lands at the same wall-clock moment.)
 The honest remaining levers on the TUI path are therefore: the **~6 s TTFT floor** (immovable — see
 [`README.md`](README.md)), the **generation time** (a function of output length — not OCP's to
 optimize), and OCP's **own 2–4 s of overhead** (per-request cold boot ≈1 s → backlog #3 warm pool;
 plus readiness/paste/transcript poll granularity).
 ---
 ## Options for the maintainer (this document does not choose)
 1. **Accept and document (recommended).** TUI mode cannot stream; its SSE is replay-only. Record the
   constraint next to the ~6 s TTFT floor already in `README.md`, and close backlog #2 as *not
   achievable honestly*. Redirect the effort to the overhead that *is* recoverable (#3).
 2. **Ship lossy "rendered-text" streaming behind an explicit opt-in.** Technically possible; the
   client would receive the TUI's rendered view — headings as `⏺`, bold markers stripped, code
   fences gone, every line indented. **Recommended against**: it delivers bytes the model did not
   produce, silently corrupts structured replies (the JSON/marker protocols proxy consumers actually
   use), and pins OCP's output to a claude TUI theme. It also cannot be reconciled with the
   transcript afterwards — OpenAI SSE deltas cannot be un-sent.
 3. **Reconstruct markdown from `capture-pane -e` ANSI styling.** High effort, brittle across claude
   versions, and *still* not byte-exact (no unique inverse). Recommended against.
 ## Reproduction
 ```bash
 # (a) transcript growth granularity — poll a live turn's JSONL
 #     watch bytes/lines while a TUI turn runs; dump event types + timestamps afterwards
 # (b) pane vs transcript — the decisive one
 #     spawn claude in tmux (prefix NOT ocp-tui-*, never kill-server), ask for markdown,
 #     capture `tmux capture-pane -p -J -S -500` frames during the turn, then diff the pane's
 #     answer region against extractLatestAssistantText() over the session's transcript JSONL.
 #     Expect: T.startsWith(paneText) === false, '## ' and '**' and '```' absent from the pane.
 # (c) debug log
 claude --model claude-sonnet-5 --session-id "$(uuidgen)" --effort low --debug-file /tmp/d.log
 grep -cE 'content_block_delta|text_delta' /tmp/d.log     # → 0
 grep -o 'last_assistant_message":"[^"]*' /tmp/d.log      # → byte-exact text, at Stop-hook time
 ```
 Banner check (mandatory for every spawn-flag change, per [`billing-banner.txt`](billing-banner.txt)):
 every configuration used in this spike was verified to stay on `· Claude Max`.