docs(plans): TUI streaming is not achievable — prereq spike result + honest README constraints

Backlog #2 of docs/plans/2026-07-13-tui-latency demanded a prereq spike before any
streaming design: does the transcript JSONL grow during a turn, or only at the end?
The spike was run. All three candidate sources are dead:

  (a) transcript JSONL — grows at EVENT granularity; the assistant's text event is
      written as ONE complete line, ~0.3s before the terminal turn_duration event
      (observed: turn_duration 7319ms; text event at t+7.0s, terminal at t+7.3s).
  (b) tmux capture-pane — the pane is a RENDERED view, not the text. Same turn,
      transcript T = '## Semaphore\n\nA **semaphore** is a synchronization…'
      pane        = '⏺ Semaphore' / '  A semaphore is a synchronization…'
      '## ', '**' and ```-fences are absent from the pane entirely (rendered to ANSI,
      then stripped by capture-pane -p). T.startsWith(paneText) is FALSE both raw and
      indent-stripped — not on redraw, but on essentially every markdown answer.
      capture-pane -e recovers styling, never source spelling: no unique inverse.
  (c) --debug-file — byte-exact ('last_assistant_message':'## Title\n\n**alpha…'),
      but only inside end-of-turn Stop-hook payloads; zero content_block_delta /
      text_delta events; ~2.7MB per turn.

--output-format stream-json, the only interface emitting token deltas, requires -p —
the metered-billing path TUI mode exists to avoid (cc_entrypoint=sdk-cli). The
constraint is structural. OCP's TUI SSE is, and remains, replay-only.

Also corrects this plan's own "~20s waiting for the whole turn" decomposition, which
was inferred from an external 30-32s report and never measured through OCP. Measured
through a real OCP instance (TUI, claude-sonnet-4-6, n=5): median 11.30s before #156,
9.55s after, vs a native turn_duration of ~7.3s → OCP's own overhead is ~2-4s, not
~20s. The remainder is generation time, which streaming would not shorten (it moves
the first byte, not the last) — so a consumer needing the COMPLETE answer, which is
the JSON-card case that motivated this work, would have gained nothing from streaming.

Backlog #4 measured while here: --exclude-dynamic-system-prompt-sections gives ZERO
marginal benefit (TTFT median 6.39s vs 6.17s for --effort low alone, n=5, one worse
outlier). Do not adopt. Banner stayed on Claude Max.

README: documents the ~6s TTFT floor plainly (TUI mode cannot serve interactive-latency
consumers) and states that no-token-streaming is structural rather than a missing feature.

No code change. No version bump (docs-only). Not endpoint-touching: no server.mjs diff,
so no cli.js citation applies.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VqgWJcjxrjjL9L9SkpZyXR
This commit is contained in:
2026-07-13 15:32:41 +10:00
co-authored by Claude Fable 5
parent 5258d5d395
commit 6848f9751c
4 changed files with 254 additions and 4 deletions
+28 -1
View File
@@ -1033,7 +1033,7 @@ Then restart OCP. At boot you will see (with the env token set, isolated home au
### What changes / what doesn't
- **Callers see no API change.** The response is a normal OpenAI completion object or chunked SSE — identical wire format.
- **No real token streaming.** TUI-mode buffers the full response then replays it as chunked SSE. You will see a delay then the complete response rather than real-time tokens.
- **No real token streaming — and it cannot be added.** TUI-mode buffers the full response then replays it as chunked SSE. You will see a delay then the complete response rather than real-time tokens. This is **structural, not a missing feature**: interactive `claude` exposes no byte-faithful incremental output. Its three observables are a *rendered* terminal (the pane's text has markdown source markers — `## `, `**`, code fences — stripped by rendering, so it is not the model's text), a transcript written at *event* granularity (the whole answer lands in one line ~0.3 s before the turn ends), and a debug log also written at end-of-turn. The one interface that emits token deltas, `--output-format stream-json`, requires `-p` — the metered-billing path TUI mode exists to avoid. Evidence: [`docs/plans/2026-07-13-tui-latency/streaming-spike.md`](docs/plans/2026-07-13-tui-latency/streaming-spike.md).
- **Cache and singleflight work normally.** TUI-mode writes the buffered response to the cache on success; cache-hits skip the interactive turn entirely.
- **The host's `CLAUDE.md` / auto-memory is never injected.** OCP is a proxy — the proxied client (OpenClaw / your IDE) owns its own context and memory. TUI-mode always runs `claude` with `CLAUDE_CODE_DISABLE_CLAUDE_MDS` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY`, so a `CLAUDE.md` on the OCP host can never leak into proxied turns (verified live; see #4). Built-in tool schemas + the interactive system prompt remain (the inherent ~2035K context floor of interactive mode); MCP is hard-disabled.
- **Authenticate via `CLAUDE_CODE_OAUTH_TOKEN` in a credential-isolated home (recommended).** tmux does not forward the parent process's env to the pane, so OCP sets the token explicitly on the spawned `claude` when `CLAUDE_CODE_OAUTH_TOKEN` is present. But passing the token is **not enough on its own**: interactive `claude` *prefers* `~/.claude/.credentials.json` over the env var (unlike the `-p` path), so a stale `credentials.json` would shadow the token. With the env token set and `OCP_TUI_HOME` unset, OCP therefore runs claude in a **credential-isolated home** (`$HOME/.ocp-tui/home`) that has **no `credentials.json`** — so the env token is the only credential and is authoritative, and claude never runs the token-refresh path (so the single-use refresh token can't be corrupted by the spawn/teardown cycle). On a long-running host the credentials.json path produced a permanent `Please run /login · API Error: 401` that re-login could not fix (the next spawn re-corrupted it); the isolated home ends that at the root. Transcripts land under the same isolated home, so the answer-reader is unaffected. Without the env token, claude falls back to the real home's `credentials.json` (byte-for-byte the previous behaviour). (The token is visible in `ps` on the pane command — acceptable for the single-user A-path; the multi-user B-path is refused at boot.) See ADR 0007 PR-C / PR-D amendments.
@@ -1041,6 +1041,33 @@ Then restart OCP. At boot you will see (with the env token set, isolated home au
- **Default path unchanged.** Unset `CLAUDE_TUI_MODE` and restart → `callClaude` / `callClaudeStreaming` are used again, byte-for-byte identical to today.
- **Concurrency is bounded separately.** TUI turns are heavy (per-request cold-boot + long wallclock), so the TUI path has its own limiter — `OCP_TUI_MAX_CONCURRENT` (default `2`), independent of `CLAUDE_MAX_CONCURRENT`. Excess turns queue; a full queue returns a 503. Tune it up only on a host that can run more interactive `claude` sessions at once.
### ⚠️ Latency: TUI mode has a ~6-second floor, and it is immovable
**TUI mode cannot serve real-time or interactive-latency consumers.** This is a hard property of the
path, stated plainly so you can rule it out before building on it:
| | measured |
|---|---|
| **TTFT floor (first token)** | **≈ 6 s** — immovable |
| cold boot → input bar ready | ~1 s (per request; not the bottleneck) |
| OCP's own overhead above the CLI | ~24 s |
| direct Anthropic API, same prompt (for scale) | 0.841.64 s |
The ~6 s floor is the `claude` CLI itself: it always injects the full Claude Code system prompt plus
its tool definitions before your prompt, on every turn, no matter what you ask. No flag removes it
(`--exclude-dynamic-system-prompt-sections` was measured: **no effect** on the floor). Extended
thinking is *not* the cause — `OCP_TUI_EFFORT` already defaults to `low`, which is what cuts a
formerly-inherited `xhigh` down to this floor and collapses its variance.
On top of the floor you pay the model's generation time (a function of output length), and there is
**no progressive output** — see "No real token streaming" above; it is structural and cannot be
fixed. So a turn returns as one blob after the full generation completes.
**Use TUI mode for**: batch, background, and latency-insensitive work where the subscription pool is
the point. **Do not use it for**: anything a person is waiting on interactively, or any consumer with
a sub-5-second budget. Full measurements and methodology:
[`docs/plans/2026-07-13-tui-latency/`](docs/plans/2026-07-13-tui-latency/).
### Monitoring drift via `/health`
`GET /health` includes a `tui` block so you can poll for a silent billing-pool drift (the top risk after the 6/15 flip — a lost TTY flipping `cc_entrypoint` from `cli` to the metered `sdk-cli` pool would still return answers but burn metered credits). The block is **always present** (with `enabled:false` when TUI-mode is off):