taodeng/ocp
1
0
Fork 0
mirror of https://github.com/dtzp555-max/ocp.git synced 2026-07-22 05:25:08 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(plans): TUI streaming is not achievable — prereq spike result + honest README constraints

Browse Source 
Backlog #2 of docs/plans/2026-07-13-tui-latency demanded a prereq spike before any
streaming design: does the transcript JSONL grow during a turn, or only at the end?
The spike was run. All three candidate sources are dead:

  (a) transcript JSONL — grows at EVENT granularity; the assistant's text event is
      written as ONE complete line, ~0.3s before the terminal turn_duration event
      (observed: turn_duration 7319ms; text event at t+7.0s, terminal at t+7.3s).
  (b) tmux capture-pane — the pane is a RENDERED view, not the text. Same turn,
      transcript T = '## Semaphore\n\nA **semaphore** is a synchronization…'
      pane        = '⏺ Semaphore' / '  A semaphore is a synchronization…'
      '## ', '**' and ```-fences are absent from the pane entirely (rendered to ANSI,
      then stripped by capture-pane -p). T.startsWith(paneText) is FALSE both raw and
      indent-stripped — not on redraw, but on essentially every markdown answer.
      capture-pane -e recovers styling, never source spelling: no unique inverse.
  (c) --debug-file — byte-exact ('last_assistant_message':'## Title\n\n**alpha…'),
      but only inside end-of-turn Stop-hook payloads; zero content_block_delta /
      text_delta events; ~2.7MB per turn.

--output-format stream-json, the only interface emitting token deltas, requires -p —
the metered-billing path TUI mode exists to avoid (cc_entrypoint=sdk-cli). The
constraint is structural. OCP's TUI SSE is, and remains, replay-only.

Also corrects this plan's own "~20s waiting for the whole turn" decomposition, which
was inferred from an external 30-32s report and never measured through OCP. Measured
through a real OCP instance (TUI, claude-sonnet-4-6, n=5): median 11.30s before #156,
9.55s after, vs a native turn_duration of ~7.3s → OCP's own overhead is ~2-4s, not
~20s. The remainder is generation time, which streaming would not shorten (it moves
the first byte, not the last) — so a consumer needing the COMPLETE answer, which is
the JSON-card case that motivated this work, would have gained nothing from streaming.

Backlog #4 measured while here: --exclude-dynamic-system-prompt-sections gives ZERO
marginal benefit (TTFT median 6.39s vs 6.17s for --effort low alone, n=5, one worse
outlier). Do not adopt. Banner stayed on Claude Max.

README: documents the ~6s TTFT floor plainly (TUI mode cannot serve interactive-latency
consumers) and states that no-token-streaming is structural rather than a missing feature.

No code change. No version bump (docs-only). Not endpoint-touching: no server.mjs diff,
so no cli.js citation applies.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VqgWJcjxrjjL9L9SkpZyXR
This commit is contained in:
taodeng
2026-07-13 15:32:41 +10:00
co-authored by Claude Fable 5
parent 5258d5d395
commit 6848f9751c
4 changed files with 254 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-07-13-tui-latency/README.md
+37 -3
View File
@@ -1,7 +1,9 @@
# TUI-mode latency: measured floor, and the four things worth fixing
**Date**: 2026-07-13
**Status**: findings + backlog (no code changed yet)
**Status**: findings + backlog. **Superseded in part** — see the dated update boxes below.
Item #1 shipped ([#156](https://github.com/dtzp555-max/ocp/pull/156)); item #2 is **dead**
([`streaming-spike.md`](streaming-spike.md)); item #4 measured, **no effect**; item #3 stands.
**Measured on**: Mac mini / macOS 26.5.2 / Claude Code **v2.1.207** / Sonnet 5 / Claude Max subscription / **real-home mode** (no `CLAUDE_CODE_OAUTH_TOKEN`, no `OCP_TUI_HOME` in the service env)
**Evidence**: [`measurements.jsonl`](measurements.jsonl) — **n=15** (3 configs × 5) · banner captures [`billing-banner.txt`](billing-banner.txt) · harness [`floor.sh`](floor.sh)
@@ -47,6 +49,17 @@ All rows in [`measurements.jsonl`](measurements.jsonl); every number below is re
before returning anything. There is no streaming path. The ~20 s delta between this harness's
real TTFT and OCP's reported 30–32 s is exactly that.
> **⚠️ 2026-07-13 correction — this decomposition attributes the ~20 s to the wrong thing.** It was
> inferred from the external 30–32 s report, never measured *through* OCP. It has since been measured
> through a real OCP instance (TUI mode, `claude-sonnet-4-6`, the same ~1850-token prompt, n=5):
> **median 11.30 s** before [#156](https://github.com/dtzp555-max/ocp/pull/156), **9.55 s** after —
> against a native `turn_duration` of ~7.3 s for a comparable turn. So **OCP's own overhead above the
> CLI is ~2–4 s, not ~20 s.** The rest of any larger number is the model *generating a long answer*,
> which the blocking wait does not cause and streaming would not shorten — it would only move the
> first byte earlier. The 30–32 s figure therefore reflects a much longer output (and/or the
> then-inherited `xhigh` effort), not 20 s of OCP dead time. See
> [`streaming-spike.md`](streaming-spike.md) § "What streaming would have bought".
---
## ⚠️ Blocking constraint: `--bare` silently drops you off the subscription pool
@@ -105,7 +118,21 @@ the effort level silently changes if the operator ever switches to env-token mod
- **Risk**: none — banner confirms it stays on `Claude Max` (see `billing-banner.txt`).
- ⚠️ Do **not** reach for `--bare` to shave boot: see above.
### 2. Real streaming instead of blocking on turn-terminal — **the big one (~20 s)**
### 2. Real streaming instead of blocking on turn-terminal — ~~the big one (~20 s)~~ → **DEAD**
> **2026-07-13 update — the prereq spike below was run, and it kills this item.** The transcript
> grows at *event* granularity (the whole answer lands in one line, ~0.3 s before terminal), and the
> pane is a **rendered** view whose `capture-pane` text no longer contains the answer's source bytes
> (`## `, `**`, code fences are gone) — so no pane-derived stream can be reconciled with the
> authoritative text. A third source (`--debug-file`) is byte-exact but also end-of-turn.
> `--output-format stream-json`, the only interface that emits token deltas, requires `-p` — the
> metered-billing path that TUI mode exists to avoid. **True token streaming is not achievable on the
> TUI path**; OCP's TUI SSE is replay-only. Note also that streaming would never have shortened a
> turn — only its first byte — so a consumer that needs the *complete* answer (the JSON-card case
> that motivated this investigation) gains nothing from it.
>
> Full evidence, the value re-assessment, and the maintainer's options: **[`streaming-spike.md`](streaming-spike.md)**.
> The original framing is preserved below for the record.
Today `runTuiTurn` blocks on the transcript until the turn is *finished*. The pane is already
rendering tokens incrementally the whole time — this harness proves you can observe first token
@@ -132,7 +159,14 @@ the background) amortizes it to zero for any workload below the pool refill rate
#148 — pooled panes must not look like zombies to the sweep.
- Lower priority than #1 and #2: it is the smallest slice.
### 4. Trim the prefill — **probably not worth it; know the floor**
### 4. Trim the prefill — ~~probably not worth it~~ **MEASURED: no effect. Do not adopt.**
> **2026-07-13 update.** `--exclude-dynamic-system-prompt-sections` was measured with the same
> harness (`floor.sh`, n=5, Sonnet 5, on top of `--effort low`): **TTFT median 6.39 s**
> (5.87–10.54 s) vs **6.17 s** (5.87–6.44 s) for `--effort low` alone — i.e. **zero marginal
> benefit**, within noise and with one worse outlier. The banner stayed on `· Claude Max`
> (no billing-pool drop), but there is no win to bank. The ~6 s floor stands as stated below.
After #1–#3, the floor is **~6 s**, and it does not go lower. `claude` always injects the full
Claude Code system prompt + tool definitions (thousands to tens of thousands of prefill tokens)
docs/plans/2026-07-13-tui-latency/prefill-spike-measurements.jsonl
+5
View File
@@ -0,0 +1,5 @@
{"i":1,"tag":"effort-low-exclude-dynamic","model":"claude-sonnet-5","extra_args":"--effort low --exclude-dynamic-system-prompt-sections","prompt_chars":7451,"boot_ms":934,"ttft_ms":5867,"complete_ms":9953}
{"i":2,"tag":"effort-low-exclude-dynamic","model":"claude-sonnet-5","extra_args":"--effort low --exclude-dynamic-system-prompt-sections","prompt_chars":7451,"boot_ms":1275,"ttft_ms":6388,"complete_ms":9874}
{"i":3,"tag":"effort-low-exclude-dynamic","model":"claude-sonnet-5","extra_args":"--effort low --exclude-dynamic-system-prompt-sections","prompt_chars":7451,"boot_ms":874,"ttft_ms":10537,"complete_ms":11782}
{"i":4,"tag":"effort-low-exclude-dynamic","model":"claude-sonnet-5","extra_args":"--effort low --exclude-dynamic-system-prompt-sections","prompt_chars":7451,"boot_ms":1170,"ttft_ms":6379,"complete_ms":9947}
{"i":5,"tag":"effort-low-exclude-dynamic","model":"claude-sonnet-5","extra_args":"--effort low --exclude-dynamic-system-prompt-sections","prompt_chars":7451,"boot_ms":1329,"ttft_ms":6443,"complete_ms":9884}
docs/plans/2026-07-13-tui-latency/streaming-spike.md
+184
View File
@@ -0,0 +1,184 @@
# Backlog #2 (real streaming): the prereq spike says **no**
**Date**: 2026-07-13
**Status**: negative result — evidence, not a decision. The maintainer chooses what to do with it.
**Scope**: this answers the prereq spike that [`README.md`](README.md) § "Backlog #2" demanded *before* any streaming design:
> **Prereq spike (do this before designing anything)**: does the transcript JSONL grow *during* a
> turn, or only at the end? If only at the end, (a) is dead and you are stuck with (b).
The spike was run. **Both (a) and (b) are dead**, and so is a third candidate the original backlog
did not consider. The conclusion is stronger than "streaming is hard": **the interactive `claude`
CLI does not expose its token stream to any observer.** It renders tokens to a terminal and records
the *finished* text. There is no byte-faithful incremental source for a proxy to forward.
**Measured on**: Mac mini / Claude Code **v2.1.207** / Sonnet 5 + Sonnet 4.6 / Claude Max /
real-home mode. Every claim below is reproducible from the commands given.
---
## The three candidate sources, and how each one dies
### (a) Incremental transcript reads — **dead: event granularity, not token granularity**
The transcript JSONL *does* grow during a turn, but it grows one **whole event at a time**, and the
assistant's text event is written as **one complete line** — it appears only ~0.3 s before the
terminal `turn_duration` event.
Observed (session `efd5b161`, `turn_duration: 7319 ms`, polling the file every 0.5 s):
```
bytes=64347 lines=13 → 64479 lines=14 → 92192 lines=21 (during the turn)
#6 t+0.0s type=user (the prompt)
#15 t+4.7s type=assistant blocks=thinking
#16 t+7.0s type=assistant blocks=text ← the ENTIRE answer, in one line
#21 t+7.3s type=system subtype=turn_duration ← terminal
```
Reading the transcript incrementally therefore buys ~0.3 s, not a token stream. The "clean option"
the backlog hoped for does not exist.
### (b) `tmux capture-pane` diffing — **dead: the pane is a RENDERED view, not the text**
This is the one the backlog expected to fall back to, noting it is "lossy for exact text (wrapping,
scrollback, spinner lines)". The loss is **worse than formatting noise: the pane does not contain
the answer's source bytes at all.** claude's TUI *renders* markdown, and `capture-pane -p` strips
the ANSI styling that rendering produced — so the source markers are gone, irrecoverably.
Same turn, same lines — authoritative transcript text vs. what the pane holds
(`capture-pane -p -J -S -500`, pane width 220):
```
TRANSCRIPT (authoritative T, via extractLatestAssistantText):
'## Semaphore'
''
'A **semaphore** is a synchronization primitive that controls access to a shared resource by …'
''
'- Tracks available "permits" and decrements the count when a thread acquires access, …'
PANE (the same turn):
''
'⏺ Semaphore' ← '## ' heading became a bullet glyph
''
' A semaphore is a synchronization primitive that controls access to a shared resource by …'
'' ← '**' bold markers GONE; every line indented 2 sp
' - Tracks available "permits" and decrements the count when a thread acquires access, …'
```
Token-presence check over the whole final frame:
| token in the answer | in transcript `T` | in pane |
|---|---|---|
| `## ` (ATX heading) | yes | **no** — rendered as `⏺` |
| `**` (bold markers) | yes | **no** — rendered as ANSI bold, then stripped by `-p` |
| ` ```javascript ` (fence + language) | yes | **no** — fence gone, language tag gone |
| `- ` (list item) | yes | yes |
Consequence for any design built on pane diffing: with `S` = text streamed from the pane and `T` =
the transcript's authoritative text, the prefix invariant **`T.startsWith(S)` is false** — both raw
and with the 2-space indent stripped. Not "sometimes, on redraw" — **on essentially every answer
containing markdown**, which is most answers.
And it cannot be repaired:
- `capture-pane -e` (keep ANSI) recovers *styling*, never the *source spelling*: you cannot tell
`**bold**` from `__bold__`, recover a fence's language tag, or distinguish an H2 from bold text.
- Reconstructing markdown from rendered output is a lossy inverse with no unique solution, and it
would be pinned to one claude TUI version's theme.
A proxy that streams pane text is streaming **something the model did not say**. For OCP that is not
a quality trade-off, it is a correctness violation (`ALIGNMENT.md` Core Principle: forward what
`cli.js` emits; do not invent).
### (c) `--debug-file` — **dead: byte-exact, but only at end-of-turn**
Not considered in the original backlog; checked for completeness because it is the only other
observable claude writes.
`claude --debug-file <path>` does contain the **byte-exact raw markdown** — but only inside
**end-of-turn hook payloads**:
```
[DEBUG] Hooks: Parsed initial response: {… "hook_event_name":"Stop", …
"last_assistant_message":"## Title\n\n**alpha bravo charlie**" …}
```
Note `## ` and `**` survive here — this is the real text. But it arrives with the `Stop` hook, i.e.
the same end-of-turn granularity as the transcript. Grepping the log for streaming events
(`content_block_delta` / `text_delta`) finds **zero**. The log is also ~2.7 MB per turn, which rules
it out as a production mechanism regardless.
---
## What this means
**True token streaming is not achievable on the TUI path**, at any effort, with any flag combination
available in claude 2.1.207. The three observables are: a rendered terminal (lossy beyond repair), a
transcript written at event granularity (end-of-turn), and a debug log written at end-of-turn.
`--output-format stream-json` — the one interface that *does* emit `text_delta` events — works
**only with `--print`/`-p`**, and the `-p` path is precisely what TUI mode exists to avoid (it is
classified `cc_entrypoint=sdk-cli` → the metered credit pool, not the subscription pool). The
constraint is structural, not a missing feature.
**Therefore OCP's SSE on the TUI path is, and remains, replay-only**: the answer is buffered until
the turn is terminal and then replayed as SSE chunks (`server.mjs`, the `streamStringAsSSE` branch).
It is wire-valid OpenAI SSE; it is not progressive.
### What streaming would have bought — smaller than the backlog implied
Worth stating plainly, because it changes the value calculus:
- **Streaming never makes the answer arrive sooner.** It makes the *first byte* arrive sooner. A
consumer that needs the *complete* answer before it can act — e.g. a JSON-card consumer parsing a
structured reply, which is exactly the 知音 AI use case that motivated this whole investigation —
gains **nothing at all** from streaming. Only a progressively-rendering consumer (a chat UI) gains.
- The backlog's "~20 s" for item #2 was inferred from an external report of 30–32 s, not measured
through OCP. Measured through a real OCP instance (TUI mode, `claude-sonnet-4-6`, ~1850-token
prompt, n=5): **median 11.30 s** before PR #156, **9.55 s** after — against a native
`turn_duration` of ~7.3 s for a comparable turn. So OCP's own overhead over the CLI is roughly
**2–4 s**, not 20 s; the rest of any large number is the model generating a long answer, which
streaming hides but does not shorten. (A 30 s turn is a 30 s turn either way — the last token
lands at the same wall-clock moment.)
The honest remaining levers on the TUI path are therefore: the **~6 s TTFT floor** (immovable — see
[`README.md`](README.md)), the **generation time** (a function of output length — not OCP's to
optimize), and OCP's **own 2–4 s of overhead** (per-request cold boot ≈1 s → backlog #3 warm pool;
plus readiness/paste/transcript poll granularity).
---
## Options for the maintainer (this document does not choose)
1. **Accept and document (recommended).** TUI mode cannot stream; its SSE is replay-only. Record the
constraint next to the ~6 s TTFT floor already in `README.md`, and close backlog #2 as *not
achievable honestly*. Redirect the effort to the overhead that *is* recoverable (#3).
2. **Ship lossy "rendered-text" streaming behind an explicit opt-in.** Technically possible; the
client would receive the TUI's rendered view — headings as `⏺`, bold markers stripped, code
fences gone, every line indented. **Recommended against**: it delivers bytes the model did not
produce, silently corrupts structured replies (the JSON/marker protocols proxy consumers actually
use), and pins OCP's output to a claude TUI theme. It also cannot be reconciled with the
transcript afterwards — OpenAI SSE deltas cannot be un-sent.
3. **Reconstruct markdown from `capture-pane -e` ANSI styling.** High effort, brittle across claude
versions, and *still* not byte-exact (no unique inverse). Recommended against.
## Reproduction
```bash
# (a) transcript growth granularity — poll a live turn's JSONL
# watch bytes/lines while a TUI turn runs; dump event types + timestamps afterwards
# (b) pane vs transcript — the decisive one
# spawn claude in tmux (prefix NOT ocp-tui-*, never kill-server), ask for markdown,
# capture `tmux capture-pane -p -J -S -500` frames during the turn, then diff the pane's
# answer region against extractLatestAssistantText() over the session's transcript JSONL.
# Expect: T.startsWith(paneText) === false, '## ' and '**' and '```' absent from the pane.
# (c) debug log
claude --model claude-sonnet-5 --session-id "$(uuidgen)" --effort low --debug-file /tmp/d.log
grep -cE 'content_block_delta|text_delta' /tmp/d.log # → 0
grep -o 'last_assistant_message":"[^"]*' /tmp/d.log # → byte-exact text, at Stop-hook time
```
Banner check (mandatory for every spawn-flag change, per [`billing-banner.txt`](billing-banner.txt)):
every configuration used in this spike was verified to stay on `· Claude Max`.