diff --git a/README.md b/README.md index ea3fd9d..e6f6119 100644 --- a/README.md +++ b/README.md @@ -1033,7 +1033,7 @@ Then restart OCP. At boot you will see (with the env token set, isolated home au ### What changes / what doesn't - **Callers see no API change.** The response is a normal OpenAI completion object or chunked SSE — identical wire format. -- **No real token streaming — and it cannot be added.** TUI-mode buffers the full response then replays it as chunked SSE. You will see a delay then the complete response rather than real-time tokens. This is **structural, not a missing feature**: interactive `claude` exposes no byte-faithful incremental output. Its three observables are a *rendered* terminal (the pane's text has markdown source markers — `## `, `**`, code fences — stripped by rendering, so it is not the model's text), a transcript written at *event* granularity (the whole answer lands in one line ~0.3 s before the turn ends), and a debug log that reports *when* tokens arrive but never *what* they are (its mid-turn events carry no text payload at any verbosity). The one interface that emits token deltas, `--output-format stream-json`, requires `-p` — the metered-billing path TUI mode exists to avoid. Evidence: [`docs/plans/2026-07-13-tui-latency/streaming-spike.md`](docs/plans/2026-07-13-tui-latency/streaming-spike.md). +- **No real token streaming *today* — but it is achievable, and planned.** TUI-mode currently buffers the full response then replays it as chunked SSE: you see a delay, then the complete response. This is a limitation of the current implementation, **not** of the path — `claude` fires a `MessageDisplay` hook carrying incremental, byte-faithful `delta`s of the raw reply (they concatenate exactly to the final text, and stay prefix-stable), on the subscription pool, without `-p`. Wiring it into OCP's SSE is tracked as backlog item #2. What is *not* available is token-by-token granularity (the hook fires per rendered block, ~5–7 chunks per answer) — which is plenty for SSE. Evidence: [`docs/plans/2026-07-13-tui-latency/streaming-spike.md`](docs/plans/2026-07-13-tui-latency/streaming-spike.md). - **Cache and singleflight work normally.** TUI-mode writes the buffered response to the cache on success; cache-hits skip the interactive turn entirely. - **The host's `CLAUDE.md` / auto-memory is never injected.** OCP is a proxy — the proxied client (OpenClaw / your IDE) owns its own context and memory. TUI-mode always runs `claude` with `CLAUDE_CODE_DISABLE_CLAUDE_MDS` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY`, so a `CLAUDE.md` on the OCP host can never leak into proxied turns (verified live; see #4). Built-in tool schemas + the interactive system prompt remain (the inherent ~20–35K context floor of interactive mode); MCP is hard-disabled. - **Authenticate via `CLAUDE_CODE_OAUTH_TOKEN` in a credential-isolated home (recommended).** tmux does not forward the parent process's env to the pane, so OCP sets the token explicitly on the spawned `claude` when `CLAUDE_CODE_OAUTH_TOKEN` is present. But passing the token is **not enough on its own**: interactive `claude` *prefers* `~/.claude/.credentials.json` over the env var (unlike the `-p` path), so a stale `credentials.json` would shadow the token. With the env token set and `OCP_TUI_HOME` unset, OCP therefore runs claude in a **credential-isolated home** (`$HOME/.ocp-tui/home`) that has **no `credentials.json`** — so the env token is the only credential and is authoritative, and claude never runs the token-refresh path (so the single-use refresh token can't be corrupted by the spawn/teardown cycle). On a long-running host the credentials.json path produced a permanent `Please run /login · API Error: 401` that re-login could not fix (the next spawn re-corrupted it); the isolated home ends that at the root. Transcripts land under the same isolated home, so the answer-reader is unaffected. Without the env token, claude falls back to the real home's `credentials.json` (byte-for-byte the previous behaviour). (The token is visible in `ps` on the pane command — acceptable for the single-user A-path; the multi-user B-path is refused at boot.) See ADR 0007 PR-C / PR-D amendments. @@ -1059,9 +1059,11 @@ its tool definitions before your prompt, on every turn, no matter what you ask. thinking is *not* the cause — `OCP_TUI_EFFORT` already defaults to `low`, which is what cuts a formerly-inherited `xhigh` down to this floor and collapses its variance. -On top of the floor you pay the model's generation time (a function of output length), and there is -**no progressive output** — see "No real token streaming" above; it is structural and cannot be -fixed. So a turn returns as one blob after the full generation completes. +On top of the floor you pay the model's generation time (a function of output length). Progressive +output is not wired up **yet** (see "No real token streaming" above — it is achievable and planned), +so today a turn returns as one blob once generation completes. Note that streaming, when it lands, +will move the *first* byte earlier — it does **not** shorten the turn, and a consumer that needs the +complete answer gains nothing from it. **Use TUI mode for**: batch, background, and latency-insensitive work where the subscription pool is the point. **Do not use it for**: anything a person is waiting on interactively, or any consumer with diff --git a/docs/plans/2026-07-13-tui-latency/README.md b/docs/plans/2026-07-13-tui-latency/README.md index 18886b7..db8f10f 100644 --- a/docs/plans/2026-07-13-tui-latency/README.md +++ b/docs/plans/2026-07-13-tui-latency/README.md @@ -119,23 +119,28 @@ the effort level silently changes if the operator ever switches to env-token mod - **Risk**: none — banner confirms it stays on `Claude Max` (see `billing-banner.txt`). - ⚠️ Do **not** reach for `--bare` to shave boot: see above. -### 2. Real streaming instead of blocking on turn-terminal — ~~the big one (~20 s)~~ → **DEAD** +### 2. Real streaming instead of blocking on turn-terminal — **ACHIEVABLE → [`streaming-spike.md`](streaming-spike.md)** -> **2026-07-13 update — the prereq spike below was run, and it kills this item.** The transcript -> grows at *event* granularity (the whole answer lands in one line, ~0.3 s before terminal), and the -> pane is a **rendered** view whose `capture-pane` text no longer contains the answer's source bytes -> (`## `, `**`, code fences are gone) — so no pane-derived stream can be reconciled with the -> authoritative text. A third source (`--debug-file`) does emit mid-turn stream events at -> `CLAUDE_CODE_DEBUG_LOG_LEVEL=verbose` — but they carry **timing only, no text payload** (0 -> `text_delta` at any verbosity); its only byte-exact text is the end-of-turn `Stop` hook payload. -> `--output-format stream-json`, the only interface that emits token deltas, requires `-p` — the -> metered-billing path that TUI mode exists to avoid. **True token streaming is not achievable on the -> TUI path**; OCP's TUI SSE is replay-only. Note also that streaming would never have shortened a -> turn — only its first byte — so a consumer that needs the *complete* answer (the JSON-card case -> that motivated this investigation) gains nothing from it. +> **2026-07-13 update — the prereq spike was run. The answer is YES, but not from either source this +> item guessed at.** (a) The transcript grows at *event* granularity (the whole answer lands in one +> line, ~0.3 s before terminal) — dead. (b) The pane is a **rendered** view whose `capture-pane` text +> no longer contains the answer's source bytes (`## `, `**`, code fences are gone) — dead, and worse +> than "lossy": it is *not the model's text*. **But there is a third source neither this backlog nor +> the first spike considered: `claude` fires a `MessageDisplay` hook carrying incremental, +> byte-faithful `delta`s of the raw reply.** Verified live on a plain interactive TUI spawn (no `-p`), +> banner `· Claude Max`: 7 fires spread across generation, `concat(deltas) === T` **byte-exactly** +> (579 == 579), `T.startsWith(S)` true at every step, `## ` / `**` / ```` ```javascript ```` all +> present in the deltas. Granularity is block-level (~5–7 chunks/answer), not token-level — plenty for +> SSE. **Build it.** > -> Full evidence, the value re-assessment, and the maintainer's options: **[`streaming-spike.md`](streaming-spike.md)**. -> The original framing is preserved below for the record. +> ⚠️ Two corrections to this item as written: the **"~20 s" is wrong** (inferred from an external +> report, never measured through OCP — the same-turn decomposition puts OCP's own overhead at **~4 s**, +> n=1), and **streaming moves the first byte, not the last** — so a consumer needing the *complete* +> answer (the JSON-card case that motivated this) gains **nothing** from it. Build it for +> progressively-rendering consumers, not as a throughput win. +> +> Full evidence + implementer caveats (the hook is `forceSyncExecution` — claude BLOCKS on it): +> **[`streaming-spike.md`](streaming-spike.md)**. Original framing preserved below. Today `runTuiTurn` blocks on the transcript until the turn is *finished*. The pane is already rendering tokens incrementally the whole time — this harness proves you can observe first token diff --git a/docs/plans/2026-07-13-tui-latency/messagedisplay-deltas.jsonl b/docs/plans/2026-07-13-tui-latency/messagedisplay-deltas.jsonl new file mode 100644 index 0000000..3dc4f14 --- /dev/null +++ b/docs/plans/2026-07-13-tui-latency/messagedisplay-deltas.jsonl @@ -0,0 +1,7 @@ +{"hook_event_name": "MessageDisplay", "index": 0, "final": false, "delta": "## Mutex\n\n"} +{"hook_event_name": "MessageDisplay", "index": 1, "final": false, "delta": "A **mutual exclusion lock** prevents concurrent access to a shared resource, ensuring only one thread runs the critical section at a time.\n\n"} +{"hook_event_name": "MessageDisplay", "index": 2, "final": false, "delta": "- Acquiring a locked mutex blocks the caller until the current holder releases it.\n"} +{"hook_event_name": "MessageDisplay", "index": 3, "final": false, "delta": "- Failing to release a mutex causes a deadlock, freezing all waiting threads.\n\n```javascript\nconst { Mutex } = require('async-mutex');\n\nconst mutex = new Mutex();\n"} +{"hook_event_name": "MessageDisplay", "index": 4, "final": false, "delta": "let counter = 0;\n\nasync function increment() {\n const release = await mutex.acquire();\n try {\n"} +{"hook_event_name": "MessageDisplay", "index": 5, "final": false, "delta": " counter++; // only one caller here at a time\n } finally {\n release();\n }\n}\n"} +{"hook_event_name": "MessageDisplay", "index": 6, "final": true, "delta": "```"} diff --git a/docs/plans/2026-07-13-tui-latency/streaming-spike.md b/docs/plans/2026-07-13-tui-latency/streaming-spike.md index a8e2d86..8800cef 100644 --- a/docs/plans/2026-07-13-tui-latency/streaming-spike.md +++ b/docs/plans/2026-07-13-tui-latency/streaming-spike.md @@ -1,249 +1,237 @@ -# Backlog #2 (real streaming): the prereq spike says **no** +# Backlog #2 (real streaming): **achievable** — via the `MessageDisplay` hook **Date**: 2026-07-13 -**Status**: negative result — evidence, not a decision. The maintainer chooses what to do with it. -**Scope**: this answers the prereq spike that [`README.md`](README.md) § "Backlog #2" demanded *before* any streaming design: +**Status**: prereq-spike result. **Streaming IS achievable on the TUI path**, byte-faithfully, on the +subscription pool. Three obvious sources are dead ends; a fourth one works. +**Scope**: answers the prereq spike that [`README.md`](README.md) § "Backlog #2" demanded *before* any +streaming design: > **Prereq spike (do this before designing anything)**: does the transcript JSONL grow *during* a > turn, or only at the end? If only at the end, (a) is dead and you are stuck with (b). -The spike was run. **Both (a) and (b) are dead**, and so is a third candidate the original backlog -did not consider. The conclusion is stronger than "streaming is hard": **the interactive `claude` -CLI does not expose its token stream to any observer.** It renders tokens to a terminal and records -the *finished* text. There is no byte-faithful incremental source for a proxy to forward. +The answer: **(a) is dead, (b) is dead — and you are not stuck with either.** The CLI exposes its own +streaming interface as a **hook**, which the backlog did not consider. -**Measured on**: Mac mini / Claude Code **v2.1.207** / Sonnet 5 + Sonnet 4.6 / Claude Max / +**Measured on**: Mac mini / Claude Code **v2.1.207** / Sonnet 4.6 + Sonnet 5 / Claude Max / real-home mode. Every claim below is reproducible from the commands given. +> **Honesty note on how this document was produced.** Its first version concluded the exact opposite — +> "streaming is not achievable; the CLI exposes no byte-faithful incremental source" — and was **wrong**. +> An adversarial reviewer, commissioned specifically to *refute* it, found `MessageDisplay` on a second +> pass; its own first pass had enumerated the hook registry with a truncated grep (it reported 21 +> events — there are **30**). Both the wrong conclusion and its refutation are preserved here, because +> "we checked, it's impossible" is the most expensive kind of claim to get wrong: it closes a door and +> nobody re-opens it. + --- -## The three candidate sources, and how each one dies +## ✅ The source that works: the `MessageDisplay` hook + +`claude` fires a **`MessageDisplay`** hook as it renders each block of the assistant's reply. The +payload carries the **raw markdown source** of an incremental `delta`, plus a monotonic `index` and a +`final` flag: + +```json +{ "hook_event_name": "MessageDisplay", + "turn_id": "6cb31d21-…", "message_id": "84ab9832-…", + "index": 0, "final": false, "delta": "## Mutex\n\n" } +``` +*(payload also carries `session_id`, `transcript_path`, `prompt_id`, `cwd`)* + +Registered as an ordinary command hook via `--settings` on a **plain interactive TUI spawn** (no `-p`, +no `--bare`), `claude-sonnet-4-6`, `--effort low`. Banner verified: +`▝▜█████▛▘ Sonnet 4.6 with low effort · Claude Max` — **subscription pool, not metered billing**. + +One live turn — 7 fires, spread across generation: + +``` +index=0 final=false len= 10 '## Mutex\n\n' +index=1 final=false len= 140 'A **mutual exclusion lock** prevents concurrent access to a shar…' +index=2 final=false len= 83 '- Acquiring a locked mutex blocks the caller until the current h…' +index=3 final=false len= 163 '- Failing to release a mutex causes a deadlock, freezing all wai…' +index=4 final=false len= 96 'let counter = 0;\n\nasync function increment() {\n const release =…' +index=5 final=false len= 84 ' counter++; // only one caller here at a time\n } finally {\n …' +index=6 final=true len= 3 '```' +``` + +**Every invariant a proxy needs — all hold:** + +| requirement | result | +|---|---| +| **byte-faithful** — deltas are the model's *source*, not the rendered pane | ✅ `## `, `**`, ```` ```javascript ```` all present in the deltas | +| **exactness** — `concat(deltas) === T` (the transcript-authoritative text) | ✅ **true**, 579 == 579 bytes | +| **prefix-stable** — `T.startsWith(concat(deltas[0..n]))` at every n | ✅ **true at all 7 steps** | +| **incremental** — arrives during generation, not at the end | ✅ 7 fires spread across the turn | +| **no `-p`** — stays out of the metered `sdk-cli` pool | ✅ plain interactive TUI | +| **subscription pool** | ✅ banner `· Claude Max` | + +This is exactly the contract a streaming design needs: deltas forward straight into SSE +`delta.content` chunks, and the transcript's final text `T` stays a cheap end-of-turn assertion +(`concat === T`) instead of a reconciliation problem. + +### Caveats for the implementer + +- **Block-level granularity, not token-level** — 5–7 chunks for a ~600-byte answer, not one per token. + Plenty for SSE (`delta.content` has no minimum size), but do not promise token-by-token output. +- **⚠️ `forceSyncExecution: true` in the hook's source — `claude` BLOCKS on the hook.** A slow hook + adds latency to *every* delta. The hook must write and exit immediately (e.g. write to a FIFO / unix + socket that OCP reads; never work inline). **Measure the added per-delta latency.** +- Only `text` blocks fire it (`content.map(c => c.type === "text" ? c.text : "")`) — **thinking blocks + are excluded**, which is what OCP wants. +- OCP already owns the spawn (isolated HOME, its own flags), so injecting `--settings` with a + `MessageDisplay` hook sits inside the existing architecture. +- **`ALIGNMENT.md`**: this consumes `claude`'s **own** hook surface as emitted — forwarding, not + inventing. Not a new endpoint, not a fabricated protocol. (Class B / ADR 0007 — the TUI spawn is + OCP-owned; no `cli.js` citation applies.) + +### Reproduce in 60 seconds + +```bash +# hook script: append the payload (arrives on stdin) and exit immediately +printf '#!/bin/bash\ncat >> "$MD_LOG"; printf "\\n" >> "$MD_LOG"; exit 0\n' > /tmp/h.sh && chmod +x /tmp/h.sh +echo '{"hooks":{"MessageDisplay":[{"hooks":[{"type":"command","command":"MD_LOG=/tmp/deltas.jsonl /tmp/h.sh"}]}]}}' > /tmp/s.json + +# plain interactive claude in tmux (prefix NOT ocp-tui-*, and never kill-server) +tmux new-session -d -s md-probe -x 220 -y 50 \ + "claude --model claude-sonnet-4-6 --effort low --session-id $(uuidgen) --settings /tmp/s.json" +# …wait for '? for shortcuts', paste a markdown-producing prompt, press Enter… + +jq -r '"\(.index) \(.final) \(.delta|@json)"' /tmp/deltas.jsonl # incremental raw-markdown deltas +# then assert: concat(deltas) == extractLatestAssistantText(.jsonl) +``` + +--- + +## The three dead ends (still worth knowing — they say what NOT to build) ### (a) Incremental transcript reads — **dead: event granularity, not token granularity** -The transcript JSONL *does* grow during a turn, but it grows one **whole event at a time**, and the -assistant's text event is written as **one complete line** — it appears only ~0.3 s before the -terminal `turn_duration` event. +The transcript JSONL *does* grow during a turn, but one **whole event at a time**; the assistant's text +event is written as **one complete line**, appearing only ~0.3 s before the terminal `turn_duration`. -Observed (session `efd5b161`, `turn_duration: 7319 ms`, polling the file every 0.5 s): +Observed (session `efd5b161`, `turn_duration: 7319 ms`): ``` -bytes=64347 lines=13 → 64479 lines=14 → 92192 lines=21 (during the turn) - #6 t+0.0s type=user (the prompt) #15 t+4.7s type=assistant blocks=thinking #16 t+7.0s type=assistant blocks=text ← the ENTIRE answer, in one line #21 t+7.3s type=system subtype=turn_duration ← terminal ``` -Reading the transcript incrementally therefore buys ~0.3 s, not a token stream. The "clean option" -the backlog hoped for does not exist. +Cross-checked at **20 ms polling + `fs.watch`** (25× finer): a partial line **never touches disk** — +one write, `+1` line, carrying the complete answer. Also forced with the undocumented +`CLAUDE_CODE_INCLUDE_PARTIAL_MESSAGES=1`: still 1 assistant event, 0 partials (interactive mode has no +stream-json *sink* for it to write to). + +**The transcript is still needed** — as the terminal-turn signal, as the authoritative `concat === T` +check, and as the input to the existing honesty gates (auth-banner detection, `truncated`). It is just +not the *streaming* source. ### (b) `tmux capture-pane` diffing — **dead: the pane is a RENDERED view, not the text** -This is the one the backlog expected to fall back to, noting it is "lossy for exact text (wrapping, -scrollback, spinner lines)". The loss is **worse than formatting noise: the pane does not contain -the answer's source bytes at all.** claude's TUI *renders* markdown, and `capture-pane -p` strips -the ANSI styling that rendering produced — so the source markers are gone, irrecoverably. +The backlog expected to fall back to this, calling it "lossy … (wrapping, scrollback, spinner lines)". +The loss is far worse than formatting noise: **the pane does not contain the answer's source bytes at +all.** The TUI *renders* markdown, and `capture-pane -p` strips the ANSI that rendering produced. -Same turn, same lines — authoritative transcript text vs. what the pane holds -(`capture-pane -p -J -S -500`, pane width 220): +Same turn, same lines: ``` -TRANSCRIPT (authoritative T, via extractLatestAssistantText): - '## Semaphore' - '' - 'A **semaphore** is a synchronization primitive that controls access to a shared resource by …' - '' - '- Tracks available "permits" and decrements the count when a thread acquires access, …' - -PANE (the same turn): - '' - '⏺ Semaphore' ← '## ' heading became a bullet glyph - '' - ' A semaphore is a synchronization primitive that controls access to a shared resource by …' - '' ← '**' bold markers GONE; every line indented 2 sp - ' - Tracks available "permits" and decrements the count when a thread acquires access, …' +TRANSCRIPT (authoritative T): PANE (capture-pane -p -J -S -500): + '## Semaphore' '⏺ Semaphore' ← heading marker gone + '' '' + 'A **semaphore** is a synchro…' ' A semaphore is a synchro…' ← bold markers gone, indented ``` -Token-presence check **over the pane's answer region** (the prompt echo above it is excluded — it -does contain a literal `**`, because the prompt itself asked for bold): - -| token in the answer | in transcript `T` | in pane's answer region | +| token in the answer | in `T` | in the pane's answer region | |---|---|---| | `## ` (ATX heading) | yes | **no** — rendered as `⏺` | -| `**` (bold markers) | yes | **no** — rendered as ANSI bold, then stripped by `-p` | -| ` ```javascript ` (fence + language) | yes | **no** — fence gone, language tag gone | +| `**` (bold markers) | yes | **no** — rendered to ANSI bold, then stripped by `-p` | +| ` ```javascript ` (fence + language) | yes | **no** — fence and language tag both gone | | `- ` (list item) | yes | yes | -**`capture-pane -e` (keep the ANSI) does not rescue it — the inverse is provably non-unique.** -Minimal case, transcript `T` = ``"## Alpha\n\n**bravo**\n\n```javascript\nlet x=1;\n```"``: +*(A literal `**` does appear elsewhere in the pane — in the **prompt echo**, because the prompt asked +for bold. Not in the answer.)* + +**`capture-pane -e` (keeping the ANSI) does not rescue it — the inverse is provably non-unique.** +With `T` = ``"## Alpha\n\n**bravo**\n\n```javascript\nlet x=1;\n```"``: ``` ⏺\e[39m \e[1mAlpha\n\n\e[0m \e[1mbravo\n\n\e[0m \e[34mlet\e[39m x=\e[32m1\e[39m; ``` -`## Alpha` → **SGR 1 (bold)**. `**bravo**` → **SGR 1 (bold)**. *Identical ANSI* — an H2 and a bold -span are indistinguishable, never mind `**` vs `__`. The fence and its `javascript` tag are consumed -by the syntax highlighter into colours (`\e[34mlet`); recovering the tag would mean inverting a -highlighter, and `let x=1;` is valid in several languages. Marker check on the `-e` capture: `## `, -`**`, ```` ``` ````, ```` ```javascript ```` — **all absent**. +`## Alpha` → **SGR 1 (bold)**. `**bravo**` → **SGR 1 (bold)**. *Identical ANSI* — an H2 and a bold span +are indistinguishable, never mind `**` vs `__`. The fence and its `javascript` tag are consumed by the +syntax highlighter into colours; recovering the tag would mean inverting a highlighter, and +`let x=1;` is valid in several languages. -Consequence for any design built on pane diffing: with `S` = text streamed from the pane and `T` = -the transcript's authoritative text, the prefix invariant **`T.startsWith(S)` is false** — both raw -and with the 2-space indent stripped. Not "sometimes, on redraw" — **on essentially every answer -containing markdown**, which is most answers. - -And it cannot be repaired: - -- `capture-pane -e` (keep ANSI) recovers *styling*, never the *source spelling*: you cannot tell - `**bold**` from `__bold__`, recover a fence's language tag, or distinguish an H2 from bold text. -- Reconstructing markdown from rendered output is a lossy inverse with no unique solution, and it - would be pinned to one claude TUI version's theme. - -A proxy that streams pane text is streaming **something the model did not say**. For OCP that is not -a quality trade-off, it is a correctness violation (`ALIGNMENT.md` Core Principle: forward what -`cli.js` emits; do not invent). +So `T.startsWith(paneText)` is **false** — raw and indent-stripped, on essentially every markdown +answer. A proxy streaming pane text would be streaming **something the model did not say**. With +`MessageDisplay` available there is no reason to go near it. ### (c) `--debug-file` — **dead: it logs stream *timing*, never stream *content*** -Not considered in the original backlog; checked because it is the only other observable claude writes. +Worth stating precisely, because a casual check misleads in **both** directions here. -**There ARE mid-turn stream events in the debug log** — this is the one place a casual check -misleads, so it is worth stating precisely. The default log level is `debug`, which **suppresses -every `verbose` site**. Raise it and per-chunk lines appear, spread across the generation: +The default log level is `debug`, which **suppresses every `verbose` site**. Raise it and per-chunk +lines *do* appear, spread across generation: ```bash CLAUDE_CODE_DEBUG_LOG_LEVEL=verbose claude --debug-file /tmp/d.log … ``` ``` 05:51:11.088 [VERBOSE] [shoji-engine] yield stream_event/- ← 16 of these, mid-turn, -05:51:11.537 [VERBOSE] [shoji-engine] yield stream_event/- spread over ~3.9 s of generation -… -05:51:14.910 [VERBOSE] [shoji-engine] yield assistant/- +05:51:11.537 [VERBOSE] [shoji-engine] yield stream_event/- over ~3.9 s of generation 05:51:15.192 [DEBUG] [shoji-engine] turn 1 end (usage in=575 out=255 api=6736ms stop=end_turn resultLen=857) ``` -**But they carry no payload.** The format is `yield /` — a bare presence marker. At -verbose level, with every category filter, the log contains: +**But they carry no payload** — the format is `yield /`, a bare presence marker. Run with +no category filter (i.e. all categories) at verbose level: `content_block_delta` = **0**, `text_delta` = +**0**, `content_block_start` / `message_start` = **0**. The only byte-exact text in the log is the +end-of-turn `Stop` hook payload (`"last_assistant_message":"## Title\n\n**alpha bravo charlie**"`) — +transcript granularity. The log tells you **when** tokens arrive, never **what** they are. It is also +~2.7 MB per turn. -| event | count | +### Also checked, also not the answer + +| candidate | outcome | |---|---| -| `content_block_delta` | **0** | -| `text_delta` | **0** | -| `content_block_start` / `message_start` | **0** | - -The only byte-exact text in the log is the **end-of-turn `Stop` hook payload** -(`"last_assistant_message":"## Title\n\n**alpha bravo charlie**"` — note `## ` and `**` survive, so -this *is* the real text), which is the same granularity as the transcript. So the debug log gives you -**when** tokens arrive, never **what** they are. It is also ~2.7 MB per turn. - -*(An incremental "a token arrived" signal with no token in it cannot feed an SSE `delta.content`.)* - -### Everything else that could plausibly carry tokens — also checked, also dead - -An adversarial second pass (independent reviewer, tasked with *refuting* this document) enumerated -the rest of the search space rather than sampling it. Nothing survived: - -| candidate | how it dies | -|---|---| -| **Hooks** — is there a per-message/per-chunk hook? | The hook registry was enumerated from the shipped binary: `PreToolUse, PostToolUse, PostToolUseFailure, PostToolBatch, Notification, UserPromptSubmit, UserPromptExpansion, SessionStart, SessionEnd, Stop, StopFailure, SubagentStart, SubagentStop, PreCompact, PostCompact, PermissionRequest, PermissionDenied, Setup, TeammateIdle, TaskCreated, TaskCompleted`. **No streaming/per-chunk hook exists.** | -| `CLAUDE_CODE_INCLUDE_PARTIAL_MESSAGES=1` (undocumented env twin of the flag; bypasses flag validation) | Ran live in a TUI turn: transcript still had **1** assistant event, **0** partials. Interactive mode has no stream-json *sink* for it to write to. Banner stayed `· Claude Max`. | +| `--output-format stream-json` (the one interface that emits `text_delta`) | **requires `--print`/`-p`** → `cc_entrypoint=sdk-cli` → the **metered** credit pool, which is exactly what TUI mode exists to avoid. Reproduced live. | +| `--input-format stream-json` | `Error: --input-format=stream-json requires output-format=stream-json` → same gate. | +| `CLAUDE_CODE_INCLUDE_PARTIAL_MESSAGES=1` (undocumented) | No stream-json sink in interactive mode → no partials. Banner stayed `· Claude Max`. | | `sessionMirror` (undocumented) | Gated on `outputFormat === "stream-json"` → the `-p` family. | -| `--sdk-url` (hidden) | Forces stream-json + non-interactive → `sdk-cli` (metered pool). | -| `--input-format stream-json` | Live: `Error: --input-format=stream-json requires output-format=stream-json` → which requires `--print`. And it is an *input* format regardless. | -| `~/.claude/sessions/.json` (modified mid-turn) | Registry metadata only: `{pid, sessionId, cwd, status, version, entrypoint:"cli", kind:"interactive"}`. **No assistant text.** (Its `entrypoint:"cli"` is incidental independent confirmation that the TUI path stays on the subscription pool.) | -| `~/.claude/history.jsonl` (modified mid-turn) | Keys: `[display, pastedContents, timestamp, project, sessionId]` — **user prompts only**; the answer text is absent. | -| `~/.claude/settings.json` | No key for partial/streaming output. | -| **Ask the model to emit plain text** (so the pane render is faithful) | Would require OCP to **mutate the caller's prompt** — a correctness violation for a proxy (`ALIGNMENT.md`: forward what `cli.js` emits; do not invent), it would corrupt callers who legitimately want markdown or JSON, and it *still* would not be byte-faithful (the TUI keeps wrapping and 2-space indenting). Rejected. | +| `--sdk-url` (hidden) | Forces stream-json + non-interactive → `sdk-cli`. *(inferred from the minified bundle; not banner-tested)* | +| `~/.claude/sessions/.json` | Registry metadata only (`{pid, sessionId, cwd, status, version, entrypoint:"cli", kind:"interactive"}`). No assistant text. *(Its `entrypoint:"cli"` incidentally confirms the TUI path stays on the subscription pool.)* | +| `~/.claude/history.jsonl` | User prompts only; the answer text is absent. | +| Asking the model to emit plain text (so the pane renders faithfully) | Would mean **mutating the caller's prompt** — a correctness violation for a proxy, and still not byte-faithful (wrapping + indent remain). Rejected. | --- -## What this means +## Value: what streaming actually buys (read before building) -**True token streaming is not achievable on the TUI path**, at any effort, with any flag combination -available in claude 2.1.207. The three observables are: a rendered terminal (lossy beyond -repair), a transcript written at event granularity (the answer lands whole, end-of-turn), and a debug -log that reports *when* tokens arrive but never *what* they are. -`--output-format stream-json` — the one interface that *does* emit `text_delta` events — works -**only with `--print`/`-p`**, and the `-p` path is precisely what TUI mode exists to avoid (it is -classified `cc_entrypoint=sdk-cli` → the metered credit pool, not the subscription pool). The -constraint is structural, not a missing feature. +Streaming is *possible*. Whether it is *worth it* depends on the consumer, and the honest answer is +uncomfortable: -**Therefore OCP's SSE on the TUI path is, and remains, replay-only**: the answer is buffered until -the turn is terminal and then replayed as SSE chunks (`server.mjs`, the `streamStringAsSSE` branch). -It is wire-valid OpenAI SSE; it is not progressive. +- **Streaming never makes the answer arrive sooner. It moves the *first* byte, not the *last*.** The + final token lands at the same wall-clock moment either way. +- So a consumer that must have the **complete** answer before it can act — e.g. one parsing a structured + JSON reply, **which is exactly the 知音 AI use case that motivated this entire investigation** — gains + **nothing at all**. Only a **progressively-rendering** consumer (a chat UI) gains. -### What streaming would have bought — smaller than the backlog implied +And the number the backlog attached to this item was wrong: -Worth stating plainly, because it changes the value calculus: +- The backlog's "~20 s" was inferred from an external 30–32 s report, **never measured through OCP**. + Measured through a real OCP instance (TUI mode, `claude-sonnet-4-6`, ~1850-token prompt, n=5): + **median 11.30 s** before [#156](https://github.com/dtzp555-max/ocp/pull/156), **9.55 s** after. +- **Same-turn decomposition** (baseline row `i=5`): **11.563 s** wall through OCP vs `turn_duration: + 7.319 s` of CLI-internal time on that same turn → **OCP's own overhead ≈ 4.2 s** (n=1, baseline + `effort=high` config). *Caveats*: n=1; and `turn_duration` is the CLI's internal duration of an + **OCP-driven** turn, not a separate "native" baseline. Do **not** subtract this `effort=high` 7.3 s + from the `effort=low` 9.55 s median — a low-effort turn generates faster, so mixing them + *understates* the overhead. +- So OCP's own overhead is **single-digit seconds**, not ~20 s. The rest of any large number is the + model generating a long answer — which streaming hides but does not shorten. -- **Streaming never makes the answer arrive sooner.** It makes the *first byte* arrive sooner. A - consumer that needs the *complete* answer before it can act — e.g. a JSON-card consumer parsing a - structured reply, which is exactly the 知音 AI use case that motivated this whole investigation — - gains **nothing at all** from streaming. Only a progressively-rendering consumer (a chat UI) gains. -- The backlog's "~20 s" for item #2 was inferred from an external report of 30–32 s, not measured - through OCP. Measured through a real OCP instance (TUI mode, `claude-sonnet-4-6`, ~1850-token - prompt, n=5): **median 11.30 s** before PR #156, **9.55 s** after. - - **Same-turn decomposition** (the honest form — one turn, both numbers): baseline row `i=5` took - **11.563 s** end-to-end through OCP, and that same turn's transcript records - `turn_duration: 7.319 s` of CLI-internal time. → **OCP's own overhead ≈ 4.2 s** (n=1, baseline - `effort=high` config). Not 20 s. - - Two caveats, stated so the number is not over-trusted: this is **n=1**, and `turn_duration` is the - *CLI's* internal duration on an **OCP-driven** turn (it is not a "native, non-OCP" baseline — the - same interactive `claude` is doing the work either way). Do **not** subtract this 7.3 s - (`effort=high`) from the 9.55 s `effort=low` median: a low-effort turn generates faster, so its own - `turn_duration` would be lower, and mixing the two would *understate* the overhead. There is no - `turn_duration` sample for the `effort=low` config. - - Either way the direction is settled: OCP's overhead is **single-digit seconds**, not the ~20 s the - plan assumed. The rest of any large number is the model generating a long answer — which streaming - hides but does not shorten. (A 30 s turn is a 30 s turn either way; the last token lands at the same - wall-clock moment.) - -The honest remaining levers on the TUI path are therefore: the **~6 s TTFT floor** (immovable — see -[`README.md`](README.md)), the **generation time** (a function of output length — not OCP's to -optimize), and OCP's **own ~4 s of overhead** (n=1 same-turn decomposition; per-request cold boot ≈1 s → backlog #3 warm pool; -plus readiness/paste/transcript poll granularity). - ---- - -## Options for the maintainer (this document does not choose) - -1. **Accept and document (recommended).** TUI mode cannot stream; its SSE is replay-only. Record the - constraint next to the ~6 s TTFT floor already in `README.md`, and close backlog #2 as *not - achievable honestly*. Redirect the effort to the overhead that *is* recoverable (#3). -2. **Ship lossy "rendered-text" streaming behind an explicit opt-in.** Technically possible; the - client would receive the TUI's rendered view — headings as `⏺`, bold markers stripped, code - fences gone, every line indented. **Recommended against**: it delivers bytes the model did not - produce, silently corrupts structured replies (the JSON/marker protocols proxy consumers actually - use), and pins OCP's output to a claude TUI theme. It also cannot be reconciled with the - transcript afterwards — OpenAI SSE deltas cannot be un-sent. -3. **Reconstruct markdown from `capture-pane -e` ANSI styling.** High effort, brittle across claude - versions, and *still* not byte-exact (no unique inverse). Recommended against. - -## Reproduction - -```bash -# (a) transcript growth granularity — poll a live turn's JSONL -# watch bytes/lines while a TUI turn runs; dump event types + timestamps afterwards - -# (b) pane vs transcript — the decisive one -# spawn claude in tmux (prefix NOT ocp-tui-*, never kill-server), ask for markdown, -# capture `tmux capture-pane -p -J -S -500` frames during the turn, then diff the pane's -# answer region against extractLatestAssistantText() over the session's transcript JSONL. -# Expect: T.startsWith(paneText) === false, '## ' and '**' and '```' absent from the pane. - -# (c) debug log — NOTE: default level is `debug`, which SUPPRESSES the verbose stream sites. -# Raise it, or you will wrongly conclude there are no mid-turn events at all. -CLAUDE_CODE_DEBUG_LOG_LEVEL=verbose \ - claude --model claude-sonnet-5 --session-id "$(uuidgen)" --effort low --debug-file /tmp/d.log -grep -c 'yield stream_event' /tmp/d.log # → 16 (mid-turn — timing exists!) -grep -cE 'content_block_delta|text_delta' /tmp/d.log # → 0 (…but no text payload, ever) -grep -o 'last_assistant_message":"[^"]*' /tmp/d.log # → byte-exact text, only at Stop-hook time -``` - -Banner check (mandatory for every spawn-flag change, per [`billing-banner.txt`](billing-banner.txt)): -every configuration used in this spike was verified to stay on `· Claude Max`. +**Recommendation**: build it — the contract is clean and the cost is small — but size the expectation +honestly. It is a *perceived-latency* feature for progressively-rendering consumers, not a throughput +win, and it does not move the **~6 s TTFT floor** ([`README.md`](README.md)) that rules TUI mode out for +interactive-latency consumers regardless.