taodeng/ocp
1
0
Fork 0
mirror of https://github.com/dtzp555-max/ocp.git synced 2026-07-22 05:25:08 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
551d4e7db6c627494a93237647bfd8e2a6bfdf63
ocp/docs/plans/2026-07-13-tui-latency/README.md
T
taodengandClaude Fable 5 ddaea4df17 docs: REVERSE the streaming verdict — MessageDisplay hook makes it achievable 
The previous commit on this branch concluded TUI streaming was impossible. That was
WRONG, and this corrects it before it could be merged.

The adversarial reviewer commissioned to refute the claim found, on a second pass while
verifying the fold-in, that its OWN first-pass hook enumeration had been truncated by a
400-char grep cap: it reported 21 hook events; the shipped 2.1.207 bundle has 30.
Event #30 is MessageDisplay.

Independently reproduced before acting on it (30 events confirmed via `strings` on the
binary; payload shape `hook_event_name:"MessageDisplay",turn_id,message_id,index,final,
delta`), then live-tested with a MessageDisplay command hook registered via --settings on
a PLAIN INTERACTIVE TUI spawn (no -p, no --bare), claude-sonnet-4-6, --effort low:

  banner: "Sonnet 4.6 with low effort · Claude Max"   ← subscription pool, verified

  7 fires, mid-turn, spread across generation:
    index=0 final=false  '## Mutex\n\n'
    index=1 final=false  'A **mutual exclusion lock** prevents concurrent access to a shar…'
    index=4 final=false  'let counter = 0;\n\nasync function increment() {\n  const release =…'
    index=6 final=true   '```'

  concat(deltas) === T (transcript-authoritative)  ->  TRUE  (579 == 579 bytes)
  T.startsWith(S) at EVERY step                    ->  TRUE  (prefix-stable)
  '## ' / '**' / '```javascript' present in deltas ->  raw markdown SOURCE, not rendered

This satisfies every invariant the previous version declared unobtainable: byte-faithful,
incremental, prefix-stable, no -p, subscription pool. Granularity is block-level (~5-7
chunks/answer), not token-level — which is all an SSE delta.content needs.

Backlog #2 REOPENS and should be built. Implementer caveat recorded: the hook's source
sets forceSyncExecution -> claude BLOCKS on it, so the hook must write and exit
immediately (FIFO/socket), never work inline. Only text blocks fire it (thinking excluded).
ALIGNMENT: consumes claude's OWN hook surface as emitted — forwarding, not inventing
(Class B / ADR 0007; no cli.js citation applies).

Everything still true is kept, and the dead ends are kept as dead ends (they document what
NOT to build): the pane is a rendered view whose source markers are irrecoverable
(capture-pane -e emits IDENTICAL SGR 1 for an H2 and a bold span — a provably non-unique
inverse); the transcript is event-granular; --debug-file carries timing but no payload;
--output-format stream-json requires -p (the metered pool). Also kept: the ~4s (n=1
same-turn) overhead correction, backlog #4's null result with its mechanistic single-user
reason, and the honest value framing — streaming moves the FIRST byte, not the last, so
the complete-answer consumer that motivated this work gains nothing from it.

The wrong conclusion and its refutation are both preserved in the doc. "We checked, it's
impossible" is the most expensive claim to get wrong: it closes a door nobody re-opens.

Docs-only. No code change, no version bump.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VqgWJcjxrjjL9L9SkpZyXR
2026-07-13 16:16:12 +10:00

237 lines
14 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# TUI-mode latency: measured floor, and the four things worth fixing
**Date**: 2026-07-13
**Status**: findings + backlog. **Superseded in part** — see the dated update boxes below.
Item #1 shipped ([#156](https://github.com/dtzp555-max/ocp/pull/156)); item #2 is **dead**
([`streaming-spike.md`](streaming-spike.md)); item #4 measured, **no effect**; item #3 stands.
**Measured on**: Mac mini / macOS 26.5.2 / Claude Code **v2.1.207** / Sonnet 5 / Claude Max subscription / **real-home mode** (no `CLAUDE_CODE_OAUTH_TOKEN`, no `OCP_TUI_HOME` in the service env)
**Evidence**: [`measurements.jsonl`](measurements.jsonl) — **n=15** (3 configs × 5) · banner captures [`billing-banner.txt`](billing-banner.txt) · harness [`floor.sh`](floor.sh)
## Why this exists
An external consumer (the 知音 AI project) benchmarked OCP's prompt path and measured
**TTFT p50 ≈ 30–32 s**, and excluded OCP as a backend on that basis. That number is real,
but it is *not* the model being slow — this document decomposes where the 30 seconds
actually go, and what OCP can do about it.
**The harness deliberately does not go through OCP.** It spawns `tmux` + `claude` directly
(session prefix `zhiyin-floor-`, never `ocp-tui-*`) and polls `tmux capture-pane` for
incremental render, so it measures the **true first-token time** of the underlying
subscription path — the floor OCP could reach if it were perfect.
---
## Measurements
All rows in [`measurements.jsonl`](measurements.jsonl); every number below is recomputable from it.
| Config | n | boot→input-ready (median) | **TTFT (median)** | TTFT range | full answer (median) |
|---|---|---|---|---|---|
| baseline (inherits global `effortLevel: xhigh`) | 5 | 1.07 s | **10.35 s** | 8.32 – 17.19 s | 11.32 s |
| **`--effort low`** | 5 | 1.03 s | **6.17 s** | **5.87 – 6.44 s** | 9.98 s |
| `--bare` | 5 | 0.44 s | **no answer at all** (5/5 `ttft_ms: -1`) | — | — |
> **Not from this harness**: the direct Anthropic API reference figure (TTFT 0.84–1.64 s, n=2)
> comes from the 知音 AI project's own smoke test, not from `measurements.jsonl`. It is quoted
> only to size the gap; do not look for it in the evidence file.
### Where the 30 seconds go
```
~1.0 s spawn → claude's input bar is ready ← NOT the bottleneck
~6-10 s true TTFT (first token rendered in the pane)
~20 s ████ waiting for the whole turn to finish ████ ← this is the 30s
```
`runTuiTurn` blocks on the native transcript until a terminal event (`lib/tui/session.mjs`
"Block on the native transcript … until terminal"; `readTuiTranscript` in
`lib/tui/transcript.mjs`; ADR 0007 step 4) — i.e. it waits for the **entire turn** to complete
before returning anything. There is no streaming path. The ~20 s delta between this harness's
real TTFT and OCP's reported 30–32 s is exactly that.
> **⚠️ 2026-07-13 correction — this decomposition attributes the ~20 s to the wrong thing.** It was
> inferred from the external 30–32 s report, never measured *through* OCP. It has since been measured
> through a real OCP instance (TUI mode, `claude-sonnet-4-6`, the same ~1850-token prompt, n=5):
> **median 11.30 s** before [#156](https://github.com/dtzp555-max/ocp/pull/156), **9.55 s** after.
> Same-turn decomposition (baseline row `i=5`): **11.563 s** wall through OCP vs `turn_duration:
> 7.319 s` of CLI-internal time on that same turn → **OCP's own overhead ≈ 4.2 s** (n=1), **not
> ~20 s**. The rest of any larger number is the model *generating a long answer*,
> which the blocking wait does not cause and streaming would not shorten — it would only move the
> first byte earlier. The 30–32 s figure therefore reflects a much longer output (and/or the
> then-inherited `xhigh` effort), not 20 s of OCP dead time. See
> [`streaming-spike.md`](streaming-spike.md) § "What streaming would have bought".
---
## ⚠️ Blocking constraint: `--bare` silently drops you off the subscription pool
Captured live ([`billing-banner.txt`](billing-banner.txt)) — the startup banner is the **only**
reliable indicator:
```
[] | Sonnet 5 with xhigh effort · Claude Max
[--effort low] | Sonnet 5 with low effort · Claude Max
[--bare] | Sonnet 5 with xhigh effort · API Usage Billing ← ❌
```
`--bare` ("skip hooks, LSP, plugin…") **also skips the subscription-credential resolution
path**. It really does cut boot to 0.43–0.45 s — but you are no longer on the subscription,
which defeats the entire purpose of TUI mode (ADR 0007 exists solely to reach the
subscription pool).
**The failure is silent.** All 5 `--bare` samples reached input-ready (boot 0.43–0.45 s), were
sent the prompt, and then produced **no answer at all** — 60 s timeout, no error, no crash, the
pane simply never rendered a token (the API-billing account had no credit balance). Nothing in
the transcript or the exit status reveals this.
**Anyone changing spawn flags must diff the banner line before and after.**
---
## Backlog — four items, ranked by value ÷ effort
### 1. Pass `--effort` explicitly on spawn — **do this first**
`buildTuiCmd` (`lib/tui/session.mjs`) does not pass `--effort` — `grep -rn -- "--effort\|effortLevel" lib/ server.mjs`
returns zero hits. What the pane's `claude` ends up using therefore depends on **which HOME mode
`resolveTuiHome()` picked**:
| mode | HOME | effort the pane gets |
|---|---|---|
| **real-home** (legacy default — *current* service config: no `CLAUDE_CODE_OAUTH_TOKEN`, no `OCP_TUI_HOME`) | `~` | **inherits the operator's `~/.claude/settings.json` → `effortLevel: xhigh` on this host** |
| env-token scratch (`CLAUDE_CODE_OAUTH_TOKEN` set — the direction #146/#150 pushed) | `~/.ocp-tui/home` | that settings.json contains only `permissions.additionalDirectories`; `prepareTuiHome()` never writes `effortLevel` → **claude's built-in default** |
**Scope note**: TUI mode is currently *off* on this host (`CLAUDE_TUI_MODE=false`; `/health` →
`"tui": {"enabled": false}`), so live traffic takes the `-p` path today. The statement below is
about what happens **when TUI mode is enabled**.
On the current HOME config, **every TUI request would run extended thinking** — pure waste
for the typical "generate this JSON" request, and it makes latency depend on an unrelated global
setting the operator may have changed for their own interactive use. And the mode split means
the effort level silently changes if the operator ever switches to env-token mode.
**Passing `--effort` explicitly fixes both problems at once.**
- **Effect (real-home, measured)**: TTFT p50 **10.35 s → 6.17 s (−40 %)**, and the spread
collapses from 8.32–17.19 s to **5.87–6.44 s**. For a proxy, the variance reduction matters
more than the median.
- **Cost**: one flag. Suggested: a new `OCP_TUI_EFFORT` env var (default `low`), documented in
README § "Environment Variables" per `release_kit.new_feature_doc_expectations`.
- **Risk**: none — banner confirms it stays on `Claude Max` (see `billing-banner.txt`).
- ⚠️ Do **not** reach for `--bare` to shave boot: see above.
### 2. Real streaming instead of blocking on turn-terminal — **ACHIEVABLE → [`streaming-spike.md`](streaming-spike.md)**
> **2026-07-13 update — the prereq spike was run. The answer is YES, but not from either source this
> item guessed at.** (a) The transcript grows at *event* granularity (the whole answer lands in one
> line, ~0.3 s before terminal) — dead. (b) The pane is a **rendered** view whose `capture-pane` text
> no longer contains the answer's source bytes (`## `, `**`, code fences are gone) — dead, and worse
> than "lossy": it is *not the model's text*. **But there is a third source neither this backlog nor
> the first spike considered: `claude` fires a `MessageDisplay` hook carrying incremental,
> byte-faithful `delta`s of the raw reply.** Verified live on a plain interactive TUI spawn (no `-p`),
> banner `· Claude Max`: 7 fires spread across generation, `concat(deltas) === T` **byte-exactly**
> (579 == 579), `T.startsWith(S)` true at every step, `## ` / `**` / ```` ```javascript ```` all
> present in the deltas. Granularity is block-level (~5–7 chunks/answer), not token-level — plenty for
> SSE. **Build it.**
>
> ⚠️ Two corrections to this item as written: the **"~20 s" is wrong** (inferred from an external
> report, never measured through OCP — the same-turn decomposition puts OCP's own overhead at **~4 s**,
> n=1), and **streaming moves the first byte, not the last** — so a consumer needing the *complete*
> answer (the JSON-card case that motivated this) gains **nothing** from it. Build it for
> progressively-rendering consumers, not as a throughput win.
>
> Full evidence + implementer caveats (the hook is `forceSyncExecution` — claude BLOCKS on it):
> **[`streaming-spike.md`](streaming-spike.md)**. Original framing preserved below.
Today `runTuiTurn` blocks on the transcript until the turn is *finished*. The pane is already
rendering tokens incrementally the whole time — this harness proves you can observe first token
at ~6 s by polling `tmux capture-pane`.
- **Effect**: turns a 30 s wall into a ~6 s TTFT with progressive output; enables SSE streaming
on the OCP endpoint instead of a single blob at the end.
- **Cost**: real work. Pane capture is ANSI/redraw-based and lossy for exact text (wrapping,
scrollback, spinner lines). Two candidate sources: (a) incremental reads of the transcript
JSONL, (b) `capture-pane` diffing with a stable start marker. (a) is much cleaner **if it
holds**.
- **Prereq spike (do this before designing anything)**: does the transcript JSONL grow *during*
a turn, or only at the end? If only at the end, (a) is dead and you are stuck with (b).
### 3. Warm pane pool — ~1 s
Every request spawns a fresh tmux session + `claude` (`randomUUID()` + `new-session`, then
`kill-session` in `finally`; `grep -rn "pool\|warm\|reuse" lib/tui/*.mjs` → zero hits). Boot to
input-ready is ~1.0 s, paid on every request. A pool of pre-booted panes (single-use, replaced in
the background) amortizes it to zero for any workload below the pool refill rate.
- **Effect**: −1.0 s.
- **Cost**: moderate; interacts with the session reaper and the per-port prefix scoping added in
#148 — pooled panes must not look like zombies to the sweep.
- Lower priority than #1 and #2: it is the smallest slice.
### 4. Trim the prefill — ~~probably not worth it~~ **MEASURED: no detectable benefit. Do not adopt.**
> **2026-07-13 update.** `--exclude-dynamic-system-prompt-sections` was measured with the same
> harness (`floor.sh`, n=5, Sonnet 5, on top of `--effort low`): **TTFT median 6.39 s**
> (5.87–10.54 s) vs **6.17 s** (5.87–6.44 s) for `--effort low` alone — i.e. **0.22 s worse, inside
> the noise band**, with one worse outlier; dropping that outlier does not change the verdict. n=5
> cannot prove "zero", only "no benefit detectable above noise" — but there is also a **mechanistic**
> reason not to expect one: `--help` says the flag *"Improves cross-user prompt-cache **reuse**"*, and
> **OCP is single-user** — there is no cross-user cache to share, so the flag has nothing to buy here.
> The banner stayed on `· Claude Max` (no billing-pool drop), but there is no win to bank. The ~6 s
> floor stands as stated below. Raw rows: [`prefill-spike-measurements.jsonl`](prefill-spike-measurements.jsonl).
After #1–#3, the floor is **~6 s**, and it does not go lower. `claude` always injects the full
Claude Code system prompt + tool definitions (thousands to tens of thousands of prefill tokens)
regardless of what you ask it. `--exclude-dynamic-system-prompt-sections` exists and may shave
some of it — **unmeasured**; worth one spike, but do not expect to reach the direct API's
~1 s.
**Consequence to accept, and to state in the README**: even fully optimized, TUI mode has a
**~6 s TTFT floor**, so it cannot serve real-time / interactive-latency consumers. It remains
appropriate for batch, background, and cost-insensitive-latency use. The 知音 AI project
excluded it on this basis (their prompt-latency budget is 2–4 s) *independently* of the ToS
question already documented in the README.
---
## Reproduction
```bash
# harness never touches OCP's :3456 service or ocp-tui-* sessions, and never kill-server
bash docs/plans/2026-07-13-tui-latency/floor.sh 5 # baseline
TAG=effort-low EXTRA_ARGS="--effort low" bash .../floor.sh 5 # −40 %
TAG=bare EXTRA_ARGS="--bare" bash .../floor.sh 5 # the trap
# billing-pool check for ANY spawn-flag change — the banner is the only source of truth
tmux new-session -d -s probe -x 200 -y 50 -c "$HOME" \
"claude --model claude-sonnet-5 --session-id $(uuidgen) <your-flags-here>"
sleep 6; tmux capture-pane -p -t probe | grep -E "Claude Max|API Usage Billing"
tmux kill-session -t probe
```
## Interaction with OCP while the harness runs
- **Kill direction is safe both ways**: `reapStaleTuiSessions()` only `kill-session`s names
matching `ocp-tui-<port>-`, which `zhiyin-floor-*` never matches; and the harness only
`kill-session`s its own single session — it contains **no `kill-server`**.
- **One benign interaction** (only when TUI mode is enabled — the reap tick is itself gated on
`TUI_MODE`): OCP's periodic `kill-server` (zombie reaping) is gated on
`othersRemain` — *any* foreign-prefixed tmux session suppresses it. So while the harness is
running, that sweep is skipped. This is the coexistence guard working as designed; it resumes
on the next tick.
## Harness caveats (stated so the numbers are not over-trusted)
- **n=5 per config**, single host, single model (Sonnet 5), single prompt size (~1850 tokens).
Enough to separate 6 s from 10 s from 30 s; **not** enough for a p95.
- TTFT is "marker visible in `capture-pane`", which includes tmux render latency (small, but
nonzero) — it is an upper bound on the true first-token time.
- **The harness's readiness marker is not OCP's.** `floor.sh` waits for `│ >|❯|Try "`; OCP's
`tuiInputReady()` matches `/\? for shortcuts/`. These are different events, so the ~1.0 s
boot figure is **not** directly comparable to OCP's `BOOT_MS` gate (default cap 4000 ms). It
does not affect the conclusions (1 s ≪ 6 s TTFT), but it is not apples-to-apples.
- The first version of this harness reported TTFT **0.08 s** — a false positive: the prompt
literally contained the marker string it was grepping for, so the match fired the instant the
prompt was pasted. Fixed by describing the marker instead of spelling it. **The script exited 0
and "successfully" produced 5 samples both times** — exit status proves nothing here.
Reference in New Issue View Git Blame Copy Permalink