mirror of https://github.com/dtzp555-max/ocp.git synced 2026-07-21 21:15:09 +00:00

Files

T

e7ce9899f3 docs(plans): TUI streaming IS achievable (MessageDisplay hook) — prereq spike + honest latency constraints (#157 )

* docs(plans): TUI streaming is not achievable — prereq spike result + honest README constraints

Backlog #2 of docs/plans/2026-07-13-tui-latency demanded a prereq spike before any
streaming design: does the transcript JSONL grow during a turn, or only at the end?
The spike was run. All three candidate sources are dead:

  (a) transcript JSONL — grows at EVENT granularity; the assistant's text event is
      written as ONE complete line, ~0.3s before the terminal turn_duration event
      (observed: turn_duration 7319ms; text event at t+7.0s, terminal at t+7.3s).
  (b) tmux capture-pane — the pane is a RENDERED view, not the text. Same turn,
      transcript T = '## Semaphore\n\nA **semaphore** is a synchronization…'
      pane        = '⏺ Semaphore' / '  A semaphore is a synchronization…'
      '## ', '**' and ```-fences are absent from the pane entirely (rendered to ANSI,
      then stripped by capture-pane -p). T.startsWith(paneText) is FALSE both raw and
      indent-stripped — not on redraw, but on essentially every markdown answer.
      capture-pane -e recovers styling, never source spelling: no unique inverse.
  (c) --debug-file — byte-exact ('last_assistant_message':'## Title\n\n**alpha…'),
      but only inside end-of-turn Stop-hook payloads; zero content_block_delta /
      text_delta events; ~2.7MB per turn.

--output-format stream-json, the only interface emitting token deltas, requires -p —
the metered-billing path TUI mode exists to avoid (cc_entrypoint=sdk-cli). The
constraint is structural. OCP's TUI SSE is, and remains, replay-only.

Also corrects this plan's own "~20s waiting for the whole turn" decomposition, which
was inferred from an external 30-32s report and never measured through OCP. Measured
through a real OCP instance (TUI, claude-sonnet-4-6, n=5): median 11.30s before #156,
9.55s after, vs a native turn_duration of ~7.3s → OCP's own overhead is ~2-4s, not
~20s. The remainder is generation time, which streaming would not shorten (it moves
the first byte, not the last) — so a consumer needing the COMPLETE answer, which is
the JSON-card case that motivated this work, would have gained nothing from streaming.

Backlog #4 measured while here: --exclude-dynamic-system-prompt-sections gives ZERO
marginal benefit (TTFT median 6.39s vs 6.17s for --effort low alone, n=5, one worse
outlier). Do not adopt. Banner stayed on Claude Max.

README: documents the ~6s TTFT floor plainly (TUI mode cannot serve interactive-latency
consumers) and states that no-token-streaming is structural rather than a missing feature.

No code change. No version bump (docs-only). Not endpoint-touching: no server.mjs diff,
so no cli.js citation applies.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VqgWJcjxrjjL9L9SkpZyXR

* docs: fold in adversarial review — correct the debug-log reasoning + the overhead number

Independent adversarial reviewer (tasked with REFUTING this doc) confirmed the central
claim — no byte-faithful incremental source exists on the TUI path — but found four
factual defects in the prose. A negative claim that will be cited for years has to be
right in its reasoning, not just its conclusion.

1. --debug-file: the "written at end-of-turn" reasoning was WRONG. The default log level
   is `debug`, which suppresses every `verbose` site; the original probe therefore ran
   with the stream logging OFF. At CLAUDE_CODE_DEBUG_LOG_LEVEL=verbose there ARE 16
   mid-turn `[shoji-engine] yield stream_event/-` lines spread over ~3.9s of generation.
   The conclusion survives because those lines carry TIMING ONLY, no text payload
   (content_block_delta / text_delta / content_block_start / message_start = 0 at any
   verbosity or category filter). Reasoning rewritten: "logs when tokens arrive, never
   what they are" — as written before, the doc was falsifiable in 30 seconds.

2. The 7.319s `turn_duration` is NOT a "native" (non-OCP) baseline: it comes from an
   OCP-driven turn (cwd .ocp-tui/work, same 7451-char prompt, same 204-char answer as
   pr1 baseline row i=5, elapsed 11563ms). Reframed as what it actually is — a SAME-TURN
   decomposition, 11.563s wall - 7.319s CLI-internal = ~4.2s OCP overhead (n=1), which is
   a cleaner comparison than the doc originally claimed.

3. Dropped the "~2-4s" range: its low end mixed an effort-HIGH turn_duration with the
   effort-LOW wall-clock median, which understates overhead (a low-effort turn generates
   faster, so its own turn_duration would be lower). No turn_duration sample exists for
   the effort-low config. Now stated as ~4s (n=1, baseline config), with both caveats.

4. Softened "ZERO marginal benefit" (backlog #4) to "no benefit detectable at n=5" — n=5
   cannot prove zero — and added the mechanistic reason the reviewer supplied, which is
   far stronger than the empirical null: `--help` says the flag improves cross-user
   prompt-cache REUSE, and OCP is single-user, so there is no cross-user cache to share.

Also folded in the reviewer's independent sweep, which closes the search space rather
than sampling it: the hook registry was enumerated from the shipped binary (no per-chunk
/ streaming hook exists among the 21 events); `capture-pane -e` was tested and shown to
be a provably non-unique inverse (an H2 and a bold span emit IDENTICAL SGR 1); and
sessions/<pid>.json, history.jsonl, CLAUDE_CODE_INCLUDE_PARTIAL_MESSAGES (undocumented),
sessionMirror, --sdk-url and --input-format stream-json were each checked and each dies
(contentless, or gated behind --output-format stream-json -> --print -> the metered
sdk-cli pool). Prompt-mutation (asking the model for plain text) is named and rejected
on ALIGNMENT grounds so it is not re-litigated later.

Docs-only. No code change, no version bump.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VqgWJcjxrjjL9L9SkpZyXR

* docs: REVERSE the streaming verdict — MessageDisplay hook makes it achievable

The previous commit on this branch concluded TUI streaming was impossible. That was
WRONG, and this corrects it before it could be merged.

The adversarial reviewer commissioned to refute the claim found, on a second pass while
verifying the fold-in, that its OWN first-pass hook enumeration had been truncated by a
400-char grep cap: it reported 21 hook events; the shipped 2.1.207 bundle has 30.
Event #30 is MessageDisplay.

Independently reproduced before acting on it (30 events confirmed via `strings` on the
binary; payload shape `hook_event_name:"MessageDisplay",turn_id,message_id,index,final,
delta`), then live-tested with a MessageDisplay command hook registered via --settings on
a PLAIN INTERACTIVE TUI spawn (no -p, no --bare), claude-sonnet-4-6, --effort low:

  banner: "Sonnet 4.6 with low effort · Claude Max"   ← subscription pool, verified

  7 fires, mid-turn, spread across generation:
    index=0 final=false  '## Mutex\n\n'
    index=1 final=false  'A **mutual exclusion lock** prevents concurrent access to a shar…'
    index=4 final=false  'let counter = 0;\n\nasync function increment() {\n  const release =…'
    index=6 final=true   '```'

  concat(deltas) === T (transcript-authoritative)  ->  TRUE  (579 == 579 bytes)
  T.startsWith(S) at EVERY step                    ->  TRUE  (prefix-stable)
  '## ' / '**' / '```javascript' present in deltas ->  raw markdown SOURCE, not rendered

This satisfies every invariant the previous version declared unobtainable: byte-faithful,
incremental, prefix-stable, no -p, subscription pool. Granularity is block-level (~5-7
chunks/answer), not token-level — which is all an SSE delta.content needs.

Backlog #2 REOPENS and should be built. Implementer caveat recorded: the hook's source
sets forceSyncExecution -> claude BLOCKS on it, so the hook must write and exit
immediately (FIFO/socket), never work inline. Only text blocks fire it (thinking excluded).
ALIGNMENT: consumes claude's OWN hook surface as emitted — forwarding, not inventing
(Class B / ADR 0007; no cli.js citation applies).

Everything still true is kept, and the dead ends are kept as dead ends (they document what
NOT to build): the pane is a rendered view whose source markers are irrecoverable
(capture-pane -e emits IDENTICAL SGR 1 for an H2 and a bold span — a provably non-unique
inverse); the transcript is event-granular; --debug-file carries timing but no payload;
--output-format stream-json requires -p (the metered pool). Also kept: the ~4s (n=1
same-turn) overhead correction, backlog #4's null result with its mechanistic single-user
reason, and the honest value framing — streaming moves the FIRST byte, not the last, so
the complete-answer consumer that motivated this work gains nothing from it.

The wrong conclusion and its refutation are both preserved in the doc. "We checked, it's
impossible" is the most expensive claim to get wrong: it closes a door nobody re-opens.

Docs-only. No code change, no version bump.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VqgWJcjxrjjL9L9SkpZyXR

* docs: fold in re-review — the two caveats that would have bitten the implementer

Re-review of the reversed doc came back APPROVE_WITH_MINOR. The reversal itself was
verified complete (worktree-wide grep: no surviving impossibility claim) and NOT
over-claimed in the other direction (the reviewer recomputed every headline number from
the committed messagedisplay-deltas.jsonl and re-ran the invariant on two further turns:
4 independent turns total, 5/6/4/18 fires, 609/696/239/1973 bytes, concat(deltas) === T
TRUE in all four). But two additive caveats were missing, and both are load-bearing for
the streaming PR now in flight:

1. CONCURRENCY DEMUX (severe, and live TODAY — not a warm-pool future problem).
   OCP_TUI_MAX_CONCURRENT defaults to 2, so two `claude` processes already run
   concurrently. One MessageDisplay hook writing to one shared sink would INTERLEAVE
   deltas from two different turns into a single stream — request A's client receiving
   request B's text. A single-request test never surfaces it. The payload carries
   session_id, so the sink must be keyed by it (which also keeps the design warm-pool
   compatible: a pre-booted pane's session-id is fixed at boot, so one static hook script
   serves every pane). Documented, with the required ≥2-concurrent-request test.

2. THINKING-EXCLUSION IS NOT STRESS-TESTED (severe if wrong). The exclusion was inferred
   from a code snippet that turns out to be the final:true call site, not the incremental
   one. Four live turns showed no thinking in any delta — but every transcript's thinking
   block was EMPTY (thinking:"", 0 chars), so it was never actually stressed. If thinking
   deltas do fire on Opus/xhigh, concat(deltas) !== T AND OCP streams the model's private
   reasoning to the caller; the concat === T assertion detects that but cannot un-send an
   SSE delta. Flagged as a must-verify-before-shipping item.

Also: the "5-7 chunks per answer" figure is size-dependent (18 fires on a ~2 KB answer) —
rescoped to "once per rendered block, scales with answer length" in both the doc and the
README, so no implementer hard-codes a chunk-count assumption.

Both caveats were relayed to the streaming implementation immediately rather than waiting
for this merge.

Docs-only. No code change, no version bump.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VqgWJcjxrjjL9L9SkpZyXR

---------

Co-authored-by: dtzp555 <dtzp555@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

2026-07-13 16:30:55 +10:00

14 KiB

Raw Permalink Blame History

TUI-mode latency: measured floor, and the four things worth fixing

Date: 2026-07-13 Status: findings + backlog. Superseded in part — see the dated update boxes below. Item #1 shipped (#156); item #2 is dead (streaming-spike.md); item #4 measured, no effect; item #3 stands. Measured on: Mac mini / macOS 26.5.2 / Claude Code v2.1.207 / Sonnet 5 / Claude Max subscription / real-home mode (no CLAUDE_CODE_OAUTH_TOKEN, no OCP_TUI_HOME in the service env) Evidence: measurements.jsonl — n=15 (3 configs × 5) · banner captures billing-banner.txt · harness floor.sh

Why this exists

An external consumer (the 知音 AI project) benchmarked OCP's prompt path and measured TTFT p50 ≈ 30–32 s, and excluded OCP as a backend on that basis. That number is real, but it is not the model being slow — this document decomposes where the 30 seconds actually go, and what OCP can do about it.

The harness deliberately does not go through OCP. It spawns tmux + claude directly (session prefix zhiyin-floor-, never ocp-tui-*) and polls tmux capture-pane for incremental render, so it measures the true first-token time of the underlying subscription path — the floor OCP could reach if it were perfect.

Measurements

All rows in measurements.jsonl; every number below is recomputable from it.

Config	n	boot→input-ready (median)	TTFT (median)	TTFT range	full answer (median)
baseline (inherits global `effortLevel: xhigh`)	5	1.07 s	10.35 s	8.32 – 17.19 s	11.32 s
`--effort low`	5	1.03 s	6.17 s	5.87 – 6.44 s	9.98 s
`--bare`	5	0.44 s	no answer at all (5/5 `ttft_ms: -1`)	—	—

Not from this harness: the direct Anthropic API reference figure (TTFT 0.84–1.64 s, n=2) comes from the 知音 AI project's own smoke test, not from measurements.jsonl. It is quoted only to size the gap; do not look for it in the evidence file.

Where the 30 seconds go

 ~1.0 s   spawn → claude's input bar is ready          ← NOT the bottleneck
 ~6-10 s  true TTFT (first token rendered in the pane)
 ~20 s    ████ waiting for the whole turn to finish ████  ← this is the 30s

runTuiTurn blocks on the native transcript until a terminal event (lib/tui/session.mjs "Block on the native transcript … until terminal"; readTuiTranscript in lib/tui/transcript.mjs; ADR 0007 step 4) — i.e. it waits for the entire turn to complete before returning anything. There is no streaming path. The ~20 s delta between this harness's real TTFT and OCP's reported 30–32 s is exactly that.

⚠️ 2026-07-13 correction — this decomposition attributes the ~20 s to the wrong thing. It was inferred from the external 30–32 s report, never measured through OCP. It has since been measured through a real OCP instance (TUI mode, claude-sonnet-4-6, the same ~1850-token prompt, n=5): median 11.30 s before #156, 9.55 s after. Same-turn decomposition (baseline row i=5): 11.563 s wall through OCP vs turn_duration: 7.319 s of CLI-internal time on that same turn → OCP's own overhead ≈ 4.2 s (n=1), not ~20 s. The rest of any larger number is the model generating a long answer, which the blocking wait does not cause and streaming would not shorten — it would only move the first byte earlier. The 30–32 s figure therefore reflects a much longer output (and/or the then-inherited xhigh effort), not 20 s of OCP dead time. See streaming-spike.md § "What streaming would have bought".

⚠️ Blocking constraint: `--bare` silently drops you off the subscription pool

Captured live (billing-banner.txt) — the startup banner is the only reliable indicator:

[]                     | Sonnet 5 with xhigh effort · Claude Max
[--effort low]         | Sonnet 5 with low effort  · Claude Max
[--bare]               | Sonnet 5 with xhigh effort · API Usage Billing     ← ❌

--bare ("skip hooks, LSP, plugin…") also skips the subscription-credential resolution path. It really does cut boot to 0.43–0.45 s — but you are no longer on the subscription, which defeats the entire purpose of TUI mode (ADR 0007 exists solely to reach the subscription pool).

The failure is silent. All 5 --bare samples reached input-ready (boot 0.43–0.45 s), were sent the prompt, and then produced no answer at all — 60 s timeout, no error, no crash, the pane simply never rendered a token (the API-billing account had no credit balance). Nothing in the transcript or the exit status reveals this.

Anyone changing spawn flags must diff the banner line before and after.

Backlog — four items, ranked by value ÷ effort

1. Pass `--effort` explicitly on spawn — do this first

buildTuiCmd (lib/tui/session.mjs) does not pass --effort — grep -rn -- "--effort\|effortLevel" lib/ server.mjs returns zero hits. What the pane's claude ends up using therefore depends on which HOME mode resolveTuiHome() picked:

mode	HOME	effort the pane gets
real-home (legacy default — current service config: no `CLAUDE_CODE_OAUTH_TOKEN`, no `OCP_TUI_HOME`)	`~`	inherits the operator's `~/.claude/settings.json` → `effortLevel: xhigh` on this host
env-token scratch (`CLAUDE_CODE_OAUTH_TOKEN` set — the direction #146/#150 pushed)	`~/.ocp-tui/home`	that settings.json contains only `permissions.additionalDirectories`; `prepareTuiHome()` never writes `effortLevel` → claude's built-in default

Scope note: TUI mode is currently off on this host (CLAUDE_TUI_MODE=false; /health → "tui": {"enabled": false}), so live traffic takes the -p path today. The statement below is about what happens when TUI mode is enabled.

On the current HOME config, every TUI request would run extended thinking — pure waste for the typical "generate this JSON" request, and it makes latency depend on an unrelated global setting the operator may have changed for their own interactive use. And the mode split means the effort level silently changes if the operator ever switches to env-token mode. Passing --effort explicitly fixes both problems at once.

Effect (real-home, measured): TTFT p50 10.35 s → 6.17 s (−40 %), and the spread collapses from 8.32–17.19 s to 5.87–6.44 s. For a proxy, the variance reduction matters more than the median.
Cost: one flag. Suggested: a new OCP_TUI_EFFORT env var (default low), documented in README § "Environment Variables" per release_kit.new_feature_doc_expectations.
Risk: none — banner confirms it stays on Claude Max (see billing-banner.txt).
⚠️ Do not reach for --bare to shave boot: see above.

2. Real streaming instead of blocking on turn-terminal — ACHIEVABLE → `streaming-spike.md`

2026-07-13 update — the prereq spike was run. The answer is YES, but not from either source this item guessed at. (a) The transcript grows at event granularity (the whole answer lands in one line, ~0.3 s before terminal) — dead. (b) The pane is a rendered view whose capture-pane text no longer contains the answer's source bytes (## , **, code fences are gone) — dead, and worse than "lossy": it is not the model's text. But there is a third source neither this backlog nor the first spike considered: claude fires a MessageDisplay hook carrying incremental, byte-faithful deltas of the raw reply. Verified live on a plain interactive TUI spawn (no -p), banner · Claude Max: 7 fires spread across generation, concat(deltas) === T byte-exactly (579 == 579), T.startsWith(S) true at every step, ## / ** / ```javascript all present in the deltas. Granularity is block-level (~5–7 chunks/answer), not token-level — plenty for SSE. Build it.

⚠️ Two corrections to this item as written: the "~20 s" is wrong (inferred from an external report, never measured through OCP — the same-turn decomposition puts OCP's own overhead at ~4 s, n=1), and streaming moves the first byte, not the last — so a consumer needing the complete answer (the JSON-card case that motivated this) gains nothing from it. Build it for progressively-rendering consumers, not as a throughput win.

Full evidence + implementer caveats (the hook is forceSyncExecution — claude BLOCKS on it): streaming-spike.md. Original framing preserved below.

Today runTuiTurn blocks on the transcript until the turn is finished. The pane is already rendering tokens incrementally the whole time — this harness proves you can observe first token at ~6 s by polling tmux capture-pane.

Effect: turns a 30 s wall into a ~6 s TTFT with progressive output; enables SSE streaming on the OCP endpoint instead of a single blob at the end.
Cost: real work. Pane capture is ANSI/redraw-based and lossy for exact text (wrapping, scrollback, spinner lines). Two candidate sources: (a) incremental reads of the transcript JSONL, (b) capture-pane diffing with a stable start marker. (a) is much cleaner if it holds.
Prereq spike (do this before designing anything): does the transcript JSONL grow during a turn, or only at the end? If only at the end, (a) is dead and you are stuck with (b).

3. Warm pane pool — ~1 s

Every request spawns a fresh tmux session + claude (randomUUID() + new-session, then kill-session in finally; grep -rn "pool\|warm\|reuse" lib/tui/*.mjs → zero hits). Boot to input-ready is ~1.0 s, paid on every request. A pool of pre-booted panes (single-use, replaced in the background) amortizes it to zero for any workload below the pool refill rate.

Effect: −1.0 s.
Cost: moderate; interacts with the session reaper and the per-port prefix scoping added in #148 — pooled panes must not look like zombies to the sweep.
Lower priority than #1 and #2: it is the smallest slice.

4. Trim the prefill — probably not worth it MEASURED: no detectable benefit. Do not adopt.

2026-07-13 update. --exclude-dynamic-system-prompt-sections was measured with the same harness (floor.sh, n=5, Sonnet 5, on top of --effort low): TTFT median 6.39 s (5.87–10.54 s) vs 6.17 s (5.87–6.44 s) for --effort low alone — i.e. 0.22 s worse, inside the noise band, with one worse outlier; dropping that outlier does not change the verdict. n=5 cannot prove "zero", only "no benefit detectable above noise" — but there is also a mechanistic reason not to expect one: --help says the flag "Improves cross-user prompt-cache reuse", and OCP is single-user — there is no cross-user cache to share, so the flag has nothing to buy here. The banner stayed on · Claude Max (no billing-pool drop), but there is no win to bank. The ~6 s floor stands as stated below. Raw rows: prefill-spike-measurements.jsonl.

After #1–#3, the floor is ~6 s, and it does not go lower. claude always injects the full Claude Code system prompt + tool definitions (thousands to tens of thousands of prefill tokens) regardless of what you ask it. --exclude-dynamic-system-prompt-sections exists and may shave some of it — unmeasured; worth one spike, but do not expect to reach the direct API's ~1 s.

Consequence to accept, and to state in the README: even fully optimized, TUI mode has a ~6 s TTFT floor, so it cannot serve real-time / interactive-latency consumers. It remains appropriate for batch, background, and cost-insensitive-latency use. The 知音 AI project excluded it on this basis (their prompt-latency budget is 2–4 s) independently of the ToS question already documented in the README.

Reproduction

# harness never touches OCP's :3456 service or ocp-tui-* sessions, and never kill-server
bash docs/plans/2026-07-13-tui-latency/floor.sh 5                                  # baseline
TAG=effort-low EXTRA_ARGS="--effort low" bash .../floor.sh 5                       # −40 %
TAG=bare       EXTRA_ARGS="--bare"       bash .../floor.sh 5                       # the trap

# billing-pool check for ANY spawn-flag change — the banner is the only source of truth
tmux new-session -d -s probe -x 200 -y 50 -c "$HOME" \
  "claude --model claude-sonnet-5 --session-id $(uuidgen) <your-flags-here>"
sleep 6; tmux capture-pane -p -t probe | grep -E "Claude Max|API Usage Billing"
tmux kill-session -t probe

Interaction with OCP while the harness runs

Kill direction is safe both ways: reapStaleTuiSessions() only kill-sessions names matching ocp-tui-<port>-, which zhiyin-floor-* never matches; and the harness only kill-sessions its own single session — it contains no kill-server.
One benign interaction (only when TUI mode is enabled — the reap tick is itself gated on TUI_MODE): OCP's periodic kill-server (zombie reaping) is gated on othersRemain — any foreign-prefixed tmux session suppresses it. So while the harness is running, that sweep is skipped. This is the coexistence guard working as designed; it resumes on the next tick.

Harness caveats (stated so the numbers are not over-trusted)

n=5 per config, single host, single model (Sonnet 5), single prompt size (~1850 tokens). Enough to separate 6 s from 10 s from 30 s; not enough for a p95.
TTFT is "marker visible in capture-pane", which includes tmux render latency (small, but nonzero) — it is an upper bound on the true first-token time.
The harness's readiness marker is not OCP's. floor.sh waits for │ >|❯|Try "; OCP's tuiInputReady() matches /\? for shortcuts/. These are different events, so the ~1.0 s boot figure is not directly comparable to OCP's BOOT_MS gate (default cap 4000 ms). It does not affect the conclusions (1 s ≪ 6 s TTFT), but it is not apples-to-apples.
The first version of this harness reported TTFT 0.08 s — a false positive: the prompt literally contained the marker string it was grepping for, so the match fired the instant the prompt was pasted. Fixed by describing the marker instead of spelling it. The script exited 0 and "successfully" produced 5 samples both times — exit status proves nothing here.

14 KiB Raw Permalink Blame History Unescape Escape