docs: fold in adversarial review — correct the debug-log reasoning + the overhead number

Independent adversarial reviewer (tasked with REFUTING this doc) confirmed the central
claim — no byte-faithful incremental source exists on the TUI path — but found four
factual defects in the prose. A negative claim that will be cited for years has to be
right in its reasoning, not just its conclusion.

1. --debug-file: the "written at end-of-turn" reasoning was WRONG. The default log level
   is `debug`, which suppresses every `verbose` site; the original probe therefore ran
   with the stream logging OFF. At CLAUDE_CODE_DEBUG_LOG_LEVEL=verbose there ARE 16
   mid-turn `[shoji-engine] yield stream_event/-` lines spread over ~3.9s of generation.
   The conclusion survives because those lines carry TIMING ONLY, no text payload
   (content_block_delta / text_delta / content_block_start / message_start = 0 at any
   verbosity or category filter). Reasoning rewritten: "logs when tokens arrive, never
   what they are" — as written before, the doc was falsifiable in 30 seconds.

2. The 7.319s `turn_duration` is NOT a "native" (non-OCP) baseline: it comes from an
   OCP-driven turn (cwd .ocp-tui/work, same 7451-char prompt, same 204-char answer as
   pr1 baseline row i=5, elapsed 11563ms). Reframed as what it actually is — a SAME-TURN
   decomposition, 11.563s wall - 7.319s CLI-internal = ~4.2s OCP overhead (n=1), which is
   a cleaner comparison than the doc originally claimed.

3. Dropped the "~2-4s" range: its low end mixed an effort-HIGH turn_duration with the
   effort-LOW wall-clock median, which understates overhead (a low-effort turn generates
   faster, so its own turn_duration would be lower). No turn_duration sample exists for
   the effort-low config. Now stated as ~4s (n=1, baseline config), with both caveats.

4. Softened "ZERO marginal benefit" (backlog #4) to "no benefit detectable at n=5" — n=5
   cannot prove zero — and added the mechanistic reason the reviewer supplied, which is
   far stronger than the empirical null: `--help` says the flag improves cross-user
   prompt-cache REUSE, and OCP is single-user, so there is no cross-user cache to share.

Also folded in the reviewer's independent sweep, which closes the search space rather
than sampling it: the hook registry was enumerated from the shipped binary (no per-chunk
/ streaming hook exists among the 21 events); `capture-pane -e` was tested and shown to
be a provably non-unique inverse (an H2 and a bold span emit IDENTICAL SGR 1); and
sessions/<pid>.json, history.jsonl, CLAUDE_CODE_INCLUDE_PARTIAL_MESSAGES (undocumented),
sessionMirror, --sdk-url and --input-format stream-json were each checked and each dies
(contentless, or gated behind --output-format stream-json -> --print -> the metered
sdk-cli pool). Prompt-mutation (asking the model for plain text) is named and rejected
on ALIGNMENT grounds so it is not re-litigated later.

Docs-only. No code change, no version bump.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VqgWJcjxrjjL9L9SkpZyXR
This commit is contained in:
2026-07-13 16:04:12 +10:00
co-authored by Claude Fable 5
parent 6848f9751c
commit fc63b8a49a
3 changed files with 107 additions and 35 deletions
+15 -8
View File
@@ -52,9 +52,10 @@ real TTFT and OCP's reported 3032 s is exactly that.
> **⚠️ 2026-07-13 correction — this decomposition attributes the ~20 s to the wrong thing.** It was
> inferred from the external 3032 s report, never measured *through* OCP. It has since been measured
> through a real OCP instance (TUI mode, `claude-sonnet-4-6`, the same ~1850-token prompt, n=5):
> **median 11.30 s** before [#156](https://github.com/dtzp555-max/ocp/pull/156), **9.55 s** after
> against a native `turn_duration` of ~7.3 s for a comparable turn. So **OCP's own overhead above the
> CLI is ~24 s, not ~20 s.** The rest of any larger number is the model *generating a long answer*,
> **median 11.30 s** before [#156](https://github.com/dtzp555-max/ocp/pull/156), **9.55 s** after.
> Same-turn decomposition (baseline row `i=5`): **11.563 s** wall through OCP vs `turn_duration:
> 7.319 s` of CLI-internal time on that same turn → **OCP's own overhead ≈ 4.2 s** (n=1), **not
> ~20 s**. The rest of any larger number is the model *generating a long answer*,
> which the blocking wait does not cause and streaming would not shorten — it would only move the
> first byte earlier. The 3032 s figure therefore reflects a much longer output (and/or the
> then-inherited `xhigh` effort), not 20 s of OCP dead time. See
@@ -124,7 +125,9 @@ the effort level silently changes if the operator ever switches to env-token mod
> grows at *event* granularity (the whole answer lands in one line, ~0.3 s before terminal), and the
> pane is a **rendered** view whose `capture-pane` text no longer contains the answer's source bytes
> (`## `, `**`, code fences are gone) — so no pane-derived stream can be reconciled with the
> authoritative text. A third source (`--debug-file`) is byte-exact but also end-of-turn.
> authoritative text. A third source (`--debug-file`) does emit mid-turn stream events at
> `CLAUDE_CODE_DEBUG_LOG_LEVEL=verbose` — but they carry **timing only, no text payload** (0
> `text_delta` at any verbosity); its only byte-exact text is the end-of-turn `Stop` hook payload.
> `--output-format stream-json`, the only interface that emits token deltas, requires `-p` — the
> metered-billing path that TUI mode exists to avoid. **True token streaming is not achievable on the
> TUI path**; OCP's TUI SSE is replay-only. Note also that streaming would never have shortened a
@@ -159,13 +162,17 @@ the background) amortizes it to zero for any workload below the pool refill rate
#148 — pooled panes must not look like zombies to the sweep.
- Lower priority than #1 and #2: it is the smallest slice.
### 4. Trim the prefill — ~~probably not worth it~~ **MEASURED: no effect. Do not adopt.**
### 4. Trim the prefill — ~~probably not worth it~~ **MEASURED: no detectable benefit. Do not adopt.**
> **2026-07-13 update.** `--exclude-dynamic-system-prompt-sections` was measured with the same
> harness (`floor.sh`, n=5, Sonnet 5, on top of `--effort low`): **TTFT median 6.39 s**
> (5.8710.54 s) vs **6.17 s** (5.876.44 s) for `--effort low` alone — i.e. **zero marginal
> benefit**, within noise and with one worse outlier. The banner stayed on `· Claude Max`
> (no billing-pool drop), but there is no win to bank. The ~6 s floor stands as stated below.
> (5.8710.54 s) vs **6.17 s** (5.876.44 s) for `--effort low` alone — i.e. **0.22 s worse, inside
> the noise band**, with one worse outlier; dropping that outlier does not change the verdict. n=5
> cannot prove "zero", only "no benefit detectable above noise" — but there is also a **mechanistic**
> reason not to expect one: `--help` says the flag *"Improves cross-user prompt-cache **reuse**"*, and
> **OCP is single-user** — there is no cross-user cache to share, so the flag has nothing to buy here.
> The banner stayed on `· Claude Max` (no billing-pool drop), but there is no win to bank. The ~6 s
> floor stands as stated below. Raw rows: [`prefill-spike-measurements.jsonl`](prefill-spike-measurements.jsonl).
After #1#3, the floor is **~6 s**, and it does not go lower. `claude` always injects the full