mirror of
https://github.com/dtzp555-max/ocp.git
synced 2026-07-21 21:15:09 +00:00
6854075c01
* docs(plans): TUI-mode latency floor — measured decomposition + backlog An external consumer measured OCP's prompt path at TTFT p50 30-32s and excluded OCP on that basis. This documents where those 30 seconds actually go, with a reproducible harness (n=15) that bypasses OCP and measures the underlying subscription path's true first-token time. Findings: - boot -> input-ready is only ~1.0s; it is NOT the bottleneck - true TTFT is 6-10s; the remaining ~20s is runTuiTurn polling the transcript until turn_duration (ADR 0007 step 4) — i.e. waiting for the WHOLE turn. There is no streaming. - buildTuiCmd never passes --effort, so the spawned claude inherits the operator's global effortLevel (xhigh on this host) — every request runs extended thinking. Passing --effort low: TTFT p50 9.70s -> 6.17s (-36%), spread 7.85-13.07s -> 5.87-6.44s. Stays on Claude Max. - ⚠️ --bare SILENTLY drops off the subscription pool (banner flips 'Claude Max' -> 'API Usage Billing'). It does cut boot to ~0.5s, but defeats the entire purpose of ADR 0007. Failure is silent: all 5 --bare samples produced no answer at all (no error, no crash, just never a token). Anyone optimizing boot MUST diff the banner line. - Floor after all fixes is ~6s (claude always injects the full CC system prompt + tool definitions). TUI mode therefore cannot serve real-time consumers — a constraint worth stating in the README. Backlog ranked by value/effort: (1) OCP_TUI_EFFORT env var, default low; (2) real streaming instead of turn_duration polling (~20s, the big one); (3) warm pane pool (~1s); (4) prefill trim (probably not worth it). Docs-only; no version bump (matches repo convention — bump lands in the chore(release) commit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dx5Ncq6wWBrF27vJKHZ9Hr * docs(plans): address review — restore --bare evidence, qualify effort claim, add banner captures Reviewer (fresh-context, Iron Rule 10) returned REQUEST_CHANGES. All four technical conclusions survived independent verification (source-read + live repro); the defects were in the evidence file, and they were real: - H-1: measurements.jsonl claimed n=15 but held 10 rows, and the --bare group — the basis of this PR's headline warning — had ZERO rows. The author had stripped them as 'invalid samples' (ttft_ms:-1) when they were in fact the evidence. Regenerated: n=15, three groups × 5, all with tag/extra_args. --bare reproduces exactly (5/5 no answer, boot 0.43-0.45s). - M-1: the effort-inheritance claim was written unconditionally, but it depends on resolveTuiHome()'s mode. Real-home (current service config) inherits the operator's effortLevel: xhigh; env-token scratch home (~/.ocp-tui/home) has no effortLevel in its settings.json and prepareTuiHome() never writes one, so the pane gets claude's built-in default. Now documented as a table — and the mode split makes passing --effort explicitly MORE valuable, not less. - M-2: baseline rows were produced by a pre-parameterized script and lacked tag/extra_args. Re-run with the committed script. Recomputed effect: -40% (was -36%). - L-1: documented that the harness suppresses OCP's periodic kill-server tick via the othersRemain coexistence guard (by design, resumes next tick). - L-2: documented that floor.sh's readiness marker differs from OCP's tuiInputReady(), so the ~1.0s boot figure is not apples-to-apples with BOOT_MS. - Direct-API reference figure now explicitly labeled as external (not in this dataset). - New: billing-banner.txt captures all three configs live, including confirmation that --effort low stays on Claude Max (reviewer noted this was asserted but unevidenced). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dx5Ncq6wWBrF27vJKHZ9Hr * docs(plans): scope the effort claim to TUI mode (currently off), drop nonexistent bin/ Re-review (APPROVE_WITH_MINOR) caught two accuracy defects: - MIN-1: 'every OCP request runs extended thinking' over-extrapolated. TUI mode is currently OFF on this host (CLAUDE_TUI_MODE=false; /health tui.enabled=false), so live traffic takes the -p path. The claim is about what happens WHEN TUI mode is enabled — now scoped, and the same qualifier applied to the kill-server interaction note (that reap tick is itself gated on TUI_MODE). - NIT-2: the quoted grep included bin/, which does not exist in the repo (exit 2). Dropped; the zero-hit result over lib/ + server.mjs is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dx5Ncq6wWBrF27vJKHZ9Hr --------- Co-authored-by: dtzp555 <dtzp555@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
16 lines
2.1 KiB
JSON
16 lines
2.1 KiB
JSON
{"i": 1, "tag": "effort-low", "model": "claude-sonnet-5", "extra_args": "--effort low", "prompt_chars": 7451, "boot_ms": 1077, "ttft_ms": 6172, "complete_ms": 9929}
|
|
{"i": 2, "tag": "effort-low", "model": "claude-sonnet-5", "extra_args": "--effort low", "prompt_chars": 7451, "boot_ms": 1026, "ttft_ms": 6160, "complete_ms": 9996}
|
|
{"i": 3, "tag": "effort-low", "model": "claude-sonnet-5", "extra_args": "--effort low", "prompt_chars": 7451, "boot_ms": 1010, "ttft_ms": 6437, "complete_ms": 9977}
|
|
{"i": 4, "tag": "effort-low", "model": "claude-sonnet-5", "extra_args": "--effort low", "prompt_chars": 7451, "boot_ms": 1033, "ttft_ms": 5872, "complete_ms": 9944}
|
|
{"i": 5, "tag": "effort-low", "model": "claude-sonnet-5", "extra_args": "--effort low", "prompt_chars": 7451, "boot_ms": 1154, "ttft_ms": 6387, "complete_ms": 9993}
|
|
{"i":1,"tag":"baseline","model":"claude-sonnet-5","extra_args":"","prompt_chars":7451,"boot_ms":1300,"ttft_ms":8321,"complete_ms":9939}
|
|
{"i":2,"tag":"baseline","model":"claude-sonnet-5","extra_args":"","prompt_chars":7451,"boot_ms":1070,"ttft_ms":10347,"complete_ms":11320}
|
|
{"i":3,"tag":"baseline","model":"claude-sonnet-5","extra_args":"","prompt_chars":7451,"boot_ms":911,"ttft_ms":13061,"complete_ms":15163}
|
|
{"i":4,"tag":"baseline","model":"claude-sonnet-5","extra_args":"","prompt_chars":7451,"boot_ms":1441,"ttft_ms":9981,"complete_ms":11066}
|
|
{"i":5,"tag":"baseline","model":"claude-sonnet-5","extra_args":"","prompt_chars":7451,"boot_ms":1036,"ttft_ms":17189,"complete_ms":17985}
|
|
{"i":1,"tag":"bare","model":"claude-sonnet-5","extra_args":"--bare","prompt_chars":7451,"boot_ms":429,"ttft_ms":-1,"complete_ms":-1}
|
|
{"i":2,"tag":"bare","model":"claude-sonnet-5","extra_args":"--bare","prompt_chars":7451,"boot_ms":437,"ttft_ms":-1,"complete_ms":-1}
|
|
{"i":3,"tag":"bare","model":"claude-sonnet-5","extra_args":"--bare","prompt_chars":7451,"boot_ms":444,"ttft_ms":-1,"complete_ms":-1}
|
|
{"i":4,"tag":"bare","model":"claude-sonnet-5","extra_args":"--bare","prompt_chars":7451,"boot_ms":446,"ttft_ms":-1,"complete_ms":-1}
|
|
{"i":5,"tag":"bare","model":"claude-sonnet-5","extra_args":"--bare","prompt_chars":7451,"boot_ms":441,"ttft_ms":-1,"complete_ms":-1}
|