mirror of
https://github.com/dtzp555-max/ocp.git
synced 2026-07-19 09:44:07 +00:00
fix(tui): honest error/truncation handling + version-robust entrypoint + short-prompt paste (PR-A) (#137)
TUI-mode (CLAUDE_TUI_MODE=true) is an OCP-owned subscription-pool bridge for the
2026-06-15 Anthropic billing split. This PR fixes four honesty/robustness defects
confirmed by a prior audit and live-reproduced on PI231 (2026-06-10).
C-1 (P1) — upstream AUTH-FAILURE banner returned/cached as a real answer.
The interactive claude CLI renders in-session errors as ordinary assistant text.
The R-1 case C-1 exists to catch is expired/invalid credentials, where EVERY turn
comes back as the same one-line auth-failure banner, e.g. the two live PI231
banners (2026-06-10):
"Please run /login · API Error: 401 Invalid authentication credentials" (69 chars)
"Failed to authenticate. API Error: 401 Invalid authentication credentials" (73 chars)
callClaudeTui returned that banner verbatim → OCP cached it, shared it via
singleflight, and recorded a model SUCCESS. New detectTuiUpstreamError()
(lib/tui/transcript.mjs) flags such a turn; callClaudeTui throws tui_upstream_error
+ logEvent("error", …) BEFORE recordModelSuccess/cache write-back, so the error
never enters the cache; the client gets a 5xx.
C-1 NARROWING (false-positive probe, 2026-06-10). An earlier generalised default
rule — ^<short auth-failure prefix>?API Error: <3-digit> <detail>$ — was TOO BROAD:
the unbounded ".*" detail tail let any short prefix + "API Error: NNN" + an
arbitrarily long sentence match, so it KILLED legitimate long answers that merely
DISCUSS an API error (e.g. "API Error: 500 happened because the server was
overloaded. To fix this, retry with exponential backoff …"). A false-positive costs
the user a missing answer AND a double-burn retry — strictly worse than the rare
false-negative (caching one transient error for the 5-min TTL). C-1 is therefore
reframed from "detect any API error" to "detect a claude-CLI AUTHENTICATION-FAILURE
banner", and is CONSERVATIVE: when unsure it PASSES. The narrowed default detector
flags a turn only when ALL of the following hold over the WHOLE trimmed text
(a conjunction; any one failing => PASS):
1. SHORT whole-message — length ≤ 100 (live banners are 69/73; cap gives headroom
while rejecting multi-sentence prose; a 226-char auth answer is dropped on
length alone).
2. Contains "API Error: 4\d{2}" — auth failures are 4xx (401/403); transient 5xx
and bare "HTTP 401 means unauthorized." (no API-Error core) are excluded.
3. Contains an auth keyword — authenticat | /login | credential (case-insensitive);
rejects "To debug a 401 … API Error: 401 Unauthorized …" (authoriz-, not
authenticat-).
4. Contains NO backtick/quote char (` ' ") — a real banner is plain text; quoted
text signals an answer QUOTING the error, e.g. "You'll see `API Error: 401` …
run /login to fix it." (short + 4xx + /login, excluded only by this signal).
Full required matrix (2 KILL + 7 PASS) encoded as tests; npm test green.
CLAUDE_TUI_ERROR_PATTERNS still overrides (a non-empty value REPLACES the default
with operator regexes; empty/whitespace DISABLES detection). New fixture
lib/tui/fixtures/error-401-failauth.jsonl + positive/negative tests retained.
C-2 (P1) — wallclock-truncated partial text returned as silent success.
readTuiTranscript returned {text, entrypoint} identically on the terminal-marker
path and the cap-with-partial path, so a cut-off turn was cached + returned as
finish_reason:stop. It now returns truncated:false on terminal-marker and
truncated:true on the cap-with-partial path (additive field; no-text cap path
still throws). callClaudeTui throws tui_wallclock_truncated on truncated.
C-3 (P1) — verifyEntrypoint only read the turn_duration line.
Some claude builds don't emit turn_duration (Mac mini: absent; PI231/2.1.104:
present), while the entrypoint field is on ordinary lines on BOTH. Reading only
turn_duration made the server.mjs tui_entrypoint_mismatch assertion get got:null
every turn on non-emitting builds. verifyEntrypoint now PREFERS the turn_duration
entrypoint and FALLS BACK to the entrypoint field on any line.
C-4 (P2) — short prompts 100% failed paste-landing.
tuiPromptLanded required needle.length >= 3, so a 1–2 char first line ("hi","ok")
never matched and 5s-failed with tui_paste_not_landed every time (live-repro:
"hi"). Threshold lowered 3 → 2; the input box starts empty (placeholder excluded
by the affirmative-signal design) so a 2-char needle present in the pane is the
prompt. Kept >=2 (not >=1) to avoid collisions with claude's chrome glyphs.
Tests: extended test-features.mjs with unit coverage for each fix + three fixtures
(lib/tui/fixtures/error-401.jsonl, error-401-failauth.jsonl, no-turn-duration.jsonl).
The C-1 block now encodes the full narrowed auth-banner matrix (2 must-kill + 7
must-pass + supporting regression guards incl. a length-cap-load-bearing test).
npm test: 224 passed, 0 failed. Default path (CLAUDE_TUI_MODE unset →
upstreamCall=callClaude) is byte-identical: the only server.mjs change is one import
+ the callClaudeTui body, and callClaudeTui is unreachable when TUI_MODE is off.
ALIGNMENT: Class B (OCP-owned compatibility surface). cli.js does not perform this
operation (TUI is an OCP-owned subscription-pool bridge); scope authorized by ADR
0007 (Class B). No Class A path touched (callClaude / handleUsage / OAuth unchanged);
no change to .github/workflows/alignment.yml or any blacklisted token.
Co-authored-by: dtzp555 <dtzp555@gmail.com>
Co-authored-by: Claude <claude-opus> <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
{"type":"user","entrypoint":"cli","cwd":"/tmp/tui-test","sessionId":"bbbb2222-3333-4444-5555-666677778888","version":"2.1.104","message":{"role":"user","content":[{"type":"text","text":"What is 2 + 2?"}]}}
|
||||
{"type":"assistant","entrypoint":"cli","cwd":"/tmp/tui-test","sessionId":"bbbb2222-3333-4444-5555-666677778888","version":"2.1.104","message":{"role":"assistant","model":"claude-haiku-4-5-20251001","stop_reason":"end_turn","content":[{"type":"text","text":"Failed to authenticate. API Error: 401 Invalid authentication credentials"}]}}
|
||||
@@ -0,0 +1,2 @@
|
||||
{"type":"user","entrypoint":"cli","cwd":"/tmp/tui-test","sessionId":"aaaa1111-2222-3333-4444-555566667777","version":"2.1.104","message":{"role":"user","content":[{"type":"text","text":"What is 2 + 2?"}]}}
|
||||
{"type":"assistant","entrypoint":"cli","cwd":"/tmp/tui-test","sessionId":"aaaa1111-2222-3333-4444-555566667777","version":"2.1.104","message":{"role":"assistant","model":"claude-haiku-4-5-20251001","stop_reason":"end_turn","content":[{"type":"text","text":"Please run /login · API Error: 401 Invalid authentication credentials"}]}}
|
||||
@@ -0,0 +1,2 @@
|
||||
{"type":"user","entrypoint":"cli","cwd":"/tmp/tui-test","sessionId":"bbbb1111-2222-3333-4444-555566667777","version":"2.1.104","message":{"role":"user","content":[{"type":"text","text":"Say PONG and nothing else."}]}}
|
||||
{"type":"assistant","entrypoint":"cli","cwd":"/tmp/tui-test","sessionId":"bbbb1111-2222-3333-4444-555566667777","version":"2.1.104","message":{"role":"assistant","model":"claude-haiku-4-5-20251001","stop_reason":"end_turn","content":[{"type":"text","text":"PONG"}]}}
|
||||
+10
-1
@@ -68,7 +68,16 @@ function tuiPromptLanded(pane, prompt) {
|
||||
if (flatPane.includes("[Pasted text")) return true;
|
||||
const firstLine = String(prompt).split("\n").map(s => s.trim()).find(Boolean) || "";
|
||||
const needle = firstLine.replace(/\s+/g, " ").slice(0, 24);
|
||||
return needle.length >= 3 && flatPane.includes(needle);
|
||||
// C-4/#133: threshold lowered 3 → 2. A prompt whose first non-blank line is 1–2
|
||||
// chars ("hi", "ok") previously NEVER matched (needle.length >= 3) and never
|
||||
// surfaced "[Pasted text", so EVERY short prompt 5s-failed with tui_paste_not_landed
|
||||
// (live-reproduced: "hi"). The input box starts EMPTY (the curly-quote placeholder
|
||||
// is excluded by the affirmative-signal design above), so a >=2-char needle present
|
||||
// in the pane is the pasted prompt, not placeholder noise — false-positive risk is
|
||||
// low. We keep >=2 rather than >=1 because a single visible char is more likely to
|
||||
// collide with incidental glyphs in claude's chrome (borders, the "❯" prompt mark);
|
||||
// 2 chars is the floor that lands real prompts while staying conservative.
|
||||
return needle.length >= 2 && flatPane.includes(needle);
|
||||
}
|
||||
|
||||
async function pollUntil(fn, { timeoutMs, intervalMs }) {
|
||||
|
||||
+161
-12
@@ -100,24 +100,169 @@ export function extractLatestAssistantText(events) {
|
||||
return text;
|
||||
}
|
||||
|
||||
// Returns the entrypoint string from the turn_duration line (e.g. "cli"),
|
||||
// Returns the entrypoint string (e.g. "cli") used for the billing-pool assertion,
|
||||
// or null if absent. Lets callers assert the subscription-classified path.
|
||||
// Fixture-confirmed: entrypoint field lives directly on the turn_duration line.
|
||||
//
|
||||
// Resolution order (C-3, issue #133):
|
||||
// 1. PREFER the turn_duration system line's `entrypoint` — the authoritative
|
||||
// end-of-turn classifier emitted by builds that produce turn_duration
|
||||
// (e.g. claude-2.1.104/2.1.157 on PI231).
|
||||
// 2. FALL BACK to the `entrypoint` field on ANY ordinary transcript line
|
||||
// (assistant / user / attachment / system) — present on BOTH emitting and
|
||||
// non-emitting builds. Some claude builds (e.g. certain Mac mini transcripts)
|
||||
// do NOT emit a turn_duration line at all; reading ONLY turn_duration made the
|
||||
// caller's tui_entrypoint_mismatch assertion (server.mjs) get got:null every
|
||||
// turn and go blind. The entrypoint value is identical across line types within
|
||||
// a single interactive session (fixture-confirmed: every line in
|
||||
// complete-haiku.jsonl carrying `entrypoint` reads "cli"), so the fallback
|
||||
// yields the same classifier. Last-writer-wins on the fallback.
|
||||
export function verifyEntrypoint(events) {
|
||||
let fallback = null;
|
||||
for (const ev of events) {
|
||||
if (ev && ev.type === "system" && ev.subtype === "turn_duration") {
|
||||
return ev.entrypoint != null ? ev.entrypoint : null;
|
||||
if (!ev || typeof ev !== "object") continue;
|
||||
if (ev.type === "system" && ev.subtype === "turn_duration" && ev.entrypoint != null) {
|
||||
return ev.entrypoint; // authoritative — short-circuit
|
||||
}
|
||||
if (ev.entrypoint != null) fallback = ev.entrypoint;
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
|
||||
// ── C-1: honest AUTH-FAILURE banner detection (issue #133) ───────────────
|
||||
// When the interactive `claude` CLI hits an in-session error it does NOT crash —
|
||||
// it renders the error as ordinary assistant text in the transcript. The specific
|
||||
// failure C-1 exists to catch is R-1: EXPIRED / INVALID credentials, where every
|
||||
// turn comes back as the same one-line auth-failure banner and OCP, none the wiser,
|
||||
// caches that banner (server.mjs setCachedResponse), shares it via singleflight, and
|
||||
// records a model SUCCESS — so a hard auth error is silently served (and cached for
|
||||
// the 5-min TTL) as a real answer. The two live-reproduced banners on PI231
|
||||
// (2026-06-10) are:
|
||||
// "Please run /login · API Error: 401 Invalid authentication credentials" (69 chars)
|
||||
// "Failed to authenticate. API Error: 401 Invalid authentication credentials" (73 chars)
|
||||
//
|
||||
// WHY THE SCOPE IS NARROW (conservatism — the load-bearing design choice).
|
||||
// An earlier generalised rule (^<short-prefix>?API Error:\s*\d{3}\b.*$) was TOO
|
||||
// BROAD: its unbounded `.*` tail let any short prefix + "API Error: NNN" + an
|
||||
// arbitrarily long sentence match, so it KILLED legitimate long answers that merely
|
||||
// DISCUSS an API error (e.g. "API Error: 500 happened because the server was
|
||||
// overloaded. To fix this, retry with exponential backoff …"). That is the worst
|
||||
// outcome: a false-positive costs the user a missing answer AND a double-burn retry,
|
||||
// whereas the rare false-negative (caching one transient error for the 5-min TTL) is
|
||||
// cheap and self-healing. So C-1 is reframed from "detect ANY API error" to "detect
|
||||
// a claude-CLI AUTHENTICATION-FAILURE banner", and when unsure it PASSES (does not
|
||||
// kill). Transient 5xx server errors are deliberately NOT detected — they are not the
|
||||
// R-1 case and the conservative choice is to let them through.
|
||||
//
|
||||
// THE SIGNAL — a turn is an auth-failure banner only if ALL of these hold over the
|
||||
// WHOLE trimmed assistant text (a conjunction; any one failing => PASS):
|
||||
// 1. SHORT whole-message. Real banners are one short line (the two live samples are
|
||||
// 69 and 73 chars). Cap = TUI_ERR_MAX_LEN (100) — headroom over 73 for a
|
||||
// slightly longer future banner, while still rejecting multi-sentence prose. A
|
||||
// long answer that happens to discuss auth (no code chars, e.g. 226 chars) is
|
||||
// rejected on length alone.
|
||||
// 2. Contains "API Error: 4\d{2}" — auth failures are 4xx (401/403). This rejects
|
||||
// transient 5xx ("API Error: 500/503 …") and bare "HTTP 401 means unauthorized."
|
||||
// (no "API Error:" core).
|
||||
// 3. Contains an auth KEYWORD — authenticat | /login | credential (case-insensitive).
|
||||
// This rejects answers that quote a 4xx but are not auth banners, e.g.
|
||||
// "To debug a 401: the server returns API Error: 401 Unauthorized …"
|
||||
// ("Unauthorized" is authoriz-, not authenticat-; no /login, no credential).
|
||||
// 4. Contains NO backtick or quote char (` ' "). A real CLI banner is plain text;
|
||||
// backticked/quoted text signals an answer that is QUOTING the error rather than
|
||||
// being the banner, e.g. "You'll see `API Error: 401` … run /login to fix it."
|
||||
// (75 chars — passes 1-3 but is excluded here). This is the conservative tie-
|
||||
// breaker for short instructional answers.
|
||||
//
|
||||
// Worked matrix (all required cases pass — see test-features.mjs C-1 block):
|
||||
// KILL: "Please run /login · API Error: 401 Invalid authentication credentials"
|
||||
// KILL: "Failed to authenticate. API Error: 401 Invalid authentication credentials"
|
||||
// PASS: "API Error: 500 happened because the server was overloaded. …" (not 4xx)
|
||||
// PASS: "Failed to parse the config. Here are the API Error: 401 details …" (too long + no auth-kw)
|
||||
// PASS: "To debug a 401: … API Error: 401 Unauthorized, then you refresh …" (no auth-kw)
|
||||
// PASS: "Here is the handler … It logs the string API Error: 503 …" (not 4xx)
|
||||
// PASS: "You'll see `API Error: 401` … run /login to fix it." (has backtick)
|
||||
// PASS: "HTTP 401 means unauthorized." (no API Error core)
|
||||
// PASS: "The capital of France is Paris." (nothing matches)
|
||||
//
|
||||
// OPERATOR OVERRIDE (unchanged): CLAUDE_TUI_ERROR_PATTERNS lets an operator REPLACE
|
||||
// the default auth-banner detector with their own newline- or `||`-separated JS regex
|
||||
// source strings (each auto-anchored ^…$ over the trimmed text, case-insensitive). A
|
||||
// non-empty override uses ONLY those regexes (the narrowed default is bypassed); an
|
||||
// empty / whitespace-only override DISABLES detection entirely (escape hatch).
|
||||
|
||||
// Whole-message length cap for the default auth-banner detector. Real banners are
|
||||
// 69/73 chars; 100 gives headroom while still rejecting multi-sentence prose.
|
||||
const TUI_ERR_MAX_LEN = 100;
|
||||
// 4xx "API Error:" core — auth failures are 4xx (401/403), never 5xx.
|
||||
const TUI_ERR_4XX = /API Error:\s*4\d{2}\b/i;
|
||||
// Auth keyword — the message must be about authentication, not just quote a 4xx.
|
||||
const TUI_ERR_AUTH_KW = /authenticat|\/login|credential/i;
|
||||
// Code/quote chars — their presence signals prose QUOTING an error, not the banner.
|
||||
const TUI_ERR_CODE_CHAR = /[`'"]/;
|
||||
|
||||
// Default detector: returns true iff `trimmed` IS a claude-CLI auth-failure banner
|
||||
// (all four signals above). Conservative — any signal failing => false (PASS).
|
||||
function isDefaultAuthFailureBanner(trimmed) {
|
||||
if (trimmed.length > TUI_ERR_MAX_LEN) return false; // 1. short whole-message
|
||||
if (!TUI_ERR_4XX.test(trimmed)) return false; // 2. 4xx API Error core
|
||||
if (!TUI_ERR_AUTH_KW.test(trimmed)) return false; // 3. auth keyword
|
||||
if (TUI_ERR_CODE_CHAR.test(trimmed)) return false; // 4. no code/quote chars
|
||||
return true;
|
||||
}
|
||||
|
||||
// Compile an OPERATOR-SUPPLIED pattern set (override path only). Each source is
|
||||
// anchored ^…$ over the trimmed text and matched case-insensitively (`s` so `.` spans
|
||||
// a multi-line banner). A pattern that fails to compile is skipped (never throws into
|
||||
// the request path).
|
||||
function compileTuiErrorPatterns(raw) {
|
||||
const sources = String(raw).split(/\r?\n|\|\|/).map((s) => s.trim()).filter(Boolean);
|
||||
const out = [];
|
||||
for (const src of sources) {
|
||||
try { out.push(new RegExp(`^(?:${src})$`, "is")); } catch { /* skip bad pattern */ }
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// Returns the matched banner text (the trimmed assistant text) if `text` IS a claude-
|
||||
// CLI auth-failure banner in its entirety, else null. `patternsRaw` defaults to
|
||||
// process.env.CLAUDE_TUI_ERROR_PATTERNS:
|
||||
// - undefined → narrowed default auth-banner detector (isDefaultAuthFailureBanner).
|
||||
// - non-empty → operator regex override REPLACES the default.
|
||||
// - empty/ws → detection disabled (escape hatch).
|
||||
export function detectTuiUpstreamError(text, patternsRaw = process.env.CLAUDE_TUI_ERROR_PATTERNS) {
|
||||
if (typeof text !== "string") return null;
|
||||
const trimmed = text.trim();
|
||||
if (!trimmed) return null;
|
||||
if (patternsRaw == null) {
|
||||
return isDefaultAuthFailureBanner(trimmed) ? trimmed : null;
|
||||
}
|
||||
// Operator override path: empty/whitespace disables; otherwise use only their regexes.
|
||||
const patterns = compileTuiErrorPatterns(patternsRaw);
|
||||
if (patterns.length === 0) return null;
|
||||
for (const re of patterns) {
|
||||
if (re.test(trimmed)) return trimmed;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Block until the session transcript is terminal (turn_duration) or
|
||||
// the wall-clock cap elapses, polling the file (no fs.watch — robust over NFS /
|
||||
// editors). Returns { text, entrypoint } where text is the latest assistant text
|
||||
// and entrypoint is the billing-pool classifier from the turn_duration line (e.g.
|
||||
// "cli"), or null if not yet present. On cap with text, returns the partial result;
|
||||
// on cap with no text at all, throws.
|
||||
// Block until the session transcript is terminal (turn_duration / final
|
||||
// stop_reason) or the wall-clock cap elapses, polling the file (no fs.watch —
|
||||
// robust over NFS / editors). Returns { text, entrypoint, truncated }:
|
||||
// - text: latest assistant text.
|
||||
// - entrypoint: billing-pool classifier (see verifyEntrypoint), or null.
|
||||
// - truncated: FALSE when a terminal marker was reached (the turn completed);
|
||||
// TRUE when the wall-clock cap was hit with partial text but NO
|
||||
// terminal marker (the turn is INCOMPLETE — what we have is a
|
||||
// cut-off prefix). (C-2, issue #133.)
|
||||
//
|
||||
// Why `truncated` matters: previously the terminal-marker path and the
|
||||
// cap-with-partial-text path BOTH returned `{text, entrypoint}` identically, so
|
||||
// callClaudeTui could not tell a complete answer from a truncated one and cached +
|
||||
// returned the partial as finish_reason:stop (silent success). The caller now
|
||||
// throws on `truncated` so a cut-off turn is neither cached nor counted as success.
|
||||
// The field is additive — existing call sites that ignore it keep working.
|
||||
//
|
||||
// On cap with NO text at all, still throws (unchanged) — there is nothing to return.
|
||||
//
|
||||
// No quiescence heuristic by design: a long Opus thinking turn stalls transcript
|
||||
// growth and a "file stable for N s" rule would false-abort it (spec §4.3).
|
||||
@@ -135,10 +280,14 @@ export async function readTuiTranscript({ transcriptPath: p, home, sessionId, wa
|
||||
lastText = extractLatestAssistantText(events) || lastText;
|
||||
const ep = verifyEntrypoint(events);
|
||||
if (ep != null) lastEntrypoint = ep;
|
||||
if (events.some(isTerminalLine)) return { text: lastText, entrypoint: lastEntrypoint };
|
||||
// Terminal marker reached → the turn is COMPLETE.
|
||||
if (events.some(isTerminalLine)) return { text: lastText, entrypoint: lastEntrypoint, truncated: false };
|
||||
}
|
||||
await sleep(pollMs);
|
||||
}
|
||||
if (lastText) return { text: lastText, entrypoint: lastEntrypoint };
|
||||
// Cap elapsed with no terminal marker. If we have partial text, flag it truncated
|
||||
// so the caller rejects it (don't cache / don't count as success). No text at all
|
||||
// → throw (nothing to return).
|
||||
if (lastText) return { text: lastText, entrypoint: lastEntrypoint, truncated: true };
|
||||
throw new Error("tui_transcript_timeout: no assistant text within wallclock cap");
|
||||
}
|
||||
|
||||
+25
-1
@@ -38,6 +38,7 @@ import { validateKey, recordUsage, getUsageByKey, getUsageTimeline, getRecentUsa
|
||||
import { DEFAULT_PORT } from "./lib/constants.mjs";
|
||||
import { isLoopbackBind } from "./lib/net.mjs";
|
||||
import { runTuiTurn, reapStaleTuiSessions } from "./lib/tui/session.mjs";
|
||||
import { detectTuiUpstreamError } from "./lib/tui/transcript.mjs";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const _pkg = JSON.parse(readFileSync(join(__dirname, "package.json"), "utf8"));
|
||||
@@ -926,7 +927,30 @@ function callClaudeTui(model, messages, _conversationId, _keyName) {
|
||||
cwd: TUI_CWD,
|
||||
wallclockMs: TUI_WALLCLOCK_MS,
|
||||
entrypointMode: TUI_ENTRYPOINT,
|
||||
}).then(({ text, entrypoint }) => {
|
||||
}).then(({ text, entrypoint, truncated }) => {
|
||||
// ── Honesty gates (issue #133) ─ run BEFORE recordModelSuccess / cache write-back.
|
||||
// A throw here propagates to the .catch below (recordModelError + reject), so the
|
||||
// result never reaches the downstream setCachedResponse / singleflight / SUCCESS path.
|
||||
|
||||
// C-2: the wall-clock cap hit with partial text and NO terminal marker — the turn
|
||||
// is INCOMPLETE. Returning the cut-off prefix would cache it and report it as
|
||||
// finish_reason:stop (a truncated answer served as a complete one). Reject instead.
|
||||
if (truncated) {
|
||||
logEvent("error", "tui_wallclock_truncated", { model: cliModel, chars: (text || "").length, wallclockMs: TUI_WALLCLOCK_MS });
|
||||
throw new Error("tui_wallclock_truncated: turn hit the wall-clock cap before completing; partial text dropped");
|
||||
}
|
||||
|
||||
// C-1: the interactive claude CLI renders in-session errors (expired/invalid
|
||||
// credentials, transient API failure) as ordinary assistant text. Returning that
|
||||
// banner would cache an error AS an answer and record a model SUCCESS. Detect a
|
||||
// known error banner (anchored whole-text match — see detectTuiUpstreamError) and
|
||||
// reject so it does NOT enter the cache and the client gets a 5xx.
|
||||
const banner = detectTuiUpstreamError(text);
|
||||
if (banner) {
|
||||
logEvent("error", "tui_upstream_error", { model: cliModel, banner: banner.slice(0, 200) });
|
||||
throw new Error("tui_upstream_error: claude CLI returned an in-session error banner instead of an answer");
|
||||
}
|
||||
|
||||
recordModelSuccess(cliModel, 0); // elapsed not measurable here; wallclock at reader level
|
||||
// Assert the subscription-pool classification. TUI exists to keep cc_entrypoint=cli
|
||||
// (subscription pool); a silent degrade to sdk-cli (metered Agent SDK pool) would still
|
||||
|
||||
+200
-4
@@ -1340,7 +1340,7 @@ test("streamStringAsSSE empty content: role + stop + [DONE] only", () => {
|
||||
});
|
||||
|
||||
// ── Suite: TUI transcript reader ────────────────────────────────────────
|
||||
import { encodeCwd, transcriptPath, findTranscriptPath, parseTranscriptLines, isTerminalLine, extractLatestAssistantText, verifyEntrypoint } from "./lib/tui/transcript.mjs";
|
||||
import { encodeCwd, transcriptPath, findTranscriptPath, parseTranscriptLines, isTerminalLine, extractLatestAssistantText, verifyEntrypoint, detectTuiUpstreamError } from "./lib/tui/transcript.mjs";
|
||||
import { readFileSync as tuiReadFileSync, mkdtempSync as tuiMkdtemp0, mkdirSync as tuiMkdir0, writeFileSync as tuiWrite0 } from "node:fs";
|
||||
import { tmpdir as tuiTmp0 } from "node:os";
|
||||
|
||||
@@ -1436,6 +1436,173 @@ test("real complete fixture: verifyEntrypoint returns 'cli'", () => {
|
||||
assert.equal(verifyEntrypoint(evs), "cli");
|
||||
});
|
||||
|
||||
// ── C-3 (#133): verifyEntrypoint is version-robust ───────────────────────
|
||||
// Some claude builds do NOT emit a turn_duration line; entrypoint lives on
|
||||
// ordinary lines on BOTH emitting and non-emitting builds. Reading ONLY
|
||||
// turn_duration made the server.mjs tui_entrypoint_mismatch assertion get null
|
||||
// every turn on non-emitting builds. verifyEntrypoint must fall back to ANY line.
|
||||
console.log("\nTUI transcript — verifyEntrypoint version-robustness (C-3, #133):");
|
||||
|
||||
test("verifyEntrypoint PREFERS the turn_duration line's entrypoint", () => {
|
||||
// turn_duration says "cli"; an earlier ordinary line says "sdk-cli" — the
|
||||
// authoritative turn_duration value must win, not last-writer-wins on the fallback.
|
||||
const evs = [
|
||||
{ type: "assistant", entrypoint: "sdk-cli", message: { content: [{ type: "text", text: "x" }] } },
|
||||
{ type: "system", subtype: "turn_duration", entrypoint: "cli" },
|
||||
];
|
||||
assert.equal(verifyEntrypoint(evs), "cli");
|
||||
});
|
||||
test("verifyEntrypoint falls back to entrypoint on an ordinary assistant line when no turn_duration", () => {
|
||||
const evs = [
|
||||
{ type: "user", entrypoint: "cli", message: { content: "hi" } },
|
||||
{ type: "assistant", entrypoint: "cli", message: { stop_reason: "end_turn", content: [{ type: "text", text: "ok" }] } },
|
||||
];
|
||||
assert.equal(verifyEntrypoint(evs), "cli");
|
||||
});
|
||||
test("verifyEntrypoint returns null when NO line carries an entrypoint", () => {
|
||||
const evs = [
|
||||
{ type: "assistant", message: { stop_reason: "end_turn", content: [{ type: "text", text: "ok" }] } },
|
||||
];
|
||||
assert.equal(verifyEntrypoint(evs), null);
|
||||
});
|
||||
test("real no-turn_duration fixture: verifyEntrypoint still resolves 'cli' (was null before C-3)", () => {
|
||||
const evs = parseTranscriptLines(tuiReadFileSync("./lib/tui/fixtures/no-turn-duration.jsonl", "utf8"));
|
||||
// Sanity: the fixture genuinely lacks a turn_duration line (so this exercises the fallback).
|
||||
assert.ok(!evs.some((e) => e && e.type === "system" && e.subtype === "turn_duration"), "fixture must NOT emit turn_duration");
|
||||
assert.equal(verifyEntrypoint(evs), "cli");
|
||||
});
|
||||
|
||||
// ── C-1 (#133): honest AUTH-FAILURE banner detection ─────────────────────
|
||||
// The interactive claude CLI renders in-session errors as ordinary assistant text.
|
||||
// C-1 catches the R-1 case: expired/invalid creds, where EVERY turn returns the same
|
||||
// one-line auth-failure banner and OCP would cache it as a real answer. The detector
|
||||
// is deliberately NARROW/conservative: a false-positive (killing a real long answer
|
||||
// that merely DISCUSSES an API error) costs the user a missing answer + a double-burn
|
||||
// retry, which is worse than the rare false-negative (caching one transient error for
|
||||
// the 5-min TTL). Signal = ALL of: SHORT whole-message (≤100; live samples 69/73) AND
|
||||
// "API Error: 4xx" AND an auth keyword (authenticat | /login | credential) AND NO
|
||||
// backtick/quote char. When unsure → PASS. The earlier generalised rule
|
||||
// (^<short-prefix>?API Error:\d{3}.*$) was TOO BROAD: its unbounded .* tail killed
|
||||
// legit long answers; this block encodes the full narrowed matrix.
|
||||
console.log("\nTUI transcript — auth-failure banner detection (C-1, #133):");
|
||||
|
||||
// ---- Required matrix: MUST detect (kill) ----
|
||||
test("C-1 KILL: live /login 401 auth banner", () => {
|
||||
const banner = "Please run /login · API Error: 401 Invalid authentication credentials";
|
||||
assert.equal(detectTuiUpstreamError(banner), banner);
|
||||
});
|
||||
test("C-1 KILL: live 'Failed to authenticate.' 401 banner variant", () => {
|
||||
// Second real PI231 banner: a different short auth-failure prefix before the same
|
||||
// "API Error: 4xx" core. Still short, still 4xx, still has 'authenticate'/'credentials'.
|
||||
const banner = "Failed to authenticate. API Error: 401 Invalid authentication credentials";
|
||||
assert.equal(detectTuiUpstreamError(banner), banner);
|
||||
});
|
||||
|
||||
// ---- Required matrix: MUST NOT kill (pass) ----
|
||||
test("C-1 PASS: long answer discussing a 500 (not 4xx, too long)", () => {
|
||||
// The exact false-positive the over-broad .* rule produced. 166 chars; 5xx.
|
||||
const legit = "API Error: 500 happened because the server was overloaded. To fix this, retry with exponential backoff and verify your rate limits before resending the request again.";
|
||||
assert.equal(detectTuiUpstreamError(legit), null);
|
||||
});
|
||||
test("C-1 PASS: long answer with 'API Error: 401 details' (too long, no auth keyword)", () => {
|
||||
// 142 chars; the literal word 'authenticate'/'credential'/'/login' never appears, and
|
||||
// it is far over the length cap — rejected on length AND keyword.
|
||||
const legit = "Failed to parse the config. Here are the API Error: 401 details you asked about: the token expired and must be refreshed before the next call.";
|
||||
assert.equal(detectTuiUpstreamError(legit), null);
|
||||
});
|
||||
test("C-1 PASS: 'To debug a 401 … API Error: 401 Unauthorized' (no auth keyword)", () => {
|
||||
// 91 chars (short!) and 4xx, but 'Unauthorized' is authoriz-, not authenticat-, and
|
||||
// there is no /login or credential — the auth-keyword signal rejects it.
|
||||
const legit = "To debug a 401: the server returns API Error: 401 Unauthorized, then you refresh the token.";
|
||||
assert.equal(detectTuiUpstreamError(legit), null);
|
||||
});
|
||||
test("C-1 PASS: handler answer logging 'API Error: 503' (not 4xx)", () => {
|
||||
const legit = "Here is the handler you asked for. It logs the string API Error: 503 on failure and retries.";
|
||||
assert.equal(detectTuiUpstreamError(legit), null);
|
||||
});
|
||||
test("C-1 PASS: short instructional answer quoting `API Error: 401` + /login (has backtick)", () => {
|
||||
// 75 chars: short, 4xx, has '/login' — passes signals 1-3. Rejected ONLY by the
|
||||
// backtick/quote constraint: it QUOTES the error in code formatting, it is not the banner.
|
||||
const legit = "You'll see `API Error: 401` when your token expires — run /login to fix it.";
|
||||
assert.equal(detectTuiUpstreamError(legit), null);
|
||||
});
|
||||
test("C-1 PASS: bare HTTP-status sentence (no 'API Error:' core)", () => {
|
||||
assert.equal(detectTuiUpstreamError("HTTP 401 means unauthorized."), null);
|
||||
});
|
||||
test("C-1 PASS: plain unrelated answer", () => {
|
||||
assert.equal(detectTuiUpstreamError("The capital of France is Paris."), null);
|
||||
});
|
||||
|
||||
// ---- Supporting / regression coverage ----
|
||||
test("C-1 PASS: transient 5xx banner is NOT detected (narrowed to 4xx auth only)", () => {
|
||||
// The old rule flagged any 3-digit code; the narrowed detector is 4xx-only by design
|
||||
// (5xx is transient/server-side, not the R-1 auth case). Accepted false-negative.
|
||||
assert.equal(detectTuiUpstreamError("API Error: 500 Internal Server Error"), null);
|
||||
});
|
||||
test("C-1 PASS: bare 4xx with no auth keyword is NOT detected", () => {
|
||||
// 'API Error: 403 Forbidden' alone — 4xx and short, but no authenticat/login/credential.
|
||||
assert.equal(detectTuiUpstreamError("API Error: 403 Forbidden"), null);
|
||||
});
|
||||
test("detectTuiUpstreamError trims surrounding whitespace before matching", () => {
|
||||
const out = detectTuiUpstreamError("\n\n Please run /login · API Error: 401 credential boom \n");
|
||||
assert.equal(out, "Please run /login · API Error: 401 credential boom");
|
||||
});
|
||||
test("detectTuiUpstreamError is case-insensitive on the banner keywords", () => {
|
||||
// lower-cased: /login + api error: 401 + 'credential' keyword, short, no code char.
|
||||
assert.ok(detectTuiUpstreamError("please run /login · api error: 401 bad credential") !== null);
|
||||
});
|
||||
test("detectTuiUpstreamError does NOT match prose that mentions an API error mid-paragraph (#133 regression guard)", () => {
|
||||
// A long, legit answer that merely discusses an API error — rejected on length alone.
|
||||
const para = "When integrating with the upstream service you may occasionally hit an API Error: 401 response if the bearer token has lapsed; the recommended remediation is to re-run the login flow and retry the request with a fresh credential, after which the 401 should clear.";
|
||||
assert.equal(detectTuiUpstreamError(para), null);
|
||||
});
|
||||
test("detectTuiUpstreamError does NOT match a long plain-text auth answer with NO code chars (length cap is load-bearing)", () => {
|
||||
// 226 chars, no backtick/quote, has 4xx + /login + credential + authenticate — passes
|
||||
// signals 2-4. ONLY the length cap rejects it. Guards against dropping the cap.
|
||||
const para = "If you call the endpoint without a bearer token the API Error: 401 response tells you the credential is missing; just authenticate again with /login and the request will succeed on the next attempt without any further changes.";
|
||||
assert.equal(detectTuiUpstreamError(para), null);
|
||||
});
|
||||
test("detectTuiUpstreamError returns null on empty / whitespace / non-string", () => {
|
||||
assert.equal(detectTuiUpstreamError(""), null);
|
||||
assert.equal(detectTuiUpstreamError(" \n "), null);
|
||||
assert.equal(detectTuiUpstreamError(null), null);
|
||||
assert.equal(detectTuiUpstreamError(undefined), null);
|
||||
assert.equal(detectTuiUpstreamError(42), null);
|
||||
});
|
||||
test("detectTuiUpstreamError respects CLAUDE_TUI_ERROR_PATTERNS override (custom banner)", () => {
|
||||
// Override with a single custom pattern; the default 401 banner no longer matches,
|
||||
// but the custom one does (anchored whole-text).
|
||||
assert.equal(detectTuiUpstreamError("Please run /login · API Error: 401 x", "Session expired, please re-auth"), null);
|
||||
assert.equal(detectTuiUpstreamError("Session expired, please re-auth", "Session expired, please re-auth"), "Session expired, please re-auth");
|
||||
});
|
||||
test("detectTuiUpstreamError with an empty override disables detection (escape hatch)", () => {
|
||||
assert.equal(detectTuiUpstreamError("API Error: 500 boom", ""), null);
|
||||
assert.equal(detectTuiUpstreamError("API Error: 500 boom", " "), null);
|
||||
});
|
||||
test("detectTuiUpstreamError override accepts '||'-separated patterns", () => {
|
||||
const raw = "First banner||Second banner";
|
||||
assert.equal(detectTuiUpstreamError("First banner", raw), "First banner");
|
||||
assert.equal(detectTuiUpstreamError("Second banner", raw), "Second banner");
|
||||
assert.equal(detectTuiUpstreamError("Third", raw), null);
|
||||
});
|
||||
test("real error fixture: latest assistant text IS the banner and detectTuiUpstreamError flags it", () => {
|
||||
const evs = parseTranscriptLines(tuiReadFileSync("./lib/tui/fixtures/error-401.jsonl", "utf8"));
|
||||
const text = extractLatestAssistantText(evs);
|
||||
assert.equal(text, "Please run /login · API Error: 401 Invalid authentication credentials");
|
||||
assert.ok(detectTuiUpstreamError(text) !== null, "error fixture's final turn must be flagged as an upstream error");
|
||||
});
|
||||
test("real error fixture (Failed-to-authenticate variant): final turn is flagged (#133 runtime gap)", () => {
|
||||
const evs = parseTranscriptLines(tuiReadFileSync("./lib/tui/fixtures/error-401-failauth.jsonl", "utf8"));
|
||||
const text = extractLatestAssistantText(evs);
|
||||
assert.equal(text, "Failed to authenticate. API Error: 401 Invalid authentication credentials");
|
||||
assert.ok(detectTuiUpstreamError(text) !== null, "Failed-to-authenticate banner must be flagged as an upstream error");
|
||||
});
|
||||
test("real complete fixture: final answer is NOT flagged as an upstream error", () => {
|
||||
const evs = parseTranscriptLines(tuiReadFileSync("./lib/tui/fixtures/complete-haiku.jsonl", "utf8"));
|
||||
const text = extractLatestAssistantText(evs);
|
||||
assert.equal(detectTuiUpstreamError(text), null);
|
||||
});
|
||||
|
||||
// ── TUI transcript — polling reader (async) ──────────────────────────────
|
||||
import { readTuiTranscript } from "./lib/tui/transcript.mjs";
|
||||
import { mkdtempSync as tuiMkdtemp, writeFileSync as tuiWriteFile } from "node:fs";
|
||||
@@ -1455,12 +1622,30 @@ await asyncTest("readTuiTranscript returns assistant text when terminal marker p
|
||||
assert.equal(out.entrypoint, "cli");
|
||||
});
|
||||
|
||||
await asyncTest("readTuiTranscript honours wall-clock cap and returns partial text", async () => {
|
||||
// C-2 (#133): the terminal-marker path must signal a COMPLETE turn.
|
||||
await asyncTest("readTuiTranscript signals truncated:false when a terminal marker is hit (complete turn)", async () => {
|
||||
const dir = tuiMkdtemp(`${tuiTmpdir()}/tui-`);
|
||||
const p = `${dir}/s.jsonl`;
|
||||
tuiWriteFile(p, [
|
||||
JSON.stringify({ type: "assistant", message: { content: [{ type: "text", text: "done" }] } }),
|
||||
JSON.stringify({ type: "system", subtype: "turn_duration", durationMs: 1200, entrypoint: "cli" }),
|
||||
].join("\n") + "\n");
|
||||
const out = await readTuiTranscript({ transcriptPath: p, wallclockMs: 2000, pollMs: 50 });
|
||||
assert.equal(out.truncated, false);
|
||||
});
|
||||
|
||||
// C-2 (#133): cap-with-partial-text must be DISTINGUISHABLE from a complete turn.
|
||||
// Previously both returned {text, entrypoint} identically and the partial was cached
|
||||
// + returned as finish_reason:stop. The cap path now returns truncated:true so the
|
||||
// caller (callClaudeTui) can throw instead of serving a cut-off answer.
|
||||
await asyncTest("readTuiTranscript honours wall-clock cap and flags partial text truncated:true", async () => {
|
||||
const dir = tuiMkdtemp(`${tuiTmpdir()}/tui-`);
|
||||
const p = `${dir}/s.jsonl`;
|
||||
// No terminal marker → reader will spin to the cap then return the partial.
|
||||
tuiWriteFile(p, JSON.stringify({ type: "assistant", message: { content: [{ type: "text", text: "partial" }] } }) + "\n");
|
||||
const out = await readTuiTranscript({ transcriptPath: p, wallclockMs: 300, pollMs: 50 });
|
||||
assert.equal(out.text, "partial");
|
||||
assert.equal(out.truncated, true);
|
||||
});
|
||||
|
||||
await asyncTest("readTuiTranscript against real fixture: entrypoint is 'cli'", async () => {
|
||||
@@ -1699,7 +1884,7 @@ function _tuiPromptLanded(pane, prompt) {
|
||||
if (flatPane.includes("[Pasted text")) return true;
|
||||
const firstLine = String(prompt).split("\n").map(s => s.trim()).find(Boolean) || "";
|
||||
const needle = firstLine.replace(/\s+/g, " ").slice(0, 24);
|
||||
return needle.length >= 3 && flatPane.includes(needle);
|
||||
return needle.length >= 2 && flatPane.includes(needle); // C-4 (#133): 3 → 2 (see lib/tui/session.mjs)
|
||||
}
|
||||
|
||||
// Real captured pane samples (empirically confirmed via live capture-pane on PI231,
|
||||
@@ -1731,12 +1916,23 @@ test("tuiPromptLanded(READY_PANE, 'Reply with exactly: PONG_TEST') === false (s
|
||||
test("tuiPromptLanded(LANDED_PANE, 'Reply with exactly: PONG_TEST') === true (prompt prefix visible)", () => {
|
||||
assert.equal(_tuiPromptLanded(TUI_LANDED_PANE, "Reply with exactly: PONG_TEST"), true);
|
||||
});
|
||||
test("tuiPromptLanded(READY_PANE, 'ping') === false (needle <3 chars, placeholder present)", () => {
|
||||
test("tuiPromptLanded(READY_PANE, 'ping') === false (prompt text absent from placeholder pane)", () => {
|
||||
assert.equal(_tuiPromptLanded(TUI_READY_PANE, "ping"), false);
|
||||
});
|
||||
test("tuiPromptLanded('❯ ping\\n ? for shortcuts', 'ping') === true (needle present, no placeholder)", () => {
|
||||
assert.equal(_tuiPromptLanded("❯ ping\n ? for shortcuts", "ping"), true);
|
||||
});
|
||||
// C-4 (#133): short prompts (1–2 char first line) MUST be able to land. Threshold
|
||||
// lowered 3 → 2. A 2-char prompt ("hi") present in the pane now lands instead of
|
||||
// 5s-failing with tui_paste_not_landed every time (live-reproduced: "hi").
|
||||
test("tuiPromptLanded('❯ hi\\n ? for shortcuts', 'hi') === true (2-char prompt lands — C-4)", () => {
|
||||
assert.equal(_tuiPromptLanded("❯ hi\n ? for shortcuts", "hi"), true);
|
||||
});
|
||||
// False-positive guard for the lowered threshold: a 2-char needle ABSENT from the
|
||||
// still-empty placeholder pane must NOT land (no spurious Enter into an empty box).
|
||||
test("tuiPromptLanded(READY_PANE, 'hi') === false (2-char prompt not yet visible — no false positive)", () => {
|
||||
assert.equal(_tuiPromptLanded(TUI_READY_PANE, "hi"), false);
|
||||
});
|
||||
// issue #130 root cause: a big bracketed paste shows "[Pasted text #N +M lines]" — must be landed.
|
||||
test("tuiPromptLanded(bracketed-paste pane, big prompt) === true", () => {
|
||||
assert.equal(_tuiPromptLanded("❯ [Pasted text #1 +301 lines]\n ? for shortcuts", "[System] Context 0."), true);
|
||||
|
||||
Reference in New Issue
Block a user