From fe12419386449d2b3d3690d1cef71a16a4a214a2 Mon Sep 17 00:00:00 2001 From: dtzp555-max Date: Sat, 18 Jul 2026 10:11:38 +1000 Subject: [PATCH] =?UTF-8?q?feat(server):=20SPOT-derived=20prompt=20budget?= =?UTF-8?q?=20=E2=80=94=20MAX=5FPROMPT=5FCHARS=20follows=20models.json=20(?= =?UTF-8?q?ADR=200009)=20(#179)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(server): SPOT-derived prompt budget — MAX_PROMPT_CHARS default follows models.json (ADR 0009) Maintainer directive (2026-07-18): the hand-set 150,000-char default (~37.5k English tokens) is obsolete in the long-context era. Instead of a new constant that would rot the same way, the default now derives from the SPOT: MAX_PROMPT_CHARS (default) = max(models.json contextWindow) x 3 chars/token = 200000 x 3 = 600,000 chars today (~150-200k tokens) x3 is the CJK-safe multiplier: English runs ~4 chars/token, CJK ~1-1.5, so a 1M-token-derived char cap would let CJK text sail past the model's real window into an upstream rejection; at x3 the cap fires at roughly the model's true window and OCP truncates gracefully (tail-first) instead. Pure derivePromptCharBudget() in lib/prompt.mjs with a 150k floor guarding degenerate SPOT states (empty models[], absent contextWindow) - a zero budget would truncate every request to nothing. CLAUDE_MAX_PROMPT_CHARS (env) and the runtime settings API remain ABSOLUTE overrides; derivation applies only when neither is set. If models.json ever advertises a larger window (e.g. 1M for the 1M-native models), the budget scales automatically - that advertisement is a separate deliberate decision (quota burn, OpenClaw compaction, TUI paste limits) explicitly NOT made here; see ADR 0009. Behavior change (intended): requests between 150k and 600k chars previously truncated now pass through whole - longer TTFT + higher quota use for those requests. Truncation mechanism/logging unchanged; only the default's provenance changed. ALIGNMENT.md Rule 2: no cli.js citation applies - the truncation guard is OCP-internal prompt shaping; no endpoint, header, or wire field changes. Tests: +4, doubly mutation-proven (max->min fails the largest-window test; dropping the floor fails the floor test). Suite 347 passed / 0 failed. ADR 0009 + index row + README env-table row included (release_kit: env var default change documented). Co-Authored-By: Claude * fix(server): empty CLAUDE_MAX_PROMPT_CHARS falls back to derived default (PR #179 review) Reviewer regression: `!= null` treated an EMPTY env value ("CLAUDE_MAX_PROMPT_CHARS=" in an EnvironmentFile/.env) as explicit -> parseInt("") = NaN -> guard disabled + a false "[System] Note: 0 older messages were truncated" injected into every prompt. Extracted resolvePromptCharBudget() (truthiness contract, matching the old `parseInt(env || default)` behavior) into lib/prompt.mjs so the semantics are mutation-tested: switching back to != null fails the empty-string test. +2 tests, 349/0. Co-Authored-By: Claude --------- Co-authored-by: dtzp555 Co-authored-by: Claude --- README.md | 2 +- docs/adr/0009-spot-derived-prompt-budget.md | 54 +++++++++++++++++++++ docs/adr/README.md | 1 + lib/prompt.mjs | 37 ++++++++++++++ server.mjs | 10 +++- test-features.mjs | 44 ++++++++++++++++- 6 files changed, 144 insertions(+), 4 deletions(-) create mode 100644 docs/adr/0009-spot-derived-prompt-budget.md diff --git a/README.md b/README.md index 412b9d3..f03c9a2 100644 --- a/README.md +++ b/README.md @@ -222,7 +222,7 @@ The canonical list lives in [`models.json`](./models.json) — the single source | `CLAUDE_MAX_CONCURRENT` | `8` | Max concurrent claude processes (`-p`/stream-json path) | | `CLAUDE_MAX_QUEUE` | `16` | Max requests **waiting** for a `-p` concurrency slot. Beyond `CLAUDE_MAX_CONCURRENT`, requests queue (up to this cap) instead of being rejected; when the queue is **also** full, the request gets `HTTP 429` + `Retry-After` (not an opaque 500). Surfaced on `/health.concurrency` + `/health.stats.queueRejections`. | | `CLAUDE_QUEUE_RETRY_AFTER` | `5` | Seconds advertised in the `Retry-After` header on a `-p` concurrency-overflow `429`. | -| `CLAUDE_MAX_PROMPT_CHARS` | `150000` | Prompt truncation limit (chars) | +| `CLAUDE_MAX_PROMPT_CHARS` | *(derived)* | Prompt truncation limit in chars. Default derives from the models.json SPOT: `max(contextWindow) × 3` — currently **600,000** (≈150–200k tokens). Setting this env var (or the runtime settings API) overrides the derivation absolutely. See [ADR 0009](docs/adr/0009-spot-derived-prompt-budget.md). Note: very large prompts burn subscription-window quota quickly and slow TTFT; the TUI-mode paste path is untested beyond ~hundreds of KB. | | `CLAUDE_SESSION_TTL` | `3600000` | Session expiry (ms, default: 1 hour) | | `CLAUDE_CACHE_TTL` | `0` | Response cache TTL (ms, 0 = disabled). Set to e.g. `300000` for 5-min cache. See [Response Cache](#response-cache). | | `CLAUDE_ALLOWED_TOOLS` | `Bash,Read,...,Agent` | Comma-separated tools to pre-approve | diff --git a/docs/adr/0009-spot-derived-prompt-budget.md b/docs/adr/0009-spot-derived-prompt-budget.md new file mode 100644 index 0000000..29515de --- /dev/null +++ b/docs/adr/0009-spot-derived-prompt-budget.md @@ -0,0 +1,54 @@ +# ADR 0009 — Prompt-char budget derives from the models.json SPOT + +Date: 2026-07-18 +Status: Accepted (maintainer directive, 2026-07-18: "37.5k 截断未免太短了吧 … 这个在现在还适用吗") + +## Context + +`MAX_PROMPT_CHARS` (the tail-first truncation guard in `messagesToPrompt`) defaulted to a +hand-set constant of 150,000 chars ≈ 37.5k English tokens — set in the 200k-window era as a +runaway-context guard. Meanwhile `models.json` advertises `contextWindow: 200000` for every +model (and the underlying CLI registry carries 1M native windows for Opus 4.8 / Sonnet 5), and +`scripts/sync-openclaw.mjs` feeds that 200k into OpenClaw's compaction budget. The result was +a standing dishonesty identified in the PR #152 review: **no advertised contextWindow value was +true**, because the proxy silently guillotined every request at ~37.5k tokens — roughly 5× +below the advertised window — logging only a server-side warning the client never sees. + +Raising the constant to another hand-set number would rot the same way. Following the model's +native 1M directly is also wrong: chars ≠ tokens (CJK runs ~1–1.5 chars/token vs ~4 for +English, so a 1M-token char cap would let CJK text sail past the model's real window into an +upstream rejection), single near-window requests can consume a large fraction of a 5-hour +subscription quota window, and the TUI paste path is untested at megabyte scale. + +## Decision + +The default budget **derives from the SPOT** instead of being a constant: + +``` +MAX_PROMPT_CHARS (default) = max(models.json models[].contextWindow) × 3 chars/token + = 200000 × 3 = 600,000 chars today +``` + +Implemented as the pure `derivePromptCharBudget(models, {charsPerToken = 3, floor = 150000})` +in `lib/prompt.mjs` (unit-tested; floor guards degenerate SPOT states). The multiplier ×3 is +deliberately conservative: full window for English, and CJK text reaches the model's real +window at roughly the same point the cap fires — so OCP truncates gracefully (tail-first) +instead of the upstream rejecting outright. + +`CLAUDE_MAX_PROMPT_CHARS` (env) and the runtime settings API remain **absolute overrides**; +the derivation applies only when neither is set. + +## Consequences + +- The advertised `contextWindow: 200000` becomes honest: the proxy now actually accepts + prompts of that order (English ≈150–200k tokens) before truncating. +- If `models.json` ever advertises a larger window (e.g. 1M for the 1M-native models), the + budget scales automatically — no code change. Whether to advertise 1M is a **separate, + deliberate decision** (quota burn per request, OpenClaw compaction memory, TUI paste + limits) and is explicitly NOT made by this ADR; the current recommendation is to keep + 200000 advertised until a real >200k use case appears. +- One-time behavior change: requests between 150k and 600k chars that were previously + truncated now pass through whole — longer TTFT and higher quota consumption for those + requests, by design. +- The truncation mechanism, logging, and the multimodal-path budget threading (PR #154's F2, + pending) are unchanged — only the default value's provenance changed. diff --git a/docs/adr/README.md b/docs/adr/README.md index c68ba8d..8c40f47 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -25,6 +25,7 @@ New ADRs increment from the highest existing number. Filenames are | [0006](0006-openai-shim-scope.md) | OpenAI Shim Scope | The Class A / Class B taxonomy. Class A endpoints (`cli.js`-mirror) keep Rules 1–5 verbatim; Class B endpoints (OCP-owned compatibility surface — `/v1/chat/completions`, `/v1/models`, admin endpoints) are anchored to OpenAI's spec (B.1) or to an authorizing ADR (B.2). Triggered by PR #99 (external `response_format` honoring). Grandfathers the existing B.2 inventory at v3.16.4. | | [0007](0007-tui-interactive-mode.md) | TUI Interactive Mode | Why TUI-mode spawns an interactive `claude` in a tmux pane (no `-p`) to reach the **subscription** billing pool (`cc_entrypoint=cli`) rather than the metered Agent SDK pool. Owns the TUI spawn machinery: entrypoint labeling, credential-isolated home, MCP hard-disable, session namespace + defunct-session reaping, the independent concurrency bound, and the `/health` `tui` block. **Single-user only** — hard FATAL on multi-user configs. | | [0008](0008-tui-warm-pane-pool.md) | TUI Warm Pane Pool | Why `OCP_TUI_POOL_SIZE` pre-boots **single-use** `claude` panes (one turn each, own `--session-id`) — and why reuse is forbidden (`transcript.mjs` returns the last assistant entry in the file, so a reused session leaks the earlier turn's text). Measured −41% end-to-end. Defines the pool↔reaper invariant (exemption by exact name from a live registry; drain before every sweep so `kill-server` zombie reaping survives) and the standing idle-process cost. Extends ADR 0007. | +| [0009](0009-spot-derived-prompt-budget.md) | SPOT-Derived Prompt Budget | Why `MAX_PROMPT_CHARS`'s default is `max(models.json contextWindow) × 3 chars/token` (600k chars today) instead of a hand-set constant — the old 150k silently under-delivered the advertised window ~5×. ×3 is the CJK-safe multiplier; env/settings stay absolute overrides; whether to advertise 1M windows is explicitly a separate decision. | ## When to write a new ADR diff --git a/lib/prompt.mjs b/lib/prompt.mjs index 2c38d56..12f9dae 100644 --- a/lib/prompt.mjs +++ b/lib/prompt.mjs @@ -15,3 +15,40 @@ export function appendOperatorPrompt(base, operatorAppend) { const op = typeof operatorAppend === "string" ? operatorAppend.trim() : ""; return op ? `${base}\n\n${op}` : base; } + +// Derive the default prompt-char budget from the models.json SPOT (ADR 0009). +// +// The old default was a hand-set constant (150000 chars ≈ 37.5k English tokens) from the +// 200k-window era — silently far below what the advertised contextWindow promises. Instead +// of picking a new constant that will also rot, the default now FOLLOWS the SPOT: +// +// budget = max(models[].contextWindow) × charsPerToken +// +// charsPerToken = 3 is deliberately conservative: English runs ~4 chars/token, CJK ~1–1.5. +// At ×3, a 200k-token window yields 600,000 chars — full window for English, and CJK text +// hits the model's real window at roughly the same point the cap fires, so we truncate +// (graceful, tail-first) rather than let the upstream reject the request outright. +// +// The floor guards the degenerate cases (empty/missing models[], absent contextWindow): +// fall back to the historical constant rather than 0 — a zero budget would truncate every +// request to nothing, which is fail-OPEN in the "serve garbage" sense. CLAUDE_MAX_PROMPT_CHARS +// remains an absolute operator override at the call site (server.mjs); this function is only +// the unset-env default. +export function derivePromptCharBudget(models, { charsPerToken = 3, floor = 150000 } = {}) { + const windows = (Array.isArray(models) ? models : []) + .map(m => m?.contextWindow) + .filter(w => Number.isFinite(w) && w > 0); + if (windows.length === 0) return floor; + return Math.max(floor, Math.max(...windows) * charsPerToken); +} + +// Resolve the effective budget from the env var + SPOT. TRUTHINESS (not != null) on the env +// value deliberately: an EMPTY value ("CLAUDE_MAX_PROMPT_CHARS=" in a systemd EnvironmentFile +// or .env) must mean "use the default" — exactly the old `parseInt(env || "150000")` contract. +// Treating "" as explicit gives parseInt("") = NaN, and a NaN cap silently DISABLES the +// runaway-context guard while injecting a false "[System] Note: 0 older messages were +// truncated" line into every prompt (caught in PR #179 review). Non-empty garbage still +// parses to NaN — the pre-existing class, slated for parseIntEnv routing in PR #154. +export function resolvePromptCharBudget(rawEnv, models, opts) { + return rawEnv ? parseInt(rawEnv, 10) : derivePromptCharBudget(models, opts); +} diff --git a/server.mjs b/server.mjs index 0102bb0..112e512 100644 --- a/server.mjs +++ b/server.mjs @@ -49,7 +49,7 @@ import { TuiSemaphore, SemaphoreAbortError, recordTuiEntrypoint, buildTuiHealthB import { TuiPanePool, resolvePoolSize, POOL_MAX_SIZE } from "./lib/tui/pool.mjs"; import { TuiDeltaAssembler, DEFAULT_HOLDBACK_CHARS, resolveStreamHoldback } from "./lib/tui/stream.mjs"; import { createSerialMutex, createTtlCache, isTokenExpiring, orderLabelsLastGoodFirst } from "./lib/spawn-auth.mjs"; -import { appendOperatorPrompt } from "./lib/prompt.mjs"; +import { appendOperatorPrompt, resolvePromptCharBudget } from "./lib/prompt.mjs"; const __dirname = dirname(fileURLToPath(import.meta.url)); const _pkg = JSON.parse(readFileSync(join(__dirname, "package.json"), "utf8")); @@ -1148,7 +1148,13 @@ function buildCliArgs(cliModel, systemPrompt) { // Truncation guard: if total chars exceed MAX_PROMPT_CHARS, keep the system // message(s) + first user message + last N messages, dropping the middle. // This prevents runaway context from gateway-side conversation accumulation. -let MAX_PROMPT_CHARS = parseInt(process.env.CLAUDE_MAX_PROMPT_CHARS || "150000", 10); +// +// Default is SPOT-DERIVED (ADR 0009): max(models.json contextWindow) × 3 chars/token — +// currently 200000 × 3 = 600,000 chars — instead of the old hand-set 150000 (≈37.5k +// English tokens), which silently under-delivered the advertised window by ~5×. The env +// var (and the runtime settings API below) remain absolute operator overrides. If +// models.json ever advertises a bigger window, this budget follows automatically. +let MAX_PROMPT_CHARS = resolvePromptCharBudget(process.env.CLAUDE_MAX_PROMPT_CHARS, modelsConfig.models); // Flatten OpenAI content (string | array of parts) to plain text for the prompt. // Array content: concatenate text parts; replace non-text parts (e.g. image_url) diff --git a/test-features.mjs b/test-features.mjs index 4b3b549..1ad17c1 100644 --- a/test-features.mjs +++ b/test-features.mjs @@ -844,7 +844,49 @@ test("doctor falls back to currentVersion when origin/main unreachable (no stale // contract lives in lib/prompt.mjs. Mutation-proof: make appendOperatorPrompt // return `base` unconditionally and the first test fails; make it stop trimming // and the whitespace test fails. -import { appendOperatorPrompt } from "./lib/prompt.mjs"; +import { appendOperatorPrompt, derivePromptCharBudget, resolvePromptCharBudget } from "./lib/prompt.mjs"; + +console.log("\nPrompt-char budget (ADR 0009 — SPOT-derived):"); + +// Mutation-proof: drop the ×charsPerToken and the first test fails; drop the +// Math.max floor guard and the floor tests fail; use min() instead of max() over +// windows and the largest-window test fails. +test("derivePromptCharBudget: LARGEST contextWindow × 3 chars/token", () => { + const models = [{ contextWindow: 200000 }, { contextWindow: 100000 }]; + assert.equal(derivePromptCharBudget(models), 600000); +}); + +test("derivePromptCharBudget: matches the live models.json SPOT (200k → 600k today)", () => { + const spot = JSON.parse(tuiReadFileSync(new URL("./models.json", import.meta.url), "utf8")); + assert.equal(derivePromptCharBudget(spot.models), 600000); +}); + +test("derivePromptCharBudget: floor wins over a tiny/absent window; empty input → floor", () => { + assert.equal(derivePromptCharBudget([{ contextWindow: 1000 }]), 150000, "3k chars would truncate everything — floor guards it"); + assert.equal(derivePromptCharBudget([]), 150000); + assert.equal(derivePromptCharBudget(undefined), 150000); + assert.equal(derivePromptCharBudget([{ id: "x" }, { contextWindow: "junk" }, { contextWindow: -5 }]), 150000); +}); + +test("derivePromptCharBudget: charsPerToken and floor are tunable parameters", () => { + assert.equal(derivePromptCharBudget([{ contextWindow: 1000000 }], { charsPerToken: 3 }), 3000000); + assert.equal(derivePromptCharBudget([], { floor: 42 }), 42); +}); + +// PR #179 review regression: EMPTY env value must mean "use the default" (the old +// `parseInt(env || "150000")` contract). Mutation-proof: switch the resolver's +// truthiness check to `!= null` and the empty-string test fails (NaN ≠ 600000). +test("resolvePromptCharBudget: empty/unset env → SPOT-derived default, never NaN", () => { + const models = [{ contextWindow: 200000 }]; + assert.equal(resolvePromptCharBudget("", models), 600000, "CLAUDE_MAX_PROMPT_CHARS= (empty) must fall back to derived"); + assert.equal(resolvePromptCharBudget(undefined, models), 600000); +}); + +test("resolvePromptCharBudget: a set env value overrides the derivation absolutely", () => { + const models = [{ contextWindow: 200000 }]; + assert.equal(resolvePromptCharBudget("300000", models), 300000); + assert.equal(resolvePromptCharBudget("150000", models), 150000, "explicit legacy value wins over the bigger derived default"); +}); console.log("\nSystem-prompt operator append:");