From 3dc023e5966b1edba35be22d14f8a54845ad105c Mon Sep 17 00:00:00 2001 From: dtzp555-max Date: Mon, 27 Jul 2026 12:26:51 +1000 Subject: [PATCH] feat(models): align maxTokens with the CLI registry (#195) (#208) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(models): align maxTokens with the CLI registry (#195) Every Opus entry and claude-sonnet-5 go 16384 -> 64000; claude-sonnet-4-6 and claude-haiku-4-5 go to 32000 (haiku was 8192). These are the max_output_tokens.default each model declares in the compiled CLI 2.1.220 registry, extracted id-anchored: claude-opus-5 max_output_tokens:{default:64000,upper:128000} claude-opus-4-8 max_output_tokens:{default:64000,upper:128000} claude-opus-4-7 max_output_tokens:{default:64000,upper:128000} claude-opus-4-6 max_output_tokens:{default:64000,upper:128000} claude-sonnet-5 max_output_tokens:{default:64000,upper:128000} claude-sonnet-4-6 max_output_tokens:{default:32000,upper:128000} claude-haiku-4-5 max_output_tokens:{default:32000,upper:64000} This is not cosmetic metadata. OCP itself never enforces maxTokens — server.mjs does not read it — but it is propagated to OpenClaw (setup.mjs, scripts/sync-openclaw.mjs), and OpenClaw CAPS the outbound request with it: maxTokens = Math.min(baseMaxTokens + thinkingBudget, modelMaxTokens) with reasoning budgets of medium 8192 / high 16384, and a clamp `if (maxTokens <= thinkingBudget) thinkingBudget = maxTokens - 1024`. So at `high` reasoning against a model declared at 16384: maxTokens resolves to 16384, the clamp fires, thinking takes 15360, and roughly 1024 tokens are left for the visible answer. That is what this repo shipped for every model until now, and it is the actual harm — not the advertised-vs-real mismatch. ocp-connect keeps its own family-prefix table (/v1/models does not expose maxTokens, so it cannot derive them). Set to the family FLOOR — opus 64000, sonnet 32000 because sonnet-4-6 is 32000 while sonnet-5 is 64000, haiku 32000 — because prefixes cannot distinguish versions and under-advertising merely caps a client lower than the model allows, whereas over-advertising would promise capacity a family member does not have. Commented in place. New test asserts the PRINCIPLE, not the numbers: every maxTokens must exceed OpenClaw's `high` thinking budget so an answer still fits. A value-pinning test would need editing on every model addition and would not explain itself. Mutation-proven: reverting one entry to 16384 fails with "claude-opus-5 declares maxTokens=16384 <= 16384: at OpenClaw's 'high' reasoning level the thinking budget would consume all but ~1024 tokens". Owner-approved decision (option: align with registry defaults). Tradeoff recorded in CHANGELOG: longer answers, correspondingly higher per-request quota burn on long generations. Verified: bash -n on ocp-connect, and py_compile on its embedded python block (lines 101-334) so the comment sits at a valid indentation. Tests: 458 passed, 0 failed (457 + 1). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017gbqUZ8HfBZpjjbzQ85oH8 * fix(models): rewrite the maxTokens rationale — my justification was factually wrong The numbers were right, the code was right, and the reason I gave for both was wrong. Verified every part of the review's finding first-hand before rewriting. WHAT I CLAIMED: OpenClaw caps outbound requests via `maxTokens = Math.min(baseMaxTokens + thinkingBudget, modelMaxTokens)`, so a 16384 declaration at `high` reasoning left ~1024 tokens for the visible answer, and raising it would produce longer answers and higher quota burn. WHY THAT CANNOT HAPPEN FOR OCP — three independent breaks, each confirmed here: scripts/sync-openclaw.mjs:75 api: "openai-completions" The clamp I cited lives behind `case "anthropic-messages"`; the openai-completions transport has no thinkingBudget concept at all, so that code never runs for a claude-local provider. buildCliArgs pushes: --input-format, --disallowedTools, --allowedTools, --dangerously-skip-permissions, --mcp-config No output-token flag of any kind reaches the CLI. OCP cannot cap output even if it wanted to. grep max_completion_tokens server.mjs lib/ -> 0 occurrences That is the field OpenClaw would send to a local openai-completions endpoint, and this repo never reads it. So "expect longer answers and higher per-request quota burn" was a promise that would not have materialised. That is worse than a wrong number: a release note that tells operators to expect a behavioural change which cannot occur. The change still stands, on the honest reason: models.json is the SPOT (ADR 0003) and its values should be true. maxTokens is ADVERTISED metadata consumed only by clients that choose to honour it. CHANGELOG and the test comment now say exactly that, and say plainly that OCP behavior is unchanged. Test replaced, not just re-worded. The old one asserted `maxTokens > 16384`, which review showed lets every entry sit at 16385 and still call itself "registry-aligned" — the actual claim had no coverage. Now pinned per model against a recorded table of CLI 2.1.220 id-anchored values, with an explicit failure for any model missing a row. Mutation-proven on both holes: all entries at 16385 -> CAUGHT (the old test passed this) sonnet-4-6 wrongly at 64000 -> CAUGHT Also fixed ocp-connect's unknown-id fallback (8192 -> 32000). Review flagged it as contradicting the invariant: 32000 is the LOWEST max_output_tokens.default in the registry, so it is the safe floor and no longer sits below every family in its own table. Unreachable today, wrong to leave inconsistent. Verified: bash -n on ocp-connect and py_compile on its embedded python block. Tests: 458 passed, 0 failed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017gbqUZ8HfBZpjjbzQ85oH8 * fix(ocp-connect): revert the unknown-id fallback to 8192 — 32000 over-advertised The previous commit raised ocp-connect's unknown-id fallback from 8192 to 32000 on the stated grounds that 32000 was "the LOWEST max_output_tokens.default in the CLI 2.1.220 registry". That is false, and the change it justified made this PR violate its own thesis. Pairing-free enumeration of the compiled registry (17 occurrences, no grep-window artifacts): 7 max_output_tokens:{default:64000,upper:128000} 5 max_output_tokens:{default:32000,upper:64000} 2 max_output_tokens:{default:8192,upper:8192} 2 max_output_tokens:{default:32000,upper:32000} 1 max_output_tokens:{default:32000,upper:128000} distinct defaults: 8192 / 32000 / 64000 GLOBAL MINIMUM: 8192 The floor is 8192, held by claude-3-5-haiku and claude-3-5-sonnet — and those are exactly the ids that reach the fallback, since "claude-3-5-*" matches none of the family prefixes. So the raise over-advertised the only two real models the branch serves, by 4x, in a PR whose entire claim is that advertised metadata should be true. Verified by exercising the real classifier (verbatim source slice, not a replica): claude-3-5-haiku 8192 FALLBACK registry 8192 exact claude-3-5-sonnet 8192 FALLBACK registry 8192 exact claude-opus-5 64000 family registry 64000 exact claude-sonnet-4-6 32000 family registry 32000 exact claude-haiku-4-5-20251001 32000 family registry 32000 exact claude-sonnet-5 32000 family registry 64000 under (safe) Also in this commit: - The family table's comment said "FAMILY FLOOR of the CLI registry". It is the floor over each family's members CURRENTLY IN models.json, not over the registry family: claude-opus-4-0 / -4-5 are opus-family at 32000, so the opus row would over-advertise them 2x. Unreachable today (ocp-connect derives ids from /v1/models, i.e. models.json), but the comment now states the real scope and what to do if a legacy opus is added. - The test table keyed haiku on "claude-haiku-4-5-20251001" while telling the reader to extract values id-anchored. That id has ZERO id-anchored hits: the registry record is id:"claude-haiku-4-5" and the dated string appears only as its provider_ids.first_party (10 bare-string hits). Following the instruction on that row returns nothing, which is precisely what pushes a reader back to the bare-string search this test exists to prevent. The row now carries the registry id. Reviewer credit: the reviewer that found this also retracted its own earlier finding, which is what produced the 32000 fallback in the first place. Both directions verified independently here before acting. server.mjs: unchanged by this commit (ALIGNMENT.md requires no cli.js citation). Suite: 462 passed, 0 failed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017gbqUZ8HfBZpjjbzQ85oH8 * docs(ocp-connect): my unreachability reason was wrong — state the invariant that holds Review of cb87b82 approved the change and flagged the new comment's reasoning. Verified both entry points it named; both are real. I wrote that claude-opus-4-0/-4-5 are unreachable because "ocp-connect derives ids from /v1/models, i.e. models.json". Neither half is reliable: - base_url is user-supplied (ocp-connect:510 builds it from arguments), so /v1/models is the REMOTE server's models.json — a different OCP version, or a fork, not this checkout. - on a JSON parse failure the ids come from a hardcoded three-id list (:114), not from /v1/models at all. Checked all three: opus-4-6 -> 64000, sonnet-4-6 -> 32000, haiku-4 -> 32000. All exact against the registry, so safe today — but it is a genuine non-SPOT entry point that my sentence denied existed. The invariant that actually holds is narrower: no released OCP has ever served a legacy opus id. Same conclusion, and now it names the thing that would break it. This is the second reasoning defect in this PR where the number was right and the justification was not. Recording it in the comment rather than quietly fixing it. Out of scope, noted for a future touch: the bare `except:` at :114 catches BaseException, so a KeyboardInterrupt mid-parse silently yields the fallback list. server.mjs: unchanged (ALIGNMENT.md requires no cli.js citation). Suite: 462 passed, 0 failed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017gbqUZ8HfBZpjjbzQ85oH8 --------- Co-authored-by: dtzp555 Co-authored-by: Claude Opus 5 (1M context) --- CHANGELOG.md | 6 ++++++ models.json | 14 +++++++------- ocp-connect | 28 +++++++++++++++++++++++++--- test-features.mjs | 32 ++++++++++++++++++++++++++++++++ 4 files changed, 70 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e47a51f..2efe66f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,11 @@ # Changelog +## Unreleased + +### Changed + +- **`maxTokens` now matches the CLI registry instead of a uniform 16384 (#195).** Every Opus entry and `claude-sonnet-5` go to **64000**, `claude-sonnet-4-6` and `claude-haiku-4-5` to **32000** — the `max_output_tokens.default` each model declares in the compiled CLI 2.1.220 registry. This corrects **advertised metadata only**: `models.json` is the SPOT (ADR 0003) and every value in it should be the truth about the model. **It changes nothing about how OCP behaves.** OCP never enforces `maxTokens` — `buildCliArgs` passes no output-token flag to the CLI at all — and OpenClaw addresses a local OCP over `openai-completions`, whose request field (`max_completion_tokens`) appears nowhere in this repo. The value is consumed only by clients that choose to honour it, via `setup.mjs` / `scripts/sync-openclaw.mjs` / `ocp-connect`. **Expect no change in answer length or quota burn.** `ocp-connect`'s independent family table moves to the floor over each family's current `models.json` members (opus 64000, sonnet 32000, haiku 32000), since prefixes cannot distinguish versions. Its unknown-id fallback stays at **8192** — the registry's global minimum, held by `claude-3-5-haiku` and `claude-3-5-sonnet`, which are the only real ids that reach it (`claude-3-5-*` matches no family prefix). + ## v3.25.0 — 2026-07-27 Minor release. Headline: **Claude Opus 5** joins the model list and the `opus` alias now resolves to it. Alongside it, two `server.mjs` correctness fixes found by review rather than by users — a monotonic in-flight-counter leak, and cache keys that were hashing the alias string instead of the model it resolves to. The three TUI/prompt fixes that landed after v3.24.0 are included here too. diff --git a/models.json b/models.json index f3c13d6..0db0b7a 100644 --- a/models.json +++ b/models.json @@ -8,7 +8,7 @@ "openclawName": "Claude Opus 5 (via CLI)", "reasoning": true, "contextWindow": 200000, - "maxTokens": 16384 + "maxTokens": 64000 }, { "id": "claude-opus-4-8", @@ -16,7 +16,7 @@ "openclawName": "Claude Opus 4.8 (via CLI)", "reasoning": true, "contextWindow": 200000, - "maxTokens": 16384 + "maxTokens": 64000 }, { "id": "claude-opus-4-7", @@ -24,7 +24,7 @@ "openclawName": "Claude Opus 4.7 (via CLI)", "reasoning": true, "contextWindow": 200000, - "maxTokens": 16384 + "maxTokens": 64000 }, { "id": "claude-opus-4-6", @@ -32,7 +32,7 @@ "openclawName": "Claude Opus 4.6 (via CLI)", "reasoning": true, "contextWindow": 200000, - "maxTokens": 16384 + "maxTokens": 64000 }, { "id": "claude-sonnet-5", @@ -40,7 +40,7 @@ "openclawName": "Claude Sonnet 5 (via CLI)", "reasoning": true, "contextWindow": 200000, - "maxTokens": 16384 + "maxTokens": 64000 }, { "id": "claude-sonnet-4-6", @@ -48,7 +48,7 @@ "openclawName": "Claude Sonnet 4.6 (via CLI)", "reasoning": true, "contextWindow": 200000, - "maxTokens": 16384 + "maxTokens": 32000 }, { "id": "claude-haiku-4-5-20251001", @@ -56,7 +56,7 @@ "openclawName": "Claude Haiku 4.5 (via CLI)", "reasoning": false, "contextWindow": 200000, - "maxTokens": 8192 + "maxTokens": 32000 } ], "aliases": { diff --git a/ocp-connect b/ocp-connect index 0722806..7d1b0ad 100755 --- a/ocp-connect +++ b/ocp-connect @@ -129,10 +129,25 @@ provider = { # re-trip it. Family prefixes classify any versioned ID correctly with no per-model # edit. (ADR 0003: models.json is the SPOT for model existence; /v1/models does not # expose reasoning/maxTokens, so family classification stays here.) +# maxTokens values are the floor of the CLI registry's max_output_tokens.default over the +# family members CURRENTLY IN models.json (opus 64000; sonnet 32000, because sonnet-4-6 is +# 32000 while sonnet-5 is 64000; haiku 32000) — NOT over the whole registry family. The +# registry also holds claude-opus-4-0 / -4-5 at 32000, which this opus row would over-advertise +# 2x. What keeps that unreachable is NOT "the ids come from models.json" — the ids reaching +# get_model_meta have two sources that are not this checkout: /v1/models is fetched from a +# user-supplied base_url, so it is the REMOTE server's models.json (possibly a different OCP +# version or a fork), and on a JSON parse failure the ids come from the hardcoded list a few +# lines above instead. The invariant that actually holds is that no released OCP has ever +# served a legacy opus id. Adding one to models.json breaks it and this row must drop to 32000. +# Family prefixes cannot distinguish versions, so a floor is used deliberately: +# under-advertising caps a client lower than the model allows, which is safe, whereas +# over-advertising would promise capacity a member of the family does not have. +# models.json is authoritative per ADR 0003; this table only exists because /v1/models does +# not expose maxTokens. model_meta = { - "claude-opus": {"name": "Claude Opus (OCP)", "reasoning": True, "maxTokens": 16384}, - "claude-sonnet": {"name": "Claude Sonnet (OCP)", "reasoning": True, "maxTokens": 16384}, - "claude-haiku": {"name": "Claude Haiku (OCP)", "reasoning": False, "maxTokens": 8192}, + "claude-opus": {"name": "Claude Opus (OCP)", "reasoning": True, "maxTokens": 64000}, + "claude-sonnet": {"name": "Claude Sonnet (OCP)", "reasoning": True, "maxTokens": 32000}, + "claude-haiku": {"name": "Claude Haiku (OCP)", "reasoning": False, "maxTokens": 32000}, } def get_model_meta(mid): @@ -140,6 +155,13 @@ def get_model_meta(mid): for prefix, meta in sorted(model_meta.items(), key=lambda x: -len(x[0])): if mid.startswith(prefix): return meta + # Unknown id. 8192 is the GLOBAL MINIMUM max_output_tokens.default across all 17 records in + # the CLI 2.1.220 registry (distinct values: 8192 / 32000 / 64000). The two records holding + # it — claude-3-5-haiku and claude-3-5-sonnet, both default:8192, upper:8192 — are precisely + # the ones that reach this line, because "claude-3-5-*" matches none of the family prefixes + # above. Raising this to a family-table value would over-advertise those two by 4x. Keep 8192: + # under-advertising caps a client lower than a model allows, which is safe; over-advertising + # promises capacity that does not exist. return {"name": mid + " (OCP)", "reasoning": False, "maxTokens": 8192} for mid in model_ids: diff --git a/test-features.mjs b/test-features.mjs index 0721442..7cd44f5 100644 --- a/test-features.mjs +++ b/test-features.mjs @@ -4382,6 +4382,38 @@ test("models.json: every aliases value resolves to a real models[].id (referenti } }); +// maxTokens is ADVERTISED metadata, not an OCP-enforced limit (#195). OCP never reads it — +// buildCliArgs passes no output-token flag to the CLI — and OpenClaw reaches a local OCP over +// `openai-completions`, whose request field (max_completion_tokens) appears nowhere in this repo. +// It is consumed only by clients that choose to honour it, via setup.mjs / sync-openclaw.mjs / +// ocp-connect. So the invariant worth testing is simply that models.json tells the truth: each +// value must equal the model's max_output_tokens.default in the CLI registry. +// +// Pinned per model deliberately. A threshold assertion would let every entry sit at some arbitrary +// value above the bar and still call itself "registry-aligned" — which is the actual claim. Adding +// a model means adding a row here, and that is the point: the row is where you record what the +// registry said when you checked. +// Keys are models.json ids; values are the CLI 2.1.220 registry's max_output_tokens.default, +// each extracted id-anchored (grep 'id:""' + the following bytes) — never by bare-string +// search, which matches cross-references inside OTHER models' records and silently attributes +// the wrong number. ONE KEY IS NOT A REGISTRY ID: models.json carries the dated haiku id, but +// the registry record is id:"claude-haiku-4-5" (the dated string appears only as that record's +// provider_ids.first_party). Anchor the haiku row on the SHORT id; anchoring on the dated one +// returns nothing, which is what tempts the next reader back into a bare-string search. +const _spotRegistryMaxTokens = { + "claude-opus-5": 64000, "claude-opus-4-8": 64000, "claude-opus-4-7": 64000, "claude-opus-4-6": 64000, + "claude-sonnet-5": 64000, "claude-sonnet-4-6": 32000, + "claude-haiku-4-5-20251001": 32000, // registry id: claude-haiku-4-5 +}; +test("models.json: every maxTokens equals the CLI registry's max_output_tokens.default (#195)", () => { + for (const m of _spotModels.models) { + const want = _spotRegistryMaxTokens[m.id]; + assert.ok(want !== undefined, + `${m.id} has no recorded registry value — extract it id-anchored from the CLI binary and add a row`); + assert.equal(m.maxTokens, want, `${m.id}: models.json says ${m.maxTokens}, CLI registry says ${want}`); + } +}); + test("models.json: every legacyAliases value resolves to a real models[].id (referential integrity)", () => { for (const [name, target] of Object.entries(_spotModels.legacyAliases || {})) { assert.ok(_spotModelIds.has(target), `legacyAliases.${name} -> '${target}' is a dangling alias (no matching models[].id)`);