mirror of
https://github.com/dtzp555-max/ocp.git
synced 2026-07-21 21:15:09 +00:00
feat(server): honor OpenAI response_format for structured-output clients (#153)
* feat(server): honor OpenAI response_format for structured-output clients `/v1/chat/completions` advertises OpenAI compatibility but ignored `response_format`, so clients requiring machine-parseable JSON (Home Assistant AI Tasks, Honcho, OpenAI-SDK scripts) received free-form assistant prose — markdown tables, ```json fences, trailing commentary — that fails JSON.parse. This honors the OpenAI `response_format` contract on the `-p` path: - New `lib/structured-output.mjs` (pure, unit-tested): `detectStructuredOutput` (json_schema / json_object), `structuredSystemInstruction` (strict JSON-only steering, escalated on retry), `extractJsonPayload` (string-aware balanced slice that unwraps fences/prose), and a minimal JSON-Schema `validateJsonSchema` (types, required, enum, const, additionalProperties, nullability, items, min/maxItems). - `server.mjs`: `runStructuredCompletion` retries up to `OCP_STRUCTURED_MAX_ATTEMPTS` (default 3), returns the canonical JSON string as `message.content`, and yields HTTP 422 (`invalid_response_error`) if no valid JSON can be produced. Structured requests take their own path (bypass the cache, which does not key on response_format). Non-structured requests are byte-for-byte unchanged, streaming included. - Nullability precedence: a `null` value is accepted whenever the schema permits null (`type:["x","null"]` / `nullable:true`), even if a bare `enum` omits null — matches OpenAI behaviour and fixes real Home Assistant schemas (`type:["string","null"], enum:["Loxone"]`) that otherwise 422 on null. - README: Structured Outputs section + `OCP_STRUCTURED_MAX_ATTEMPTS` env row. - 18 new unit tests (281 passed, 0 failed). Endpoint class: B.1 (OpenAI-compatibility surface, `/v1/chat/completions`). Specification: OpenAI chat/completions `response_format` (https://platform.openai.com/docs/api-reference/chat/create#chat-create-response_format). Authorizing ADR: ADR 0006 — OpenAI shim scope. cli.js does NOT perform this operation (it speaks Anthropic's protocol, not OpenAI's); scope is justified under ADR 0006 Class B.1 (OpenAI spec as protocol authority). Revives closed PR #99. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(server): structured-output caching + json_mode alias Follow-up on the response_format path, closing the two gaps vs closed PR #99: - **Validated caching (improves on #99).** Structured responses now use the OCP cache when CLAUDE_CACHE_TTL>0, on a structured-keyed hash: cacheHash gains an `structured` marker folding the detected response_format/schema into the key, so a JSON reply never collides with the conversational answer to the same prompt and different schemas never share a slot. Only a *validated* result is written back — a 422 is never cached. (#99 cached the fence-stripped but *unvalidated* output; this caches only schema-valid JSON.) The marker is absent for normal requests, so existing cache hashes are byte-identical. - **json_mode alias.** Honor the non-standard top-level `json_mode: true` flag as a json_object alias, matching #99's activation set. Disclosed as non-spec. - README: json_mode shape + caching note. +2 unit tests (283 passed, 0 failed). Endpoint class: B.1 (/v1/chat/completions), ADR 0006. json_mode is a non-OpenAI convenience alias (disclosed); everything else stays within OpenAI's response_format spec. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(server): address PR #153 review — $ref/strict, extraction safety, refusal, singleflight Remediates the maintainer's merge-blocking findings on the structured-output PR, and rebases onto current main (the one-hunk test-features.mjs conflict — both sides appended tests — resolved by keeping both blocks). Class B.1 (OpenAI-compat): spec authority is OpenAI chat/completions `response_format` (https://platform.openai.com/docs/api-reference/chat/create#chat-create-response_format), authorized by ADR 0006. No cli.js analogue (claude -p has no native response_format); scope is the B.1 shim, not a Class A forward. Finding 1 (correctness gate) — strict:true + $ref/$defs rejected valid objects 100% of the time. `noExtra = addl === false || (strict && addl === undefined)` treated a nested {$ref:"#/$defs/step"} as an empty-properties object and, under strict, rejected every real key as "additional property not allowed" — exactly the shape the OpenAI SDK emits (zodResponseFormat / client.beta.chat.completions.parse) and OpenAI's own docs example. Fix: validateJsonSchema now resolves same-document $ref against the root $defs/definitions, handles allOf/anyOf/oneOf composition, and only infers additionalProperties:false from strict when the object actually declares its own non-empty properties and is not a composite. Explicit additionalProperties:false is always honoured, so validation is not weakened (tests prove an extra key and a missing required key still fail under strict). Finding 2 (correctness gate) — the extractor served JSON the model did not mean. json_object mode had no validation at all: a refusal like `I can't. The schema is {"type":"object"}` returned the embedded object as the answer. Now json_object requires the WHOLE reply to parse as a single JSON value, and schema mode rejects a reply carrying more than one top-level JSON value (Schema:{}/Answer:{}, Option A/Option B) rather than silently picking the first. The schema-validated value is still returned; nothing unvalidated is served. Finding 3 — replaced the invented `invalid_response_error` 422 with OpenAI's assistant `refusal` field (200, content:null, refusal:<reason>, finish_reason:"stop"), streaming and non-streaming, so SDK clients take their refusal branch instead of throwing an opaque UnprocessableEntityError. Finding 5 — runStructuredCompletion no longer bypasses stampede protection. Identical concurrent one-off structured requests now share one singleflight (independent of cache enablement), so N callers no longer cost N × up-to-3 spawns. Cache read/write still gated on CLAUDE_CACHE_TTL; refusals are never cached. Docs: README structured-output § updated (refusal field, $ref/composition support, whole-reply json_object rule, ambiguous-multi-value rejection) plus a Caching & cost paragraph stating the post-2026-06-15 model, the up-to-N-spawn worst case, the singleflight + validated-cache guards, and the OCP_STRUCTURED_MAX_ATTEMPTS=1 / per-key quota levers. Tests: +11 (all pure-module) — OpenAI's doc $ref/$defs schema under strict:true accepts a conforming reply and still rejects extra/missing keys; anyOf/allOf; unresolvable $ref skipped; json_object refusal-embedded-json rejected; >1-top-level-value rejected. 360 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(server): address PR #153 review round 2 — cyclic-$ref guard + NaN attempts guard Remediates the two remaining merge-blocking findings from the round-2 review. Class B.1 (OpenAI-compat): spec authority is OpenAI chat/completions `response_format` (https://platform.openai.com/docs/api-reference/chat/create#chat-create-response_format), authorized by ADR 0006. No cli.js analogue (claude -p has no native response_format, and the retry cap OCP_STRUCTURED_MAX_ATTEMPTS is OCP's own coercion loop, not a cli.js operation); scope stays the B.1 shim, not a Class A forward. BLOCKER — cyclic $ref stack-overflowed the validator. resolveRef + the $ref branch of validateJsonSchema had no cycle detection: a pure ref→ref cycle ({$defs:{a:{$ref:b},b:{$ref:a}},$ref:a}) recursed independent of the data and threw RangeError for ANY reply value (even `5`), caught upstream as a 500 but only after 1–3 metered spawns — a request-controlled cost-amplification / grief vector on an authed path. Fix: validateJsonSchema now threads a `refChain` of the $ref pointers resolved on the current path WITHOUT consuming data (a $ref hop, or an allOf/anyOf/oneOf branch — all re-validate the same value); a pointer reappearing on that chain fails closed with a `cyclic $ref detected` error. Data-consuming recursion (properties/items/ additionalProperties) deliberately resets the chain, because a JSON value is a finite tree so those always terminate — a legitimately recursive schema (Node→child:Node) must NOT be flagged. A REF_DEPTH_CAP backstops any threading mistake. MUST-FIX — OCP_STRUCTURED_MAX_ATTEMPTS NaN guard was broken. `Math.max(1, parseInt(env ||"3",10))` === `Math.max(1, NaN)` === NaN for a non-integer value, so the retry loop `attempt < NaN` never ran → 0 spawns, every structured request silently refused (fails closed on cost but bricks the feature and ignores the intended floor). Fix: extracted a pure fail-closed resolveMaxAttempts() into lib/structured-output.mjs — rejects NaN/non-finite/<1, keeps the documented default of 3, and warns at startup. server.mjs now derives STRUCTURED_MAX_ATTEMPTS through it. Tests: +8 (all pure-module) — a→b→a and self (a→a) cyclic $ref fail closed without overflowing the stack; a cycle routed through anyOf; a legitimate recursive Node schema is NOT flagged; resolveMaxAttempts honors valid integers, defaults on unset/empty/null, and fails closed (not NaN, not 0) on abc/0/-1/NaN/Infinity/blank with a startup warn. 368 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: vvlasy-openclaw <vvlasy@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: dtzp555 <dtzp555@gmail.com>
This commit is contained in:
openclaw.vvlasy.cz
+135
@@ -42,6 +42,7 @@ import { dirname, join } from "node:path";
|
||||
import { homedir } from "node:os";
|
||||
import { validateKey, recordUsage, getUsageByKey, getUsageTimeline, getRecentUsage, createKey, listKeys, revokeKey, closeDb, checkQuota, updateKeyQuota, getKeyQuota, findKey, cacheHash, getCachedResponse, setCachedResponse, clearCache, getCacheStats, hasCacheControl, singleflight, getInflightStats } from "./keys.mjs";
|
||||
import { DEFAULT_PORT } from "./lib/constants.mjs";
|
||||
import { StructuredOutputError, detectStructuredOutput, validateJsonSchema, extractJsonPayload, structuredSystemInstruction, resolveMaxAttempts } from "./lib/structured-output.mjs";
|
||||
import { isLoopbackBind } from "./lib/net.mjs";
|
||||
import { runTuiTurn, reapStaleTuiSessions, resolveTuiHome, bootTuiPane, tuiPaneHealthy, poolPaneName, POOL_BOOT_MS } from "./lib/tui/session.mjs";
|
||||
import { detectTuiUpstreamError } from "./lib/tui/transcript.mjs";
|
||||
@@ -329,6 +330,16 @@ const ALLOWED_TOOLS = (process.env.CLAUDE_ALLOWED_TOOLS ||
|
||||
"Bash,Read,Write,Edit,Glob,Grep,WebSearch,WebFetch,Agent"
|
||||
).split(",").map(s => s.trim()).filter(Boolean);
|
||||
const SYSTEM_PROMPT = process.env.CLAUDE_SYSTEM_PROMPT || "";
|
||||
// Max attempts (initial + retries) to coerce a valid structured-output (OpenAI response_format)
|
||||
// JSON response out of the model before rejecting. See runStructuredCompletion.
|
||||
// Fail closed on a non-numeric value via resolveMaxAttempts(): the old `Math.max(1, parseInt("abc",10))`
|
||||
// === `Math.max(1, NaN)` === NaN, which made the retry loop `attempt < NaN` never execute → 0 spawns,
|
||||
// every structured request silently refused. The helper rejects NaN/non-finite/<1 and keeps the
|
||||
// documented default of 3. (PR #153 review round 2, NaN-guard must-fix.)
|
||||
const STRUCTURED_MAX_ATTEMPTS = resolveMaxAttempts(
|
||||
process.env.OCP_STRUCTURED_MAX_ATTEMPTS,
|
||||
{ fallback: 3, warn: (m) => console.warn(`[init] ${m}`) },
|
||||
);
|
||||
const MCP_CONFIG = process.env.CLAUDE_MCP_CONFIG || "";
|
||||
let SESSION_TTL = parseInt(process.env.CLAUDE_SESSION_TTL || "3600000", 10);
|
||||
let MAX_CONCURRENT = parseInt(process.env.CLAUDE_MAX_CONCURRENT || "8", 10);
|
||||
@@ -2143,6 +2154,31 @@ function completionResponse(res, id, model, content) {
|
||||
});
|
||||
}
|
||||
|
||||
// OpenAI's designated mechanism for "the model would not produce the required output" is the
|
||||
// assistant `refusal` field (content:null, refusal:<text>, finish_reason:"stop") — NOT an invented
|
||||
// error type. Structured-output exhaustion emits this so SDK clients take their written `refusal`
|
||||
// branch instead of throwing an opaque UnprocessableEntityError. (PR #153 review, finding 3.)
|
||||
function refusalResponse(res, id, model, refusal) {
|
||||
jsonResponse(res, 200, {
|
||||
id, object: "chat.completion",
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
model,
|
||||
choices: [{ index: 0, message: { role: "assistant", content: null, refusal }, finish_reason: "stop" }],
|
||||
usage: { prompt_tokens: 0, completion_tokens: 0, total_tokens: 0 },
|
||||
});
|
||||
}
|
||||
|
||||
// Streaming form of refusalResponse: a role chunk, a `refusal` delta, then the stop chunk.
|
||||
function streamRefusalAsSSE(res, id, model, refusal) {
|
||||
const created = Math.floor(Date.now() / 1000);
|
||||
res.writeHead(200, { "Content-Type": "text/event-stream", "Cache-Control": "no-cache", "Connection": "keep-alive", "X-Accel-Buffering": "no" });
|
||||
sendSSE(res, { id, object: "chat.completion.chunk", created, model, choices: [{ index: 0, delta: { role: "assistant" }, finish_reason: null }] });
|
||||
sendSSE(res, { id, object: "chat.completion.chunk", created, model, choices: [{ index: 0, delta: { refusal }, finish_reason: null }] });
|
||||
sendSSE(res, { id, object: "chat.completion.chunk", created, model, choices: [{ index: 0, delta: {}, finish_reason: "stop" }] });
|
||||
res.write("data: [DONE]\n\n");
|
||||
res.end();
|
||||
}
|
||||
|
||||
// Replay a complete string as a chunked SSE stream (80 codepoints/chunk).
|
||||
// Used by: (a) cache-hit replay on the streaming path; (b) TUI-mode streaming
|
||||
// (buffered response replayed as SSE so clients get the same wire format).
|
||||
@@ -2655,6 +2691,37 @@ const MAX_BODY_SIZE_LABEL = `${Math.round(MAX_BODY_SIZE / (1024 * 1024))}MB`;
|
||||
// Set of all valid model identifiers (canonical IDs + aliases)
|
||||
const VALID_MODELS = new Set(Object.keys(MODEL_MAP));
|
||||
|
||||
// Drive the model to a valid structured-output (OpenAI response_format) JSON string, retrying up to
|
||||
// STRUCTURED_MAX_ATTEMPTS. Appends a strict JSON-only steering instruction, extracts + validates the
|
||||
// reply (pure helpers in lib/structured-output.mjs), and escalates the instruction on failure.
|
||||
// Returns the canonical JSON string (message.content) or throws StructuredOutputError.
|
||||
async function runStructuredCompletion(upstreamCall, model, messages, conversationId, keyName, res, structured) {
|
||||
let lastErr = "no valid JSON produced";
|
||||
let lastRaw = "";
|
||||
for (let attempt = 0; attempt < STRUCTURED_MAX_ATTEMPTS; attempt++) {
|
||||
const augmented = [...messages, { role: "system", content: structuredSystemInstruction(structured, attempt, lastErr) }];
|
||||
const raw = await upstreamCall(model, augmented, conversationId, keyName, res);
|
||||
lastRaw = raw;
|
||||
const extracted = extractJsonPayload(raw, { whole: structured.mode === "json_object" });
|
||||
if (!extracted.ok) {
|
||||
lastErr = extracted.reason || "response was not parseable as JSON";
|
||||
logEvent("warn", "structured_retry", { attempt, reason: extracted.reason || "unparseable" });
|
||||
continue;
|
||||
}
|
||||
if (structured.mode === "schema" && structured.schema) {
|
||||
const errs = validateJsonSchema(extracted.value, structured.schema, "$", structured.strict);
|
||||
if (errs.length) {
|
||||
lastErr = "schema validation failed: " + errs.slice(0, 5).join("; ");
|
||||
logEvent("warn", "structured_retry", { attempt, reason: "schema", errors: errs.slice(0, 5) });
|
||||
continue;
|
||||
}
|
||||
}
|
||||
if (attempt > 0) logEvent("info", "structured_recovered", { attempt });
|
||||
return JSON.stringify(extracted.value); // canonical, fence-free, prose-free
|
||||
}
|
||||
throw new StructuredOutputError(lastErr, lastRaw);
|
||||
}
|
||||
|
||||
async function handleChatCompletions(req, res) {
|
||||
let body = "";
|
||||
try {
|
||||
@@ -2762,6 +2829,74 @@ async function handleChatCompletions(req, res) {
|
||||
}
|
||||
}
|
||||
|
||||
// Structured output (OpenAI response_format / json_mode): its own path — the response must be
|
||||
// schema-valid JSON, so it never shares the conversational cache slot. When caching is enabled it
|
||||
// uses a structured-keyed hash (isolated via cacheHash's `structured` marker) and writes back ONLY
|
||||
// a validated result (never a 422). Always validates on a miss.
|
||||
const structured = detectStructuredOutput(parsed);
|
||||
if (structured) {
|
||||
const t0s = Date.now();
|
||||
const promptCharsS = messages.reduce((a, m) => a + contentToText(m.content).length, 0);
|
||||
let structuredHash = null;
|
||||
if (CACHE_TTL > 0 && !conversationId && !hasCacheControl(messages)) {
|
||||
structuredHash = cacheHash(model, messages, { keyId: req._authKeyId, temperature: parsed.temperature, max_tokens: parsed.max_tokens, top_p: parsed.top_p, structured });
|
||||
try {
|
||||
const cached = getCachedResponse(structuredHash, CACHE_TTL);
|
||||
if (cached) {
|
||||
logEvent("info", "cache_hit", { model, hash: structuredHash.slice(0, 12), hits: cached.hits, structured: true });
|
||||
const id = `chatcmpl-${randomUUID()}`;
|
||||
if (stream) streamStringAsSSE(res, id, model, cached.response);
|
||||
else completionResponse(res, id, model, cached.response);
|
||||
return;
|
||||
}
|
||||
} catch (e) { logEvent("error", "cache_check_failed", { error: e.message }); }
|
||||
}
|
||||
const upstreamCall = TUI_MODE ? callClaudeTui : callClaude;
|
||||
// Stampede protection (PR #153 review, finding 5): a structured request can cost up to
|
||||
// STRUCTURED_MAX_ATTEMPTS metered spawns, so N identical concurrent requests (Home Assistant
|
||||
// firing several AI Tasks at once) must NOT each pay N× — they share one flight. We dedup every
|
||||
// one-off structured request (not stateful sessions / client-side prompt caching), independent of
|
||||
// whether OCP response caching is enabled; when caching IS on, the same key gates cache read/write.
|
||||
const dedupKey = (!conversationId && !hasCacheControl(messages))
|
||||
? cacheHash(model, messages, { keyId: req._authKeyId, temperature: parsed.temperature, max_tokens: parsed.max_tokens, top_p: parsed.top_p, structured })
|
||||
: null;
|
||||
const runStructured = async () => {
|
||||
const c = await runStructuredCompletion(upstreamCall, model, messages, conversationId, req._authKeyName, res, structured);
|
||||
if (structuredHash) { try { setCachedResponse(structuredHash, model, c); } catch (e) { logEvent("error", "cache_write_failed", { error: e.message }); } }
|
||||
return c;
|
||||
};
|
||||
try {
|
||||
const content = dedupKey
|
||||
? await singleflight(dedupKey, async () => {
|
||||
// A follower that raced in after the leader populated the cache re-reads it here.
|
||||
if (structuredHash) { const rc = getCachedResponse(structuredHash, CACHE_TTL); if (rc) return rc.response; }
|
||||
return runStructured();
|
||||
}, (err) => err instanceof RequestDisconnectedError && !res.destroyed)
|
||||
: await runStructured();
|
||||
const id = `chatcmpl-${randomUUID()}`;
|
||||
if (stream) streamStringAsSSE(res, id, model, content);
|
||||
else completionResponse(res, id, model, content);
|
||||
try { recordUsage({ keyId: req._authKeyId, keyName: req._authKeyName, model, promptChars: promptCharsS, responseChars: content.length, elapsedMs: Date.now() - t0s, success: true }); } catch (e) { logEvent("error", "usage_record_failed", { error: e.message }); }
|
||||
return;
|
||||
} catch (err) {
|
||||
if (err instanceof RequestDisconnectedError) { try { res.end(); } catch {} return; }
|
||||
try { recordUsage({ keyId: req._authKeyId, keyName: req._authKeyName, model, promptChars: promptCharsS, responseChars: 0, elapsedMs: Date.now() - t0s, success: false }); } catch {}
|
||||
if (res.headersSent || res.writableEnded || res.destroyed) { try { res.end(); } catch {} return; }
|
||||
if (err instanceof StructuredOutputError) {
|
||||
// OpenAI's spec mechanism for "model would not produce the required output" is the assistant
|
||||
// `refusal` field (200, content:null, finish_reason:"stop"), NOT an invented 422 error type —
|
||||
// so SDK clients take their written refusal branch. (PR #153 review, finding 3.)
|
||||
logEvent("warn", "structured_failed", { reason: err.reason });
|
||||
const id = `chatcmpl-${randomUUID()}`;
|
||||
const refusal = `Could not produce a response matching the requested response_format after ${STRUCTURED_MAX_ATTEMPTS} attempts (${sanitizeError(err.reason)}).`;
|
||||
if (stream) streamRefusalAsSSE(res, id, model, refusal);
|
||||
else refusalResponse(res, id, model, refusal);
|
||||
return;
|
||||
}
|
||||
return respondUpstreamError(res, err);
|
||||
}
|
||||
}
|
||||
|
||||
// Cache check (only when cache is enabled and no active conversation/session)
|
||||
if (CACHE_TTL > 0 && !conversationId) {
|
||||
// D2: skip OCP cache entirely when messages carry cache_control annotations;
|
||||
|
||||
Reference in New Issue
Block a user