1 Commits

Include renames

Author	SHA1	Message	Date
6 people	ac81badda1	feat(chat): forward OpenAI image_url parts to Claude (multimodal vision) (#154) ... * feat(chat): forward OpenAI image_url parts to Claude (multimodal vision) `POST /v1/chat/completions` previously flattened every message to plain text via contentToText(), replacing image_url parts with "[non-text content omitted]" — so images were silently dropped (issue #110). This adds real multimodal support: OpenAI `image_url` content parts are translated to Anthropic image blocks and fed to the Claude CLI over `--input-format stream-json`, keeping subscription auth (the reason OCP routes through the CLI rather than the API). Class B.1 (OpenAI-compatibility surface), authorized by ADR 0006. Request shape follows OpenAI's published vision / chat-completions spec (https://platform.openai.com/docs/guides/vision and the chat/completions `content` image_url part) — no field is introduced beyond OpenAI's shape. The CLI's `--input-format stream-json` is the transport for this Class B endpoint, not a forwarded cli.js operation, so there is no Class A cli.js citation to make; scope is justified under the ALIGNMENT.md Class B mapping of Rule 2 (no invention beyond the cited OpenAI spec). Mechanism (verified empirically against the installed CLI, v2.1.206): a user message whose `content` is an Anthropic block array including `{type:"image",source:{type:"base64",media_type,data}}` fed to `claude -p --input-format stream-json` is correctly described by the model. Confirmed live end-to-end through this endpoint (a base64 PNG returns the correct color). Design: - New pure module `lib/multimodal.mjs` (mirrors the lib/*.mjs pattern; unit- testable without a live server): hasImageContent, buildImageBlocks, buildStreamJsonInput, MultimodalError. - server.mjs: text path is byte-for-byte unchanged. Only when a request carries an image_url part does spawnClaudeProcess switch stdin to a stream-json user envelope and buildCliArgs add `--input-format stream-json`. Image parsing runs before any stats mutation so a validation failure never leaks counters/slots. - Images bypass the text char budget (CLAUDE_MAX_PROMPT_CHARS) and are bounded by explicit byte/count caps with clear 4xx errors (413 for size/count, 400 for malformed/unsupported/disabled-remote), never a silent drop. Scope decisions (v1): - Base64 data URIs supported by default (image/jpeg,png,gif,webp). - Remote http(s) image URLs OFF by default behind CLAUDE_IMAGE_ALLOW_URL; when enabled they are passed through as an Anthropic url-source (OCP never fetches the URL itself, so no OCP-side SSRF surface). - Audio/file parts deferred: existing placeholder behavior preserved. - Images anywhere in multi-turn history, not just the last message. New env vars (documented in README Environment Variables table): CLAUDE_IMAGE_ALLOW_URL, CLAUDE_MAX_IMAGE_BYTES, CLAUDE_MAX_IMAGES, CLAUDE_MAX_IMAGE_TOTAL_BYTES, CLAUDE_MAX_BODY_SIZE (now configurable; default 5 MB unchanged). Tests: 26 unit tests in test-features.mjs covering data-URI parse, multiple images, text/image ordering, multi-turn history images, malformed/oversized/ too-many handling, remote-URL policy, and text-path parity. `npm test` green (289 passed). `node --check` clean. No alignment-blacklist tokens added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chat): address PR #154 review blockers — TUI guard, fail-closed caps, text budget Remediates the three merge-blocking findings from the maintainer's review of the multimodal vision PR. Class B.1 (OpenAI-compat surface): request shape per OpenAI vision spec (image_url content parts), authorized by ADR 0006. No new wire shape and no cli.js surface change — the stream-json image contract is the CLI's native input format (already cited in the base commit); these are correctness fixes on the OCP-owned validation/dispatch layer. F1 — TUI mode silently dropped images and returned 200. callClaudeTui() renders every non-text part as "[non-text content omitted]", so a vision request in CLAUDE_TUI_MODE=true was answered about an image the model never saw. Now handleChatCompletions fails loudly with 400 images_unsupported_in_tui_mode instead of a silent drop (ALIGNMENT.md forbids serving text the model did not mean). Documented in README § Images / Multimodal. F3 — NaN env parsing failed open. CLAUDE_MAX_BODY_SIZE=unlimited -> NaN -> `body.length > NaN` always false -> body cap gone (OOM DoS); =5MB -> 5 bytes -> proxy bricked; same on CLAUDE_MAX_IMAGES / _IMAGE_BYTES / _IMAGE_TOTAL_BYTES. Added lib/env.mjs parsePositiveInt (pure, fail-closed) + a thin parseIntEnv warn wrapper; a malformed cap now keeps the safe default and warns at startup. F2 — images let unbounded text bypass MAX_PROMPT_CHARS. buildImageBlocks only counted textChars and never truncated; the budget was never passed in. Threaded maxTextChars (= MAX_PROMPT_CHARS) into the multimodal transform, which now truncates text tail-first (mirroring messagesToPrompt) while preserving image blocks, and logs prompt_truncated. Tests: +11 in test-features.mjs (all pure-module, per the repo's no-server-import pattern) covering the text-budget enforcement and the fail-closed cap parsing, including the exact F2 (500k chars + 1 image) and F3 (unlimited/5MB/0/20.5) scenarios. 300 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(server): address PR #154 review round 2 — MAX_PROMPT_CHARS fail-closed + system-only image guard Closes the two residual gaps from the round-2 review, both traced to the same root as the already-fixed blockers. Class B.1 (OpenAI-compat vision): authorized by ADR 0006; request shape is the OpenAI `image_url` content part. No new wire shape — the Anthropic image block over `--input-format stream-json` is the CLI's native contract (cli.js buildStreamJsonInput path, verified live in round 1). MAX_PROMPT_CHARS is OCP's own truncation guard, not a cli.js operation. Gap (a) — MAX_PROMPT_CHARS was left on the raw parseInt while every other cap moved to the fail-closed helper. `let MAX_PROMPT_CHARS = parseInt(env||"150000",10)` sat five lines above the parseIntEnv helper this PR added, so CLAUDE_MAX_PROMPT_CHARS=unlimited → NaN → enforceTextBudget's `!(NaN > 0)` early-return → 500k chars passed unbounded, truncated:false, silently defeating F2's text-budget guarantee under a plausible operator config. Fix: hoist parseIntEnv above the declaration and derive MAX_PROMPT_CHARS through it (keeps `let` for the settings API). A misconfigured value now keeps the 150k default and warns, like the other caps. Gap (b) — an image present ONLY in a system message silently dropped in non-TUI mode. Detection runs on the full message list, but extraction/spawn filter role==="system" out, so a system-only image was detected as multimodal, survived no filter, fell to the text path, and rendered as "[non-text content omitted]" → 200 with a hallucinated answer — the one silent-drop outcome F1 exists to forbid. Fix: after filtering, if hasImageContent(full) is true but no image survives, return `400 images_unsupported_in_system_messages`. Narrow (OpenAI disallows images in the system role) so no legitimate request is rejected. Documented in README § Images. Tests: +4 (all pure-module) — parsePositiveInt('unlimited') keeps the 150k default (gap a) and a valid override is honored; hasImageContent proves the guard predicate fires for a system-only image (true on full list, false after the system filter) and does NOT fire for a user-message image. 304 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: vvlasy-openclaw <vvlasy-openclaw@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: vvlasy-openclaw <vvlasy@gmail.com> Co-authored-by: dtzp555 <dtzp555@gmail.com>	2026-07-20 07:50:17 +10:00