From ac81badda12a2267e7177b09b65b16d831425032 Mon Sep 17 00:00:00 2001 From: "openclaw.vvlasy.cz" Date: Sun, 19 Jul 2026 23:50:17 +0200 Subject: [PATCH] feat(chat): forward OpenAI image_url parts to Claude (multimodal vision) (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(chat): forward OpenAI image_url parts to Claude (multimodal vision) `POST /v1/chat/completions` previously flattened every message to plain text via contentToText(), replacing image_url parts with "[non-text content omitted]" — so images were silently dropped (issue #110). This adds real multimodal support: OpenAI `image_url` content parts are translated to Anthropic image blocks and fed to the Claude CLI over `--input-format stream-json`, keeping subscription auth (the reason OCP routes through the CLI rather than the API). Class B.1 (OpenAI-compatibility surface), authorized by ADR 0006. Request shape follows OpenAI's published vision / chat-completions spec (https://platform.openai.com/docs/guides/vision and the chat/completions `content` image_url part) — no field is introduced beyond OpenAI's shape. The CLI's `--input-format stream-json` is the transport for this Class B endpoint, not a forwarded cli.js operation, so there is no Class A cli.js citation to make; scope is justified under the ALIGNMENT.md Class B mapping of Rule 2 (no invention beyond the cited OpenAI spec). Mechanism (verified empirically against the installed CLI, v2.1.206): a user message whose `content` is an Anthropic block array including `{type:"image",source:{type:"base64",media_type,data}}` fed to `claude -p --input-format stream-json` is correctly described by the model. Confirmed live end-to-end through this endpoint (a base64 PNG returns the correct color). Design: - New pure module `lib/multimodal.mjs` (mirrors the lib/*.mjs pattern; unit- testable without a live server): hasImageContent, buildImageBlocks, buildStreamJsonInput, MultimodalError. - server.mjs: text path is byte-for-byte unchanged. Only when a request carries an image_url part does spawnClaudeProcess switch stdin to a stream-json user envelope and buildCliArgs add `--input-format stream-json`. Image parsing runs before any stats mutation so a validation failure never leaks counters/slots. - Images bypass the text char budget (CLAUDE_MAX_PROMPT_CHARS) and are bounded by explicit byte/count caps with clear 4xx errors (413 for size/count, 400 for malformed/unsupported/disabled-remote), never a silent drop. Scope decisions (v1): - Base64 data URIs supported by default (image/jpeg,png,gif,webp). - Remote http(s) image URLs OFF by default behind CLAUDE_IMAGE_ALLOW_URL; when enabled they are passed through as an Anthropic url-source (OCP never fetches the URL itself, so no OCP-side SSRF surface). - Audio/file parts deferred: existing placeholder behavior preserved. - Images anywhere in multi-turn history, not just the last message. New env vars (documented in README Environment Variables table): CLAUDE_IMAGE_ALLOW_URL, CLAUDE_MAX_IMAGE_BYTES, CLAUDE_MAX_IMAGES, CLAUDE_MAX_IMAGE_TOTAL_BYTES, CLAUDE_MAX_BODY_SIZE (now configurable; default 5 MB unchanged). Tests: 26 unit tests in test-features.mjs covering data-URI parse, multiple images, text/image ordering, multi-turn history images, malformed/oversized/ too-many handling, remote-URL policy, and text-path parity. `npm test` green (289 passed). `node --check` clean. No alignment-blacklist tokens added. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(chat): address PR #154 review blockers — TUI guard, fail-closed caps, text budget Remediates the three merge-blocking findings from the maintainer's review of the multimodal vision PR. Class B.1 (OpenAI-compat surface): request shape per OpenAI vision spec (image_url content parts), authorized by ADR 0006. No new wire shape and no cli.js surface change — the stream-json image contract is the CLI's native input format (already cited in the base commit); these are correctness fixes on the OCP-owned validation/dispatch layer. F1 — TUI mode silently dropped images and returned 200. callClaudeTui() renders every non-text part as "[non-text content omitted]", so a vision request in CLAUDE_TUI_MODE=true was answered about an image the model never saw. Now handleChatCompletions fails loudly with 400 images_unsupported_in_tui_mode instead of a silent drop (ALIGNMENT.md forbids serving text the model did not mean). Documented in README § Images / Multimodal. F3 — NaN env parsing failed open. CLAUDE_MAX_BODY_SIZE=unlimited -> NaN -> `body.length > NaN` always false -> body cap gone (OOM DoS); =5MB -> 5 bytes -> proxy bricked; same on CLAUDE_MAX_IMAGES / _IMAGE_BYTES / _IMAGE_TOTAL_BYTES. Added lib/env.mjs parsePositiveInt (pure, fail-closed) + a thin parseIntEnv warn wrapper; a malformed cap now keeps the safe default and warns at startup. F2 — images let unbounded text bypass MAX_PROMPT_CHARS. buildImageBlocks only counted textChars and never truncated; the budget was never passed in. Threaded maxTextChars (= MAX_PROMPT_CHARS) into the multimodal transform, which now truncates text tail-first (mirroring messagesToPrompt) while preserving image blocks, and logs prompt_truncated. Tests: +11 in test-features.mjs (all pure-module, per the repo's no-server-import pattern) covering the text-budget enforcement and the fail-closed cap parsing, including the exact F2 (500k chars + 1 image) and F3 (unlimited/5MB/0/20.5) scenarios. 300 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(server): address PR #154 review round 2 — MAX_PROMPT_CHARS fail-closed + system-only image guard Closes the two residual gaps from the round-2 review, both traced to the same root as the already-fixed blockers. Class B.1 (OpenAI-compat vision): authorized by ADR 0006; request shape is the OpenAI `image_url` content part. No new wire shape — the Anthropic image block over `--input-format stream-json` is the CLI's native contract (cli.js buildStreamJsonInput path, verified live in round 1). MAX_PROMPT_CHARS is OCP's own truncation guard, not a cli.js operation. Gap (a) — MAX_PROMPT_CHARS was left on the raw parseInt while every other cap moved to the fail-closed helper. `let MAX_PROMPT_CHARS = parseInt(env||"150000",10)` sat five lines above the parseIntEnv helper this PR added, so CLAUDE_MAX_PROMPT_CHARS=unlimited → NaN → enforceTextBudget's `!(NaN > 0)` early-return → 500k chars passed unbounded, truncated:false, silently defeating F2's text-budget guarantee under a plausible operator config. Fix: hoist parseIntEnv above the declaration and derive MAX_PROMPT_CHARS through it (keeps `let` for the settings API). A misconfigured value now keeps the 150k default and warns, like the other caps. Gap (b) — an image present ONLY in a system message silently dropped in non-TUI mode. Detection runs on the full message list, but extraction/spawn filter role==="system" out, so a system-only image was detected as multimodal, survived no filter, fell to the text path, and rendered as "[non-text content omitted]" → 200 with a hallucinated answer — the one silent-drop outcome F1 exists to forbid. Fix: after filtering, if hasImageContent(full) is true but no image survives, return `400 images_unsupported_in_system_messages`. Narrow (OpenAI disallows images in the system role) so no legitimate request is rejected. Documented in README § Images. Tests: +4 (all pure-module) — parsePositiveInt('unlimited') keeps the 150k default (gap a) and a valid override is honored; hasImageContent proves the guard predicate fires for a system-only image (true on full list, false after the system filter) and does NOT fire for a user-message image. 304 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: vvlasy-openclaw Co-authored-by: Claude Opus 4.8 (1M context) Co-authored-by: vvlasy-openclaw Co-authored-by: dtzp555 --- README.md | 94 ++++++++++++- lib/env.mjs | 29 ++++ lib/multimodal.mjs | 278 +++++++++++++++++++++++++++++++++++++ server.mjs | 174 ++++++++++++++++++++---- test-features.mjs | 331 +++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 878 insertions(+), 28 deletions(-) create mode 100644 lib/env.mjs create mode 100644 lib/multimodal.mjs diff --git a/README.md b/README.md index f03c9a2..a4e47a9 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ One proxy. Multiple IDEs. All models. **$0 API cost.** - [How It Works](#how-it-works) - Reference: [Available Models](#available-models) · [API Endpoints](#api-endpoints) · [Environment Variables](#environment-variables) - Modes & operations: [LAN & multi-user](#lan--multi-user) → [`docs/lan-mode.md`](docs/lan-mode.md) · [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) → [`docs/tui-mode.md`](docs/tui-mode.md) · [Upgrading](#upgrading) → [`docs/upgrading.md`](docs/upgrading.md) -- [Built-in Usage Monitoring](#built-in-usage-monitoring) · [Response Cache](#response-cache) · [OpenClaw Integration](#openclaw-integration) +- [Built-in Usage Monitoring](#built-in-usage-monitoring) · [Response Cache](#response-cache) · [Images / Multimodal](#images--multimodal-vision) · [OpenClaw Integration](#openclaw-integration) - [Troubleshooting](#troubleshooting) → [`docs/troubleshooting.md`](docs/troubleshooting.md) - [Repository Layout](#repository-layout) · [Security](#security) · [Governance](#governance) · [Support OCP](#support-ocp) · [License](#license) @@ -222,12 +222,17 @@ The canonical list lives in [`models.json`](./models.json) — the single source | `CLAUDE_MAX_CONCURRENT` | `8` | Max concurrent claude processes (`-p`/stream-json path) | | `CLAUDE_MAX_QUEUE` | `16` | Max requests **waiting** for a `-p` concurrency slot. Beyond `CLAUDE_MAX_CONCURRENT`, requests queue (up to this cap) instead of being rejected; when the queue is **also** full, the request gets `HTTP 429` + `Retry-After` (not an opaque 500). Surfaced on `/health.concurrency` + `/health.stats.queueRejections`. | | `CLAUDE_QUEUE_RETRY_AFTER` | `5` | Seconds advertised in the `Retry-After` header on a `-p` concurrency-overflow `429`. | -| `CLAUDE_MAX_PROMPT_CHARS` | *(derived)* | Prompt truncation limit in chars. Default derives from the models.json SPOT: `max(contextWindow) × 3` — currently **600,000** (≈150–200k tokens). Setting this env var (or the runtime settings API) overrides the derivation absolutely. See [ADR 0009](docs/adr/0009-spot-derived-prompt-budget.md). Note: very large prompts burn subscription-window quota quickly and slow TTFT; the TUI-mode paste path is untested beyond ~hundreds of KB. | +| `CLAUDE_MAX_PROMPT_CHARS` | *(derived)* | Prompt truncation limit in chars. Default derives from the models.json SPOT: `max(contextWindow) × 3` — currently **600,000** (≈150–200k tokens). Setting this env var (or the runtime settings API) overrides the derivation absolutely. See [ADR 0009](docs/adr/0009-spot-derived-prompt-budget.md). Note: very large prompts burn subscription-window quota quickly and slow TTFT; the TUI-mode paste path is untested beyond ~hundreds of KB. Applies to **text only** — image bytes bypass this budget (see [Images / Multimodal](#images--multimodal-vision)). | | `CLAUDE_SESSION_TTL` | `3600000` | Session expiry (ms, default: 1 hour) | | `CLAUDE_CACHE_TTL` | `0` | Response cache TTL (ms, 0 = disabled). Set to e.g. `300000` for 5-min cache. See [Response Cache](#response-cache). | | `CLAUDE_ALLOWED_TOOLS` | `Bash,Read,...,Agent` | Comma-separated tools to pre-approve | | `CLAUDE_SKIP_PERMISSIONS` | `false` | Bypass all permission checks | | `CLAUDE_MCP_CONFIG` | *(unset)* | Path to an MCP server config JSON, passed to the spawned `claude` as `--mcp-config` (both the `-p` path and TUI `OCP_TUI_FULL_TOOLS` panes) | +| `CLAUDE_MAX_BODY_SIZE` | `5242880` | Max request body size (bytes, default 5 MB). Base64 image payloads inflate ~33%; raise this to admit larger multimodal requests. Fail-closed parsing: a garbage value keeps the default. | +| `CLAUDE_IMAGE_ALLOW_URL` | `false` | Allow remote `http(s)` image URLs in `image_url` parts. **Off by default** (v1 supports base64 `data:` URIs only). When on, the URL is passed through to Anthropic as a `url` image source — **OCP does not fetch it** (no OCP-side SSRF surface); unreachable/blocked URLs surface as an API error. | +| `CLAUDE_MAX_IMAGE_BYTES` | `5242880` | Per-image decoded-byte cap (default 5 MB). Over-cap images get `HTTP 413`. | +| `CLAUDE_MAX_IMAGES` | `20` | Max image parts per request. Over-cap gets `HTTP 413`. | +| `CLAUDE_MAX_IMAGE_TOTAL_BYTES` | `20971520` | Aggregate decoded-byte cap across all images in a request (default 20 MB). Over-cap gets `HTTP 413`. | | `CLAUDE_SYSTEM_PROMPT` | *(unset)* | Operator-wide system-prompt text appended (last) to every request's composed system prompt on the default `-p` path. TUI-mode panes are unaffected (they keep the interactive CLI's own system prompt). Echoed truncated on `/health.systemPrompt`. Note: changing this value and restarting auto-invalidates the response cache (the key carries a boot-config epoch, #177). | | `CLAUDE_NO_CONTEXT` | `false` | Suppress CLAUDE.md and auto-memory injection (pure API mode) | | `PROXY_API_KEY` | *(unset)* | Bearer token for shared-mode authentication | @@ -382,6 +387,91 @@ ocp settings cacheTTL 0 # disable at runtime Cache is **disabled by default** (`CLAUDE_CACHE_TTL=0`). All data is stored locally in `~/.ocp/ocp.db`. **Hash format upgrade in v3.13.0:** legacy `v1` cache rows don't match new `v2`-format lookups; they orphan and are reaped by the TTL cleanup interval within one window — no migration script required. +## Images / Multimodal (Vision) + +`POST /v1/chat/completions` accepts OpenAI-style multimodal `content` parts, so a +message can carry images alongside text and Claude will actually see them. This +follows OpenAI's [vision](https://platform.openai.com/docs/guides/vision) / +[chat-completions `image_url`](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages) +request shape — no OCP-invented fields. (Class B.1 endpoint; see ADR 0006.) + +Under the hood, when a request carries an image OCP feeds the conversation to the +Claude CLI as Anthropic image blocks over `--input-format stream-json`. Text-only +requests are completely unaffected (unchanged code path). + +### Supported input + +- **Base64 data URIs** (default, recommended): + `data:image/png;base64,<...>`. Media types: `image/jpeg`, `image/png`, + `image/gif`, `image/webp`. +- **Remote `http(s)` URLs** — **off by default**. Set `CLAUDE_IMAGE_ALLOW_URL=1` + to enable; the URL is passed through to Anthropic (OCP never fetches it itself, + so there is no OCP-side SSRF surface). +- Images may appear in **any** message in the history (multi-turn), not just the + last one. +- Non-image, non-text parts (audio, files) are **not** yet supported and are + replaced with a `[non-text content omitted]` placeholder (deferred to a future + version). + +### Example (base64 data URI) + +```bash +curl -X POST http://127.0.0.1:3456/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "claude-sonnet-4-6", + "messages": [{ + "role": "user", + "content": [ + { "type": "text", "text": "What is in this image?" }, + { "type": "image_url", + "image_url": { "url": "data:image/png;base64,iVBORw0KGgoAAA..." } } + ] + }] + }' +``` + +### Not supported in TUI mode + +Multimodal images require the default `-p` spawn path. In **TUI / subscription-pool +mode** (`CLAUDE_TUI_MODE=true`) the CLI is driven interactively and cannot carry +image blocks, so a request with an `image_url` part returns **`400 +images_unsupported_in_tui_mode`** rather than silently dropping the image and +answering about something the model never saw. Remove the images, or run OCP +without TUI mode, to use vision. + +Images must also live in a **user or assistant** message, not a `system` message +(system content is not forwarded to the CLI as image blocks). An `image_url` part +present only in a system message returns **`400 images_unsupported_in_system_messages`** +for the same reason — fail loudly rather than answer about an unseen image. This matches +the OpenAI vision spec, which does not place images in the system role. + +### Limits + +Images bypass the text `CLAUDE_MAX_PROMPT_CHARS` budget and are instead bounded by +their own byte/count caps. The **text** in a multimodal request is still subject to +`CLAUDE_MAX_PROMPT_CHARS` (older text is truncated exactly as on the text-only +path — only the image bytes are exempt). All numeric caps are parsed **fail-closed**: +a malformed value (e.g. `CLAUDE_MAX_BODY_SIZE=unlimited` or `=5MB`) is rejected with +a startup warning and the safe default is kept — a misconfigured cap can never +silently disable the guard. Requests that violate a cap get a clear `4xx` (never a +silent drop): + +| Cap | Env var | Default | Error | +|-----|---------|---------|-------| +| Request body | `CLAUDE_MAX_BODY_SIZE` | 5 MB | `413` request body too large | +| Per-image bytes | `CLAUDE_MAX_IMAGE_BYTES` | 5 MB | `413` `image_too_large` | +| Total image bytes | `CLAUDE_MAX_IMAGE_TOTAL_BYTES` | 20 MB | `413` `images_too_large` | +| Image count | `CLAUDE_MAX_IMAGES` | 20 | `413` `too_many_images` | +| Unsupported media type | — | — | `400` `unsupported_image_type` | +| Malformed data URI | — | — | `400` `invalid_data_uri` | +| Remote URL while disabled | `CLAUDE_IMAGE_ALLOW_URL` | off | `400` `remote_url_disabled` | + +Base64 payloads are large: a 5 MB image is ~6.7 MB as a data URI, so raise +`CLAUDE_MAX_BODY_SIZE` (and, if needed, `CLAUDE_MAX_IMAGE_BYTES`) to admit big +images. Vision support depends on the target model — request a current +vision-capable Claude model. + ## OpenClaw Integration OCP was originally built for [OpenClaw](https://github.com/openclaw/openclaw) and includes deep integration: diff --git a/lib/env.mjs b/lib/env.mjs new file mode 100644 index 0000000..b4f8227 --- /dev/null +++ b/lib/env.mjs @@ -0,0 +1,29 @@ +// OCP env-var parsing helpers. +// +// Fail-closed positive-integer parsing for numeric caps (body size, image +// byte/count limits). A misconfigured cap must NEVER silently disable a guard: +// `parseInt("unlimited", 10)` is NaN and `x > NaN` is always false, so a naive +// parse of CLAUDE_MAX_BODY_SIZE=unlimited would remove the body-size limit +// entirely (unbounded body → OOM). Likewise CLAUDE_MAX_BODY_SIZE=5MB naively +// parses to 5 (bytes) and bricks the proxy. So a present-but-invalid value is +// REJECTED (default kept, caller warns), not accepted. (PR #154 review F3.) +// +// Pure (no env access, no IO) so it is unit-testable without a live server. + +// Parse `raw` as a strictly-positive base-10 integer of bytes/count (no unit +// suffix). Returns { value, ok, reason }: +// - missing/empty → { value: def, ok: true } (use default) +// - valid positive int → { value: n, ok: true } +// - anything else → { value: def, ok: false, reason } (fail closed) +// Rejects: NaN ("unlimited"), non-positive ("0", "-1"), unit-suffixed ("5MB"), +// and fractional/ambiguous ("20.5", "0x10") values — String(n) !== trimmed catches +// any input parseInt only partially consumed. +export function parsePositiveInt(raw, def) { + if (raw === undefined || raw === null || raw === "") return { value: def, ok: true }; + const trimmed = String(raw).trim(); + const n = parseInt(trimmed, 10); + if (!Number.isFinite(n) || n <= 0 || String(n) !== trimmed) { + return { value: def, ok: false, reason: "not a strictly-positive integer (bytes/count, no unit suffix)" }; + } + return { value: n, ok: true }; +} diff --git a/lib/multimodal.mjs b/lib/multimodal.mjs new file mode 100644 index 0000000..6a0a436 --- /dev/null +++ b/lib/multimodal.mjs @@ -0,0 +1,278 @@ +// OCP multimodal helpers — OpenAI `image_url` content parts → Anthropic image +// blocks fed to `claude -p --input-format stream-json`. (issue #110) +// +// Class B.1 (OpenAI-compatibility surface). Protocol authority is OpenAI's +// chat/completions spec — the multimodal `content` parts shape +// (https://platform.openai.com/docs/guides/vision and +// https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages, +// `image_url` part with `image_url.url` = data URI or http(s) URL). Authorized +// by ADR 0006. This module introduces NO field beyond OpenAI's published shape: +// the OpenAI-side vocabulary read here is `type:"image_url"` + +// `image_url:{url, detail?}`; the Anthropic-side vocabulary written here +// (`type:"image", source:{type:"base64"|"url", ...}`) is the CLI's native +// stream-json input contract, not an OCP invention. +// +// Kept as a pure module (no I/O, no network, no process state) mirroring the +// lib/*.mjs pattern so it is unit-testable without a live server. server.mjs is +// the only consumer; it owns spawning, caps configuration, and HTTP status. + +// Anthropic vision-supported image media types. A data URI whose media type is +// outside this set is rejected with a clear 4xx rather than forwarded (the API +// would reject it anyway; failing early gives a better error). +export const SUPPORTED_IMAGE_TYPES = new Set([ + "image/jpeg", + "image/png", + "image/gif", + "image/webp", +]); + +// Default caps. server.mjs overrides these from env; they live here so the +// pure transform is self-contained and testable. +export const DEFAULT_MULTIMODAL_OPTS = { + allowRemoteUrl: false, // http(s) image URLs are OFF by default (v1: data URIs only) + maxImageBytes: 5 * 1024 * 1024, // per-image decoded-byte cap + maxImages: 20, // max image parts across the whole request + maxTotalImageBytes: 20 * 1024 * 1024, // aggregate decoded-byte cap + maxTextChars: Infinity, // text-char budget (server passes MAX_PROMPT_CHARS); Infinity = no truncation +}; + +// Typed error so server.mjs can map to the right HTTP status + OpenAI-shaped +// error body. `status` is the HTTP code; `type` is the OpenAI error `type`. +export class MultimodalError extends Error { + constructor(code, status, message) { + super(message); + this.name = "MultimodalError"; + this.code = code; + this.status = status; + this.type = "invalid_request_error"; // OpenAI error `type` for 4xx client errors + } +} + +// True if any message carries an OpenAI `image_url` content part. Cheap guard so +// the byte-for-byte text path is only left when an image is genuinely present. +export function hasImageContent(messages) { + if (!Array.isArray(messages)) return false; + for (const m of messages) { + if (m && Array.isArray(m.content)) { + for (const part of m.content) { + if (part && part.type === "image_url") return true; + } + } + } + return false; +} + +// Extract the URL string from an OpenAI image_url part. Spec form is +// `{type:"image_url", image_url:{url, detail?}}`; many OpenAI-compatible clients +// also send `image_url` as a bare string. Accept both (input leniency — no new +// output field). `detail` (auto|low|high) is OpenAI-only and has no Anthropic +// analogue, so it is read-and-ignored. +function imageUrlOf(part) { + const iu = part.image_url; + if (typeof iu === "string") return iu; + if (iu && typeof iu.url === "string") return iu.url; + return null; +} + +// Parse a base64 data URI: `data:[][;base64],`. +// Returns { mediaType, data (base64), bytes (decoded size) } or throws MultimodalError. +function parseDataUri(uri) { + const comma = uri.indexOf(","); + if (comma === -1) { + throw new MultimodalError("invalid_data_uri", 400, "Malformed image data URI (no comma)."); + } + const meta = uri.slice(5, comma); // strip leading "data:" + const segs = meta.split(";"); + const mediaType = (segs[0] || "").trim().toLowerCase(); + const isBase64 = segs.slice(1).some((s) => s.trim().toLowerCase() === "base64"); + if (!isBase64) { + throw new MultimodalError("invalid_data_uri", 400, "Only base64-encoded image data URIs are supported."); + } + if (!SUPPORTED_IMAGE_TYPES.has(mediaType)) { + throw new MultimodalError( + "unsupported_image_type", + 400, + `Unsupported image media type '${mediaType || "(none)"}'. Supported: ${[...SUPPORTED_IMAGE_TYPES].join(", ")}.` + ); + } + // Strip incidental whitespace/newlines some encoders insert into data URIs. + const data = uri.slice(comma + 1).replace(/\s/g, ""); + if (data.length === 0 || !/^[A-Za-z0-9+/]+={0,2}$/.test(data)) { + throw new MultimodalError("invalid_data_uri", 400, "Image data URI payload is not valid base64."); + } + // Decoded size from base64 length (minus padding); avoids decoding the buffer + // just to measure it. + const padding = data.endsWith("==") ? 2 : data.endsWith("=") ? 1 : 0; + const bytes = Math.floor((data.length * 3) / 4) - padding; + return { mediaType, data, bytes }; +} + +// Convert a single OpenAI image_url part to an Anthropic image block, enforcing +// caps via the mutable `acc` accumulator ({ images, bytes }). Throws MultimodalError. +function imagePartToBlock(part, opts, acc) { + const url = imageUrlOf(part); + if (!url) { + throw new MultimodalError("invalid_image_url", 400, "image_url part is missing a URL."); + } + + acc.images += 1; + if (acc.images > opts.maxImages) { + throw new MultimodalError("too_many_images", 413, `Too many images in request (max ${opts.maxImages}).`); + } + + if (url.startsWith("data:")) { + const { mediaType, data, bytes } = parseDataUri(url); + if (bytes > opts.maxImageBytes) { + throw new MultimodalError("image_too_large", 413, `Image exceeds per-image size limit (${opts.maxImageBytes} bytes).`); + } + acc.bytes += bytes; + if (acc.bytes > opts.maxTotalImageBytes) { + throw new MultimodalError("images_too_large", 413, `Total image payload exceeds limit (${opts.maxTotalImageBytes} bytes).`); + } + return { type: "image", source: { type: "base64", media_type: mediaType, data } }; + } + + if (/^https?:\/\//i.test(url)) { + if (!opts.allowRemoteUrl) { + throw new MultimodalError( + "remote_url_disabled", + 400, + "Remote image URLs are disabled. Enable CLAUDE_IMAGE_ALLOW_URL=1 to allow http(s) image URLs, or pass the image as a base64 data URI." + ); + } + // Passthrough as an Anthropic url-source block. OCP does NOT fetch the URL + // itself (no OCP-side SSRF surface); the fetch is performed upstream by the + // Anthropic API. Best-effort: unreachable/blocked URLs surface as an API error. + return { type: "image", source: { type: "url", url } }; + } + + throw new MultimodalError("unsupported_url_scheme", 400, "image_url must be a base64 data URI or an http(s) URL."); +} + +// Role prefix mirrors messagesToPrompt()'s text-path labeling so a multi-turn +// conversation reads the same whether or not it carries images. System messages +// are handled by the caller via --system-prompt and never reach here. +function rolePrefix(role) { + if (role === "assistant") return "[Assistant] "; + return ""; // user / tool / anything else: verbatim, as in the text path +} + +// Build the Anthropic content-block array for a single stream-json user +// envelope. Mirrors the text path's "collapse the whole conversation into one +// turn passed via stdin" model (OCP runs stateless, full context per spawn), but +// preserves image position relative to text and keeps images out of the text +// char budget entirely. Returns { blocks, stats } or throws MultimodalError. +export function buildImageBlocks(messages, opts = {}) { + const o = { ...DEFAULT_MULTIMODAL_OPTS, ...opts }; + const blocks = []; + const acc = { images: 0, bytes: 0 }; + let textChars = 0; + let firstMessage = true; + + const pushText = (text) => { + if (!text) return; + blocks.push({ type: "text", text }); + textChars += text.length; + }; + + for (const m of messages) { + const prefix = rolePrefix(m.role); + // Separate messages with a blank line, matching messagesToPrompt's "\n\n" join. + const sep = firstMessage ? "" : "\n\n"; + firstMessage = false; + let prefixEmitted = false; + const emitPrefixWith = (t) => { + if (prefixEmitted) return t; + prefixEmitted = true; + return sep + prefix + t; + }; + + if (typeof m.content === "string") { + pushText(emitPrefixWith(m.content)); + continue; + } + if (!Array.isArray(m.content)) { + // null / object content: mirror contentToText's fallback. + const t = m.content == null ? "" : JSON.stringify(m.content); + pushText(emitPrefixWith(t)); + continue; + } + + for (const part of m.content) { + if (part && part.type === "text" && typeof part.text === "string") { + pushText(emitPrefixWith(part.text)); + } else if (part && part.type === "image_url") { + // Ensure the role prefix isn't lost when a message leads with an image. + if (!prefixEmitted && prefix) pushText(emitPrefixWith("")); + blocks.push(imagePartToBlock(part, o, acc)); + } else { + // audio / file / unknown parts: preserve the existing placeholder + // behavior (issue #110) — deferred to a future version. + pushText(emitPrefixWith("[non-text content omitted]")); + } + } + } + + // Defensive: a stream-json user turn must have at least one content block. + if (blocks.length === 0) blocks.push({ type: "text", text: "" }); + + // Enforce the text-char budget (PR #154 review F2). Without this, attaching a + // single tiny image would let unbounded text bypass the gateway's runaway-context + // guard entirely — messagesToPrompt truncates the text path, so the multimodal + // path must too. Image blocks are preserved and are NOT counted (they are bounded + // by the byte/count caps above). + const budgeted = enforceTextBudget(blocks, o.maxTextChars); + + return { + blocks: budgeted.blocks, + stats: { + imageCount: acc.images, + totalImageBytes: acc.bytes, + textChars: budgeted.textChars, + originalTextChars: budgeted.originalTextChars, + truncated: budgeted.truncated, + }, + }; +} + +// Enforce a text-char budget over already-built content blocks, mirroring +// messagesToPrompt's "keep the tail, drop the oldest" truncation. Image blocks are +// preserved in place; only text blocks count against the budget. A truncation note +// is prepended when anything is dropped. Returns +// { blocks, truncated, originalTextChars, textChars }. +function enforceTextBudget(blocks, maxTextChars) { + let originalTextChars = 0; + for (const b of blocks) if (b.type === "text") originalTextChars += b.text.length; + if (!(maxTextChars > 0) || originalTextChars <= maxTextChars) { + return { blocks, truncated: false, originalTextChars, textChars: originalTextChars }; + } + // Keep the most recent text up to the budget; trim within the boundary block and + // drop older text blocks. Non-text (image) blocks always survive, in order. + let budget = maxTextChars; + const out = []; + for (let i = blocks.length - 1; i >= 0; i--) { + const b = blocks[i]; + if (b.type !== "text") { out.unshift(b); continue; } + if (budget <= 0) continue; // older text fully dropped + if (b.text.length <= budget) { + out.unshift(b); + budget -= b.text.length; + } else { + out.unshift({ type: "text", text: b.text.slice(b.text.length - budget) }); + budget = 0; + } + } + out.unshift({ type: "text", text: "[System] Note: older text content was truncated to fit the context limit." }); + let textChars = 0; + for (const b of out) if (b.type === "text") textChars += b.text.length; + return { blocks: out, truncated: true, originalTextChars, textChars }; +} + +// Serialize the non-system conversation to a single newline-terminated +// stream-json user message for `claude -p --input-format stream-json` stdin. +// Returns { payload, stats } or throws MultimodalError. +export function buildStreamJsonInput(messages, opts = {}) { + const { blocks, stats } = buildImageBlocks(messages, opts); + const envelope = { type: "user", message: { role: "user", content: blocks } }; + return { payload: JSON.stringify(envelope) + "\n", stats }; +} diff --git a/server.mjs b/server.mjs index 112e512..a604fab 100644 --- a/server.mjs +++ b/server.mjs @@ -49,7 +49,9 @@ import { TuiSemaphore, SemaphoreAbortError, recordTuiEntrypoint, buildTuiHealthB import { TuiPanePool, resolvePoolSize, POOL_MAX_SIZE } from "./lib/tui/pool.mjs"; import { TuiDeltaAssembler, DEFAULT_HOLDBACK_CHARS, resolveStreamHoldback } from "./lib/tui/stream.mjs"; import { createSerialMutex, createTtlCache, isTokenExpiring, orderLabelsLastGoodFirst } from "./lib/spawn-auth.mjs"; -import { appendOperatorPrompt, resolvePromptCharBudget } from "./lib/prompt.mjs"; +import { hasImageContent, buildImageBlocks, buildStreamJsonInput, MultimodalError } from "./lib/multimodal.mjs"; +import { parsePositiveInt } from "./lib/env.mjs"; +import { appendOperatorPrompt, derivePromptCharBudget } from "./lib/prompt.mjs"; const __dirname = dirname(fileURLToPath(import.meta.url)); const _pkg = JSON.parse(readFileSync(join(__dirname, "package.json"), "utf8")); @@ -1100,7 +1102,7 @@ const authCheckInterval = setInterval(checkAuth, 600000); // CLAUDE_SYSTEM_PROMPT env var is absorbed into the system prompt via // extractSystemPrompt() at the caller level; APPEND_SYSTEM_PROMPT no longer used. // Note: ALLOWED_TOOLS / SKIP_PERMISSIONS / MCP_CONFIG are preserved as before. -function buildCliArgs(cliModel, systemPrompt) { +function buildCliArgs(cliModel, systemPrompt, opts = {}) { const args = [ "--model", cliModel, "--output-format", "stream-json", @@ -1109,6 +1111,14 @@ function buildCliArgs(cliModel, systemPrompt) { "--system-prompt", systemPrompt, ]; + // Multimodal path (issue #110): images are fed as Anthropic content blocks over + // a stream-json stdin stream. `--input-format stream-json` (§ --input-format, + // choices text|stream-json; realtime streaming input) is added ONLY when the + // request carries an image part; the default (text) input path is untouched. + if (opts.streamJsonInput) { + args.push("--input-format", "stream-json"); + } + // Permissions // ADR 0007 B-path: in multi-tenant mode, suppress operator-FS tools so a guest // prompt cannot drive Bash/Read/Write/Edit/etc. on the operator's filesystem. @@ -1144,17 +1154,45 @@ function buildCliArgs(cliModel, systemPrompt) { return args; } +// Thin env wrapper over parsePositiveInt (lib/env.mjs): resolve `name` from the +// environment fail-closed, warning on a present-but-invalid value. Keeps the pure +// parse in a unit-testable module. (PR #154 review F3) +function parseIntEnv(name, def) { + const { value, ok } = parsePositiveInt(process.env[name], def); + if (!ok) console.warn(`⚠ ${name}="${process.env[name]}" is not a valid positive integer (bytes/count, no unit suffix); ignoring and using default ${def}.`); + return value; +} + // ── Format messages to prompt text ────────────────────────────────────── // Truncation guard: if total chars exceed MAX_PROMPT_CHARS, keep the system // message(s) + first user message + last N messages, dropping the middle. // This prevents runaway context from gateway-side conversation accumulation. -// -// Default is SPOT-DERIVED (ADR 0009): max(models.json contextWindow) × 3 chars/token — -// currently 200000 × 3 = 600,000 chars — instead of the old hand-set 150000 (≈37.5k -// English tokens), which silently under-delivered the advertised window by ~5×. The env -// var (and the runtime settings API below) remain absolute operator overrides. If -// models.json ever advertises a bigger window, this budget follows automatically. -let MAX_PROMPT_CHARS = resolvePromptCharBudget(process.env.CLAUDE_MAX_PROMPT_CHARS, modelsConfig.models); +// Routed through parseIntEnv so a misconfigured cap fails CLOSED to the default rather than +// NaN — CLAUDE_MAX_PROMPT_CHARS=unlimited previously → NaN → enforceTextBudget's `!(NaN > 0)` +// early-return → 500k chars passed unbounded, silently defeating F2's text-budget guarantee +// (PR #154 round 2, gap (a)). The default itself is SPOT-DERIVED (ADR 0009, PR #179): +// max(models.json contextWindow) × 3 chars/token — currently 600,000 — so the two fixes +// compose: parseIntEnv guards a SET-but-garbage value, the derivation supplies the honest +// default when unset/empty. `let` is kept for the settings API. +let MAX_PROMPT_CHARS = parseIntEnv("CLAUDE_MAX_PROMPT_CHARS", derivePromptCharBudget(modelsConfig.models)); + +// ── Multimodal image caps (issue #110) ────────────────────────────────── +// OpenAI `image_url` parts are forwarded to claude as Anthropic image blocks via +// `--input-format stream-json`. Images deliberately BYPASS the text char budget +// (MAX_PROMPT_CHARS) — they are bounded by these byte/count caps instead, and by +// MAX_BODY_SIZE at the HTTP layer. Data URIs are supported by default; remote +// http(s) image URLs are OFF unless CLAUDE_IMAGE_ALLOW_URL is set (v1: data URIs +// only). See docs/adr/0006-openai-shim-scope.md (Class B.1) and README § "Images". +const IMAGE_ALLOW_URL = /^(1|true|yes|on)$/i.test(process.env.CLAUDE_IMAGE_ALLOW_URL || ""); +const MAX_IMAGE_BYTES = parseIntEnv("CLAUDE_MAX_IMAGE_BYTES", 5 * 1024 * 1024); +const MAX_IMAGES = parseIntEnv("CLAUDE_MAX_IMAGES", 20); +const MAX_IMAGE_TOTAL_BYTES = parseIntEnv("CLAUDE_MAX_IMAGE_TOTAL_BYTES", 20 * 1024 * 1024); +const MULTIMODAL_OPTS = { + allowRemoteUrl: IMAGE_ALLOW_URL, + maxImageBytes: MAX_IMAGE_BYTES, + maxImages: MAX_IMAGES, + maxTotalImageBytes: MAX_IMAGE_TOTAL_BYTES, +}; // Flatten OpenAI content (string | array of parts) to plain text for the prompt. // Array content: concatenate text parts; replace non-text parts (e.g. image_url) @@ -1246,25 +1284,51 @@ function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlo // Circuit breaker: disabled (see comment at top of breaker section) - stats.activeRequests++; - stats.totalRequests++; - // Phase 6c: always serialize full conversation via stdin (no session resume). // System messages are extracted and passed via --system-prompt; the remaining - // messages (user/assistant/tool) are serialized by messagesToPrompt. + // messages (user/assistant/tool) are serialized for stdin. const systemPrompt = extractSystemPrompt(messages); - // messagesToPrompt skips system messages now that they go via --system-prompt. - // Filter them out before calling to avoid double-injection. + // messagesToPrompt / buildStreamJsonInput skip system messages (they go via + // --system-prompt). Filter them out first to avoid double-injection. const nonSystemMessages = messages.filter(m => m.role !== "system"); - const prompt = messagesToPrompt(nonSystemMessages); - stats.oneOffRequests++; - if (conversationId) { - console.log(`[session] stateless conv=${conversationId.slice(0, 12)}... key=${keyName || "anon"} msgs=${messages.length} prompt_chars=${prompt.length}`); + // Multimodal (issue #110): when any message carries an OpenAI image_url part, + // feed the conversation as Anthropic content blocks over --input-format + // stream-json (images preserved and kept OUT of the text char budget). + // Otherwise the text path is byte-for-byte unchanged. buildStreamJsonInput may + // throw MultimodalError on an invalid/oversized image; it runs BEFORE any stats + // mutation so a validation failure never leaks counters or the concurrency slot + // (handleChatCompletions validates first, so in practice it will not throw here). + const useStreamJson = hasImageContent(nonSystemMessages); + let stdinPayload, promptChars; + if (useStreamJson) { + // Pass MAX_PROMPT_CHARS so the multimodal text is bounded by the same + // runaway-context guard as the text path (PR #154 review F2). Images bypass it. + const built = buildStreamJsonInput(nonSystemMessages, { ...MULTIMODAL_OPTS, maxTextChars: MAX_PROMPT_CHARS }); + stdinPayload = built.payload; + promptChars = built.stats.textChars; + if (built.stats.truncated) { + logEvent("warn", "prompt_truncated", { + originalChars: built.stats.originalTextChars, + maxChars: MAX_PROMPT_CHARS, + keptChars: built.stats.textChars, + path: "multimodal", + }); + } + } else { + stdinPayload = messagesToPrompt(nonSystemMessages); + promptChars = stdinPayload.length; } - const cliArgs = buildCliArgs(cliModel, systemPrompt); + stats.activeRequests++; + stats.totalRequests++; + stats.oneOffRequests++; + if (conversationId) { + console.log(`[session] stateless conv=${conversationId.slice(0, 12)}... key=${keyName || "anon"} msgs=${messages.length} prompt_chars=${promptChars}`); + } + + const cliArgs = buildCliArgs(cliModel, systemPrompt, { streamJsonInput: useStreamJson }); const env = { ...process.env }; delete env.CLAUDECODE; @@ -1350,12 +1414,13 @@ function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlo // the spawned process, NOT on the stdin Writable — it does not catch this. proc.stdin.on("error", (e) => logEvent("warn", "stdin_write_error", { error: e.message })); - // Write prompt to stdin immediately - proc.stdin.write(prompt); + // Write the serialized turn to stdin immediately. Text path: the flat prompt. + // Multimodal path: a single newline-terminated stream-json user envelope. + proc.stdin.write(stdinPayload); proc.stdin.end(); - recordModelRequest(cliModel, prompt.length); - logEvent("info", "claude_spawned", { model: cliModel, promptChars: prompt.length, timeout: TIMEOUT, tier: getModelTier(cliModel), session: conversationId ? conversationId.slice(0, 12) + "..." : "none" }); + recordModelRequest(cliModel, promptChars); + logEvent("info", "claude_spawned", { model: cliModel, promptChars, inputFormat: useStreamJson ? "stream-json" : "text", timeout: TIMEOUT, tier: getModelTier(cliModel), session: conversationId ? conversationId.slice(0, 12) + "..." : "none" }); // Single request timeout — no separate first-byte timer. // Claude tool-use causes long pauses in the token stream (30s-5min), @@ -2579,7 +2644,13 @@ async function handleSettings(req, res) { } // ── Handle chat completions ───────────────────────────────────────────── -const MAX_BODY_SIZE = 5 * 1024 * 1024; // 5 MB +// Default 5 MB, byte-for-byte unchanged unless CLAUDE_MAX_BODY_SIZE is set. Base64 +// image payloads inflate ~33%; operators enabling large images (issue #110) can +// raise this to admit bigger requests. Parsed fail-closed (PR #154 review F3): a +// bad value (`unlimited` → NaN, `5MB` → 5) must not disable the body cap or brick +// the proxy — parseIntEnv keeps the 5 MB default and warns instead. +const MAX_BODY_SIZE = parseIntEnv("CLAUDE_MAX_BODY_SIZE", 5 * 1024 * 1024); +const MAX_BODY_SIZE_LABEL = `${Math.round(MAX_BODY_SIZE / (1024 * 1024))}MB`; // Set of all valid model identifiers (canonical IDs + aliases) const VALID_MODELS = new Set(Object.keys(MODEL_MAP)); @@ -2590,7 +2661,7 @@ async function handleChatCompletions(req, res) { for await (const chunk of req) { body += chunk; if (body.length > MAX_BODY_SIZE) { - return jsonResponse(res, 413, { error: { message: "Request body too large (max 5MB)", type: "invalid_request_error" } }); + return jsonResponse(res, 413, { error: { message: `Request body too large (max ${MAX_BODY_SIZE_LABEL})`, type: "invalid_request_error" } }); } } } catch (e) { @@ -2619,6 +2690,57 @@ async function handleChatCompletions(req, res) { return jsonResponse(res, 400, { error: { message: "'messages' must be a non-empty array", type: "invalid_request_error" } }); } + // Multimodal validation (issue #110): when a request carries OpenAI `image_url` + // content parts, validate/parse them now so an invalid, unsupported, or oversized + // image returns a clean 4xx BEFORE the cache/spawn path (rather than a silent drop + // or an opaque 500). The stream-json transform itself runs at spawn time + // (spawnClaudeProcess → buildStreamJsonInput). Class B.1: authorized by ADR 0006; + // request shape per OpenAI vision spec (image_url content parts). buildImageBlocks + // validates without stringifying, so this early pass is cheap. + if (hasImageContent(messages)) { + // F1 (PR #154 review): the TUI path (callClaudeTui → messagesToPrompt) cannot + // carry image blocks — it renders every non-text part as "[non-text content + // omitted]". Forwarding here would let the model answer about an image it never + // saw and return 200, which is strictly worse than an honest error (the one + // outcome ALIGNMENT.md forbids: silently serving text the model did not mean). + // Stream-json image input requires the `claude -p` path, so in TUI_MODE we fail + // loudly instead of dropping. Documented in README § "Images". + if (TUI_MODE) { + return jsonResponse(res, 400, { + error: { + message: "Image inputs are not supported in TUI mode (CLAUDE_TUI_MODE=true). Images require the default -p spawn path; remove images or run OCP without TUI mode.", + type: "invalid_request_error", + code: "images_unsupported_in_tui_mode", + }, + }); + } + // Detection runs on the FULL message list, but extraction/spawn drop system messages + // (system role carries no image blocks to the CLI). So an image present ONLY in a + // system message would be detected as multimodal, survive no filter, fall to the text + // path, and render as "[non-text content omitted]" → 200 with a hallucinated answer — + // the exact silent-drop this guard exists to forbid. Fail loudly instead. OpenAI + // disallows images in the system role anyway, so no legitimate request is rejected. + // (PR #154 review round 2, gap (b).) + const nonSystem = messages.filter(m => m.role !== "system"); + if (!hasImageContent(nonSystem)) { + return jsonResponse(res, 400, { + error: { + message: "Image inputs are only supported in user/assistant messages, not in system messages. Move the image_url part to a user message.", + type: "invalid_request_error", + code: "images_unsupported_in_system_messages", + }, + }); + } + try { + buildImageBlocks(nonSystem, { ...MULTIMODAL_OPTS, maxTextChars: MAX_PROMPT_CHARS }); + } catch (e) { + if (e instanceof MultimodalError) { + return jsonResponse(res, e.status, { error: { message: e.message, type: e.type, code: e.code } }); + } + throw e; + } + } + // NOTE: quota is best-effort / eventually-consistent. The gate reads the recorded count // at entry and records only after the upstream completes, so concurrent requests at the // boundary can overshoot the cap by up to MAX_CONCURRENT, and cache hits (served before diff --git a/test-features.mjs b/test-features.mjs index 1ad17c1..791104e 100644 --- a/test-features.mjs +++ b/test-features.mjs @@ -3413,6 +3413,337 @@ test("contentToText: null returns empty string", () => { assert.equal(contentToText(null), ""); }); +// ── multimodal image transform (issue #110) ────────────────────────────────── +// OpenAI image_url parts → Anthropic image blocks for `claude -p --input-format +// stream-json`. lib/multimodal.mjs is a PURE module (no server.listen()), so it is +// imported directly here. Class B.1: shape per OpenAI vision spec, authorized by +// ADR 0006. Mechanism verified live: a base64 PNG fed as an Anthropic image block +// via --input-format stream-json is correctly described by the model. +import { + hasImageContent as mmHasImageContent, + buildImageBlocks as mmBuildImageBlocks, + buildStreamJsonInput as mmBuildStreamJsonInput, + MultimodalError as MmError, + SUPPORTED_IMAGE_TYPES as MM_SUPPORTED, +} from "./lib/multimodal.mjs"; +import { parsePositiveInt } from "./lib/env.mjs"; + +console.log("\nmultimodal image transform (issue #110):"); + +// A short, valid base64 string (charset-valid; not decoded by the transform). +const MM_B64 = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAAAAAA6fptVAAAACklEQVR4nGP4DwABAQEAG7buVgAAAABJRU5ErkJggg=="; +const dataUri = (mt = "image/png") => `data:${mt};base64,${MM_B64}`; +const imgPart = (mt) => ({ type: "image_url", image_url: { url: dataUri(mt) } }); +const txtPart = (t) => ({ type: "text", text: t }); + +test("hasImageContent: plain string message → false (text path preserved)", () => { + assert.equal(mmHasImageContent([{ role: "user", content: "hello" }]), false); +}); + +test("hasImageContent: array of text-only parts → false", () => { + assert.equal(mmHasImageContent([{ role: "user", content: [txtPart("a"), txtPart("b")] }]), false); +}); + +test("hasImageContent: message with an image_url part → true", () => { + assert.equal(mmHasImageContent([{ role: "user", content: [txtPart("q"), imgPart()] }]), true); +}); + +test("hasImageContent: image anywhere in history (not just last) → true", () => { + const msgs = [ + { role: "user", content: [txtPart("look"), imgPart()] }, + { role: "assistant", content: "ok" }, + { role: "user", content: "and now?" }, + ]; + assert.equal(mmHasImageContent(msgs), true); +}); + +// ── PR #154 review round 2, gap (b): image ONLY in a system message must not silently drop ── +// The handler detects multimodal on the FULL list but extraction/spawn filter system messages out. +// The guard fires exactly when the full list has an image but the non-system list does not — proven +// here against the same predicate the guard uses, so a system-only image is rejected (400) rather +// than falling to the text path and returning a 200 hallucinated answer. +test("hasImageContent: image ONLY in a system message → true on full list, false after system filter (guard fires)", () => { + const msgs = [ + { role: "system", content: [txtPart("context"), imgPart()] }, + { role: "user", content: "describe it" }, + ]; + assert.equal(mmHasImageContent(msgs), true, "detected as multimodal on the full list"); + assert.equal(mmHasImageContent(msgs.filter(m => m.role !== "system")), false, "no image survives the system filter → guard must 400"); +}); +test("hasImageContent: image in a USER message survives the system filter (legitimate request not rejected)", () => { + const msgs = [ + { role: "system", content: "you are helpful" }, + { role: "user", content: [txtPart("describe it"), imgPart()] }, + ]; + assert.equal(mmHasImageContent(msgs.filter(m => m.role !== "system")), true, "user image survives → normal multimodal path"); +}); + +test("buildImageBlocks: data-URI parsed into an Anthropic base64 image block", () => { + const { blocks, stats } = mmBuildImageBlocks([{ role: "user", content: [txtPart("what is this?"), imgPart("image/png")] }]); + assert.equal(blocks.length, 2); + assert.equal(blocks[0].type, "text"); + assert.equal(blocks[0].text, "what is this?"); + assert.deepEqual(blocks[1], { type: "image", source: { type: "base64", media_type: "image/png", data: MM_B64 } }); + assert.equal(stats.imageCount, 1); + assert.ok(stats.totalImageBytes > 0); +}); + +test("buildImageBlocks: media_type carried through (jpeg/gif/webp)", () => { + for (const mt of ["image/jpeg", "image/gif", "image/webp"]) { + const { blocks } = mmBuildImageBlocks([{ role: "user", content: [imgPart(mt)] }]); + assert.equal(blocks.find(b => b.type === "image").source.media_type, mt); + } +}); + +test("buildImageBlocks: multiple images in one message both emitted", () => { + const { blocks, stats } = mmBuildImageBlocks([{ role: "user", content: [txtPart("compare"), imgPart(), imgPart()] }]); + const imgs = blocks.filter(b => b.type === "image"); + assert.equal(imgs.length, 2); + assert.equal(stats.imageCount, 2); +}); + +test("buildImageBlocks: text/image/text ordering preserved", () => { + const { blocks } = mmBuildImageBlocks([{ role: "user", content: [txtPart("A"), imgPart(), txtPart("B")] }]); + assert.deepEqual(blocks.map(b => (b.type === "text" ? b.text : "IMG")), ["A", "IMG", "B"]); +}); + +test("buildImageBlocks: image-first message keeps ordering (image before text)", () => { + const { blocks } = mmBuildImageBlocks([{ role: "user", content: [imgPart(), txtPart("caption")] }]); + assert.deepEqual(blocks.map(b => (b.type === "text" ? b.text : "IMG")), ["IMG", "caption"]); +}); + +test("buildImageBlocks: multi-turn history — role prefixes + separators preserved", () => { + const msgs = [ + { role: "user", content: "first q" }, + { role: "assistant", content: "prior answer" }, + { role: "user", content: [txtPart("now this"), imgPart()] }, + ]; + const { blocks } = mmBuildImageBlocks(msgs); + assert.equal(blocks[0].text, "first q"); + assert.equal(blocks[1].text, "\n\n[Assistant] prior answer"); + assert.equal(blocks[2].text, "\n\nnow this"); + assert.equal(blocks[3].type, "image"); +}); + +test("buildImageBlocks: image in an EARLIER turn is carried (history image)", () => { + const msgs = [ + { role: "user", content: [txtPart("here"), imgPart()] }, + { role: "assistant", content: "got it" }, + { role: "user", content: "thanks" }, + ]; + const { blocks, stats } = mmBuildImageBlocks(msgs); + assert.equal(stats.imageCount, 1); + assert.equal(blocks.filter(b => b.type === "image").length, 1); +}); + +test("buildImageBlocks: image_url as bare string is accepted (client leniency)", () => { + const { blocks } = mmBuildImageBlocks([{ role: "user", content: [{ type: "image_url", image_url: dataUri() }] }]); + assert.equal(blocks.find(b => b.type === "image").source.data, MM_B64); +}); + +test("buildStreamJsonInput: emits one newline-terminated user envelope", () => { + const { payload } = mmBuildStreamJsonInput([{ role: "user", content: [txtPart("hi"), imgPart()] }]); + assert.ok(payload.endsWith("\n")); + const env = JSON.parse(payload.trim()); + assert.equal(env.type, "user"); + assert.equal(env.message.role, "user"); + assert.equal(env.message.content[1].type, "image"); +}); + +// ── malformed / policy / oversized handling (clean 4xx, never a silent drop) ── +test("buildImageBlocks: unsupported media type → 400 unsupported_image_type", () => { + assert.throws( + () => mmBuildImageBlocks([{ role: "user", content: [imgPart("image/tiff")] }]), + (e) => e instanceof MmError && e.code === "unsupported_image_type" && e.status === 400 + ); +}); + +test("buildImageBlocks: non-base64 data URI → 400 invalid_data_uri", () => { + assert.throws( + () => mmBuildImageBlocks([{ role: "user", content: [{ type: "image_url", image_url: { url: "data:image/png,notbase64" } }] }]), + (e) => e instanceof MmError && e.code === "invalid_data_uri" && e.status === 400 + ); +}); + +test("buildImageBlocks: malformed data URI (no comma) → 400 invalid_data_uri", () => { + assert.throws( + () => mmBuildImageBlocks([{ role: "user", content: [{ type: "image_url", image_url: { url: "data:image/png;base64" } }] }]), + (e) => e instanceof MmError && e.code === "invalid_data_uri" + ); +}); + +test("buildImageBlocks: image_url part missing a URL → 400 invalid_image_url", () => { + assert.throws( + () => mmBuildImageBlocks([{ role: "user", content: [{ type: "image_url", image_url: {} }] }]), + (e) => e instanceof MmError && e.code === "invalid_image_url" + ); +}); + +test("buildImageBlocks: oversized single image → 413 image_too_large", () => { + assert.throws( + () => mmBuildImageBlocks([{ role: "user", content: [imgPart()] }], { maxImageBytes: 4 }), + (e) => e instanceof MmError && e.code === "image_too_large" && e.status === 413 + ); +}); + +test("buildImageBlocks: too many images → 413 too_many_images", () => { + const many = Array.from({ length: 3 }, () => imgPart()); + assert.throws( + () => mmBuildImageBlocks([{ role: "user", content: many }], { maxImages: 2 }), + (e) => e instanceof MmError && e.code === "too_many_images" && e.status === 413 + ); +}); + +test("buildImageBlocks: aggregate image bytes over cap → 413 images_too_large", () => { + assert.throws( + () => mmBuildImageBlocks([{ role: "user", content: [imgPart(), imgPart()] }], { maxTotalImageBytes: 100, maxImageBytes: 1000 }), + (e) => e instanceof MmError && e.code === "images_too_large" && e.status === 413 + ); +}); + +test("buildImageBlocks: remote http(s) URL disabled by default → 400 remote_url_disabled", () => { + assert.throws( + () => mmBuildImageBlocks([{ role: "user", content: [{ type: "image_url", image_url: { url: "https://example.com/a.png" } }] }]), + (e) => e instanceof MmError && e.code === "remote_url_disabled" && e.status === 400 + ); +}); + +test("buildImageBlocks: remote URL passthrough when allowRemoteUrl=true (url source, OCP does not fetch)", () => { + const { blocks } = mmBuildImageBlocks( + [{ role: "user", content: [{ type: "image_url", image_url: { url: "https://example.com/a.png" } }] }], + { allowRemoteUrl: true } + ); + assert.deepEqual(blocks.find(b => b.type === "image").source, { type: "url", url: "https://example.com/a.png" }); +}); + +test("buildImageBlocks: unsupported URL scheme → 400 unsupported_url_scheme", () => { + assert.throws( + () => mmBuildImageBlocks([{ role: "user", content: [{ type: "image_url", image_url: { url: "ftp://x/y.png" } }] }], { allowRemoteUrl: true }), + (e) => e instanceof MmError && e.code === "unsupported_url_scheme" + ); +}); + +test("buildImageBlocks: non-image parts (audio/file) fall back to placeholder text", () => { + const { blocks } = mmBuildImageBlocks([{ role: "user", content: [txtPart("hear this"), { type: "input_audio", input_audio: {} }] }]); + assert.deepEqual(blocks.map(b => b.text), ["hear this", "[non-text content omitted]"]); +}); + +test("SUPPORTED_IMAGE_TYPES: exactly the four Anthropic vision types", () => { + assert.deepEqual([...MM_SUPPORTED].sort(), ["image/gif", "image/jpeg", "image/png", "image/webp"]); +}); + +test("buildImageBlocks: pure-text conversation still yields text blocks (untouched-path parity)", () => { + // hasImageContent would be false for this input in server.mjs (text path taken), + // but the transform must still be well-defined for a text-only turn. + const { blocks, stats } = mmBuildImageBlocks([{ role: "user", content: "just text" }]); + assert.deepEqual(blocks, [{ type: "text", text: "just text" }]); + assert.equal(stats.imageCount, 0); + assert.equal(stats.truncated, false); +}); + +// ── F2 (PR #154 review): text char budget is enforced on the multimodal path ── +// Regression guard: without maxTextChars, attaching one tiny image let unbounded +// text bypass MAX_PROMPT_CHARS entirely (the text path truncates; the image path +// did not). server.mjs passes maxTextChars: MAX_PROMPT_CHARS into this transform. +console.log("\nmultimodal text-budget enforcement (PR #154 F2):"); + +test("buildImageBlocks: text under budget → not truncated, blocks unchanged", () => { + const { blocks, stats } = mmBuildImageBlocks( + [{ role: "user", content: [txtPart("short"), imgPart()] }], + { maxTextChars: 1000 } + ); + assert.equal(stats.truncated, false); + assert.equal(stats.textChars, "short".length); + assert.equal(blocks.filter(b => b.type === "image").length, 1); +}); + +test("buildImageBlocks: text over budget → truncated, keeps most-recent tail + note", () => { + const big = "A".repeat(300) + "TAIL_MARKER"; + const { blocks, stats } = mmBuildImageBlocks( + [{ role: "user", content: [txtPart(big)] }], + { maxTextChars: 50 } + ); + assert.equal(stats.truncated, true); + assert.equal(stats.originalTextChars, big.length); + // The most recent characters (the tail) survive; the oldest 'A's are dropped. + const joined = blocks.filter(b => b.type === "text").map(b => b.text).join(""); + assert.ok(joined.includes("TAIL_MARKER"), "tail text must be kept"); + assert.ok(joined.includes("truncated to fit"), "a truncation note must be present"); + assert.ok(stats.originalTextChars > stats.textChars, "post-truncation text is smaller"); +}); + +test("buildImageBlocks: F2 exact scenario — 500k chars + one image → text bounded, image preserved", () => { + const { blocks, stats } = mmBuildImageBlocks( + [{ role: "user", content: [txtPart("Z".repeat(500000)), imgPart()] }], + { maxTextChars: 150000 } + ); + assert.equal(stats.truncated, true); + assert.ok(stats.textChars <= 150000 + 200, "text char count is bounded by the budget (+note)"); + // The image bypasses the text budget and is NOT dropped by truncation. + assert.equal(blocks.filter(b => b.type === "image").length, 1); +}); + +test("buildImageBlocks: default (no maxTextChars) never truncates — pure module standalone", () => { + const { stats } = mmBuildImageBlocks([{ role: "user", content: [txtPart("x".repeat(10000))] }]); + assert.equal(stats.truncated, false); + assert.equal(stats.textChars, 10000); +}); + +// ── F3 (PR #154 review): fail-closed positive-int env parsing ──────────────── +// A misconfigured numeric cap must NEVER silently disable a guard (`x > NaN` is +// always false) or brick the proxy with a nonsense value. parsePositiveInt keeps +// the default and reports ok:false so the caller can warn. +console.log("\nfail-closed env-cap parsing (PR #154 F3):"); + +test("parsePositiveInt: missing/empty → default, ok", () => { + assert.deepEqual(parsePositiveInt(undefined, 42), { value: 42, ok: true }); + assert.deepEqual(parsePositiveInt("", 42), { value: 42, ok: true }); +}); + +test("parsePositiveInt: valid positive integer → parsed value", () => { + assert.equal(parsePositiveInt("5000000", 42).value, 5000000); + assert.equal(parsePositiveInt("5000000", 42).ok, true); +}); + +test("parsePositiveInt: 'unlimited' → NaN rejected, default kept (would drop the cap)", () => { + const r = parsePositiveInt("unlimited", 5 * 1024 * 1024); + assert.equal(r.value, 5 * 1024 * 1024); + assert.equal(r.ok, false); +}); + +test("parsePositiveInt: '5MB' → unit suffix rejected (naive parseInt would give 5 bytes)", () => { + const r = parsePositiveInt("5MB", 5 * 1024 * 1024); + assert.equal(r.value, 5 * 1024 * 1024); + assert.equal(r.ok, false); +}); + +test("parsePositiveInt: '0' and '-1' → non-positive rejected", () => { + assert.equal(parsePositiveInt("0", 20).ok, false); + assert.equal(parsePositiveInt("0", 20).value, 20); + assert.equal(parsePositiveInt("-1", 20).ok, false); +}); + +test("parsePositiveInt: '20.5' → fractional/ambiguous rejected", () => { + assert.equal(parsePositiveInt("20.5", 20).ok, false); +}); + +test("parsePositiveInt: surrounding whitespace tolerated", () => { + assert.deepEqual(parsePositiveInt(" 20 ", 5), { value: 20, ok: true }); +}); + +// ── PR #154 review round 2, gap (a): MAX_PROMPT_CHARS must fail closed like the other caps ── +// server.mjs now derives MAX_PROMPT_CHARS via parseIntEnv → parsePositiveInt (was a raw parseInt). +// CLAUDE_MAX_PROMPT_CHARS=unlimited previously → NaN → enforceTextBudget's `!(NaN > 0)` early-return +// → 500k chars passed unbounded, defeating F2's text-budget guarantee. The default must be kept. +test("parsePositiveInt: CLAUDE_MAX_PROMPT_CHARS='unlimited' → default kept, cap not lost to NaN (gap a)", () => { + const r = parsePositiveInt("unlimited", 150000); + assert.equal(r.ok, false); + assert.equal(r.value, 150000, "the 150k text budget must survive a bad config, not become NaN"); +}); +test("parsePositiveInt: CLAUDE_MAX_PROMPT_CHARS valid override honored", () => { + assert.deepEqual(parsePositiveInt("200000", 150000), { value: 200000, ok: true }); +}); + // ── messages guard predicate truth-table (issue #110) ──────────────────────── // Mirrors the guard at server.mjs line ~1650: Array.isArray(x) && x.length > 0 console.log("\nmessages guard predicate (issue #110):");