feat(chat): forward OpenAI image_url parts to Claude (multimodal vision) (#154)

* feat(chat): forward OpenAI image_url parts to Claude (multimodal vision)

`POST /v1/chat/completions` previously flattened every message to plain text
via contentToText(), replacing image_url parts with "[non-text content
omitted]" — so images were silently dropped (issue #110). This adds real
multimodal support: OpenAI `image_url` content parts are translated to
Anthropic image blocks and fed to the Claude CLI over
`--input-format stream-json`, keeping subscription auth (the reason OCP routes
through the CLI rather than the API).

Class B.1 (OpenAI-compatibility surface), authorized by ADR 0006. Request
shape follows OpenAI's published vision / chat-completions spec
(https://platform.openai.com/docs/guides/vision and the chat/completions
`content` image_url part) — no field is introduced beyond OpenAI's shape. The
CLI's `--input-format stream-json` is the transport for this Class B endpoint,
not a forwarded cli.js operation, so there is no Class A cli.js citation to
make; scope is justified under the ALIGNMENT.md Class B mapping of Rule 2
(no invention beyond the cited OpenAI spec).

Mechanism (verified empirically against the installed CLI, v2.1.206): a user
message whose `content` is an Anthropic block array including
`{type:"image",source:{type:"base64",media_type,data}}` fed to
`claude -p --input-format stream-json` is correctly described by the model.
Confirmed live end-to-end through this endpoint (a base64 PNG returns the
correct color).

Design:
- New pure module `lib/multimodal.mjs` (mirrors the lib/*.mjs pattern; unit-
  testable without a live server): hasImageContent, buildImageBlocks,
  buildStreamJsonInput, MultimodalError.
- server.mjs: text path is byte-for-byte unchanged. Only when a request carries
  an image_url part does spawnClaudeProcess switch stdin to a stream-json user
  envelope and buildCliArgs add `--input-format stream-json`. Image parsing runs
  before any stats mutation so a validation failure never leaks counters/slots.
- Images bypass the text char budget (CLAUDE_MAX_PROMPT_CHARS) and are bounded
  by explicit byte/count caps with clear 4xx errors (413 for size/count, 400
  for malformed/unsupported/disabled-remote), never a silent drop.

Scope decisions (v1):
- Base64 data URIs supported by default (image/jpeg,png,gif,webp).
- Remote http(s) image URLs OFF by default behind CLAUDE_IMAGE_ALLOW_URL; when
  enabled they are passed through as an Anthropic url-source (OCP never fetches
  the URL itself, so no OCP-side SSRF surface).
- Audio/file parts deferred: existing placeholder behavior preserved.
- Images anywhere in multi-turn history, not just the last message.

New env vars (documented in README Environment Variables table):
CLAUDE_IMAGE_ALLOW_URL, CLAUDE_MAX_IMAGE_BYTES, CLAUDE_MAX_IMAGES,
CLAUDE_MAX_IMAGE_TOTAL_BYTES, CLAUDE_MAX_BODY_SIZE (now configurable; default
5 MB unchanged).

Tests: 26 unit tests in test-features.mjs covering data-URI parse, multiple
images, text/image ordering, multi-turn history images, malformed/oversized/
too-many handling, remote-URL policy, and text-path parity. `npm test` green
(289 passed). `node --check` clean. No alignment-blacklist tokens added.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(chat): address PR #154 review blockers — TUI guard, fail-closed caps, text budget

Remediates the three merge-blocking findings from the maintainer's review of the
multimodal vision PR. Class B.1 (OpenAI-compat surface): request shape per OpenAI
vision spec (image_url content parts), authorized by ADR 0006. No new wire shape
and no cli.js surface change — the stream-json image contract is the CLI's native
input format (already cited in the base commit); these are correctness fixes on the
OCP-owned validation/dispatch layer.

F1 — TUI mode silently dropped images and returned 200. callClaudeTui() renders
every non-text part as "[non-text content omitted]", so a vision request in
CLAUDE_TUI_MODE=true was answered about an image the model never saw. Now
handleChatCompletions fails loudly with 400 images_unsupported_in_tui_mode instead
of a silent drop (ALIGNMENT.md forbids serving text the model did not mean).
Documented in README § Images / Multimodal.

F3 — NaN env parsing failed open. CLAUDE_MAX_BODY_SIZE=unlimited -> NaN ->
`body.length > NaN` always false -> body cap gone (OOM DoS); =5MB -> 5 bytes ->
proxy bricked; same on CLAUDE_MAX_IMAGES / _IMAGE_BYTES / _IMAGE_TOTAL_BYTES.
Added lib/env.mjs parsePositiveInt (pure, fail-closed) + a thin parseIntEnv warn
wrapper; a malformed cap now keeps the safe default and warns at startup.

F2 — images let unbounded text bypass MAX_PROMPT_CHARS. buildImageBlocks only
counted textChars and never truncated; the budget was never passed in. Threaded
maxTextChars (= MAX_PROMPT_CHARS) into the multimodal transform, which now
truncates text tail-first (mirroring messagesToPrompt) while preserving image
blocks, and logs prompt_truncated.

Tests: +11 in test-features.mjs (all pure-module, per the repo's no-server-import
pattern) covering the text-budget enforcement and the fail-closed cap parsing,
including the exact F2 (500k chars + 1 image) and F3 (unlimited/5MB/0/20.5)
scenarios. 300 passed, 0 failed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(server): address PR #154 review round 2 — MAX_PROMPT_CHARS fail-closed + system-only image guard

Closes the two residual gaps from the round-2 review, both traced to the same root as the
already-fixed blockers. Class B.1 (OpenAI-compat vision): authorized by ADR 0006; request
shape is the OpenAI `image_url` content part. No new wire shape — the Anthropic image block
over `--input-format stream-json` is the CLI's native contract (cli.js buildStreamJsonInput
path, verified live in round 1). MAX_PROMPT_CHARS is OCP's own truncation guard, not a cli.js
operation.

Gap (a) — MAX_PROMPT_CHARS was left on the raw parseInt while every other cap moved to the
fail-closed helper. `let MAX_PROMPT_CHARS = parseInt(env||"150000",10)` sat five lines above
the parseIntEnv helper this PR added, so CLAUDE_MAX_PROMPT_CHARS=unlimited → NaN →
enforceTextBudget's `!(NaN > 0)` early-return → 500k chars passed unbounded, truncated:false,
silently defeating F2's text-budget guarantee under a plausible operator config. Fix: hoist
parseIntEnv above the declaration and derive MAX_PROMPT_CHARS through it (keeps `let` for the
settings API). A misconfigured value now keeps the 150k default and warns, like the other caps.

Gap (b) — an image present ONLY in a system message silently dropped in non-TUI mode.
Detection runs on the full message list, but extraction/spawn filter role==="system" out, so a
system-only image was detected as multimodal, survived no filter, fell to the text path, and
rendered as "[non-text content omitted]" → 200 with a hallucinated answer — the one silent-drop
outcome F1 exists to forbid. Fix: after filtering, if hasImageContent(full) is true but no image
survives, return `400 images_unsupported_in_system_messages`. Narrow (OpenAI disallows images in
the system role) so no legitimate request is rejected. Documented in README § Images.

Tests: +4 (all pure-module) — parsePositiveInt('unlimited') keeps the 150k default (gap a) and a
valid override is honored; hasImageContent proves the guard predicate fires for a system-only
image (true on full list, false after the system filter) and does NOT fire for a user-message
image. 304 passed, 0 failed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: vvlasy-openclaw <vvlasy-openclaw@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: vvlasy-openclaw <vvlasy@gmail.com>
Co-authored-by: dtzp555 <dtzp555@gmail.com>
This commit is contained in:
openclaw.vvlasy.cz
2026-07-20 07:50:17 +10:00
committed by GitHub
co-authored by vvlasy-openclaw Claude Opus 4.8 vvlasy-openclaw taodeng
parent fe12419386
commit ac81badda1
5 changed files with 878 additions and 28 deletions
+92 -2
View File
@@ -27,7 +27,7 @@ One proxy. Multiple IDEs. All models. **$0 API cost.**
- [How It Works](#how-it-works) - [How It Works](#how-it-works)
- Reference: [Available Models](#available-models) · [API Endpoints](#api-endpoints) · [Environment Variables](#environment-variables) - Reference: [Available Models](#available-models) · [API Endpoints](#api-endpoints) · [Environment Variables](#environment-variables)
- Modes & operations: [LAN & multi-user](#lan--multi-user) → [`docs/lan-mode.md`](docs/lan-mode.md) · [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) → [`docs/tui-mode.md`](docs/tui-mode.md) · [Upgrading](#upgrading) → [`docs/upgrading.md`](docs/upgrading.md) - Modes & operations: [LAN & multi-user](#lan--multi-user) → [`docs/lan-mode.md`](docs/lan-mode.md) · [Subscription-pool (TUI) mode](#subscription-pool-tui-mode) → [`docs/tui-mode.md`](docs/tui-mode.md) · [Upgrading](#upgrading) → [`docs/upgrading.md`](docs/upgrading.md)
- [Built-in Usage Monitoring](#built-in-usage-monitoring) · [Response Cache](#response-cache) · [OpenClaw Integration](#openclaw-integration) - [Built-in Usage Monitoring](#built-in-usage-monitoring) · [Response Cache](#response-cache) · [Images / Multimodal](#images--multimodal-vision) · [OpenClaw Integration](#openclaw-integration)
- [Troubleshooting](#troubleshooting) → [`docs/troubleshooting.md`](docs/troubleshooting.md) - [Troubleshooting](#troubleshooting) → [`docs/troubleshooting.md`](docs/troubleshooting.md)
- [Repository Layout](#repository-layout) · [Security](#security) · [Governance](#governance) · [Support OCP](#support-ocp) · [License](#license) - [Repository Layout](#repository-layout) · [Security](#security) · [Governance](#governance) · [Support OCP](#support-ocp) · [License](#license)
@@ -222,12 +222,17 @@ The canonical list lives in [`models.json`](./models.json) — the single source
| `CLAUDE_MAX_CONCURRENT` | `8` | Max concurrent claude processes (`-p`/stream-json path) | | `CLAUDE_MAX_CONCURRENT` | `8` | Max concurrent claude processes (`-p`/stream-json path) |
| `CLAUDE_MAX_QUEUE` | `16` | Max requests **waiting** for a `-p` concurrency slot. Beyond `CLAUDE_MAX_CONCURRENT`, requests queue (up to this cap) instead of being rejected; when the queue is **also** full, the request gets `HTTP 429` + `Retry-After` (not an opaque 500). Surfaced on `/health.concurrency` + `/health.stats.queueRejections`. | | `CLAUDE_MAX_QUEUE` | `16` | Max requests **waiting** for a `-p` concurrency slot. Beyond `CLAUDE_MAX_CONCURRENT`, requests queue (up to this cap) instead of being rejected; when the queue is **also** full, the request gets `HTTP 429` + `Retry-After` (not an opaque 500). Surfaced on `/health.concurrency` + `/health.stats.queueRejections`. |
| `CLAUDE_QUEUE_RETRY_AFTER` | `5` | Seconds advertised in the `Retry-After` header on a `-p` concurrency-overflow `429`. | | `CLAUDE_QUEUE_RETRY_AFTER` | `5` | Seconds advertised in the `Retry-After` header on a `-p` concurrency-overflow `429`. |
| `CLAUDE_MAX_PROMPT_CHARS` | *(derived)* | Prompt truncation limit in chars. Default derives from the models.json SPOT: `max(contextWindow) × 3` — currently **600,000** (≈150200k tokens). Setting this env var (or the runtime settings API) overrides the derivation absolutely. See [ADR 0009](docs/adr/0009-spot-derived-prompt-budget.md). Note: very large prompts burn subscription-window quota quickly and slow TTFT; the TUI-mode paste path is untested beyond ~hundreds of KB. | | `CLAUDE_MAX_PROMPT_CHARS` | *(derived)* | Prompt truncation limit in chars. Default derives from the models.json SPOT: `max(contextWindow) × 3` — currently **600,000** (≈150200k tokens). Setting this env var (or the runtime settings API) overrides the derivation absolutely. See [ADR 0009](docs/adr/0009-spot-derived-prompt-budget.md). Note: very large prompts burn subscription-window quota quickly and slow TTFT; the TUI-mode paste path is untested beyond ~hundreds of KB. Applies to **text only** — image bytes bypass this budget (see [Images / Multimodal](#images--multimodal-vision)). |
| `CLAUDE_SESSION_TTL` | `3600000` | Session expiry (ms, default: 1 hour) | | `CLAUDE_SESSION_TTL` | `3600000` | Session expiry (ms, default: 1 hour) |
| `CLAUDE_CACHE_TTL` | `0` | Response cache TTL (ms, 0 = disabled). Set to e.g. `300000` for 5-min cache. See [Response Cache](#response-cache). | | `CLAUDE_CACHE_TTL` | `0` | Response cache TTL (ms, 0 = disabled). Set to e.g. `300000` for 5-min cache. See [Response Cache](#response-cache). |
| `CLAUDE_ALLOWED_TOOLS` | `Bash,Read,...,Agent` | Comma-separated tools to pre-approve | | `CLAUDE_ALLOWED_TOOLS` | `Bash,Read,...,Agent` | Comma-separated tools to pre-approve |
| `CLAUDE_SKIP_PERMISSIONS` | `false` | Bypass all permission checks | | `CLAUDE_SKIP_PERMISSIONS` | `false` | Bypass all permission checks |
| `CLAUDE_MCP_CONFIG` | *(unset)* | Path to an MCP server config JSON, passed to the spawned `claude` as `--mcp-config` (both the `-p` path and TUI `OCP_TUI_FULL_TOOLS` panes) | | `CLAUDE_MCP_CONFIG` | *(unset)* | Path to an MCP server config JSON, passed to the spawned `claude` as `--mcp-config` (both the `-p` path and TUI `OCP_TUI_FULL_TOOLS` panes) |
| `CLAUDE_MAX_BODY_SIZE` | `5242880` | Max request body size (bytes, default 5 MB). Base64 image payloads inflate ~33%; raise this to admit larger multimodal requests. Fail-closed parsing: a garbage value keeps the default. |
| `CLAUDE_IMAGE_ALLOW_URL` | `false` | Allow remote `http(s)` image URLs in `image_url` parts. **Off by default** (v1 supports base64 `data:` URIs only). When on, the URL is passed through to Anthropic as a `url` image source — **OCP does not fetch it** (no OCP-side SSRF surface); unreachable/blocked URLs surface as an API error. |
| `CLAUDE_MAX_IMAGE_BYTES` | `5242880` | Per-image decoded-byte cap (default 5 MB). Over-cap images get `HTTP 413`. |
| `CLAUDE_MAX_IMAGES` | `20` | Max image parts per request. Over-cap gets `HTTP 413`. |
| `CLAUDE_MAX_IMAGE_TOTAL_BYTES` | `20971520` | Aggregate decoded-byte cap across all images in a request (default 20 MB). Over-cap gets `HTTP 413`. |
| `CLAUDE_SYSTEM_PROMPT` | *(unset)* | Operator-wide system-prompt text appended (last) to every request's composed system prompt on the default `-p` path. TUI-mode panes are unaffected (they keep the interactive CLI's own system prompt). Echoed truncated on `/health.systemPrompt`. Note: changing this value and restarting auto-invalidates the response cache (the key carries a boot-config epoch, #177). | | `CLAUDE_SYSTEM_PROMPT` | *(unset)* | Operator-wide system-prompt text appended (last) to every request's composed system prompt on the default `-p` path. TUI-mode panes are unaffected (they keep the interactive CLI's own system prompt). Echoed truncated on `/health.systemPrompt`. Note: changing this value and restarting auto-invalidates the response cache (the key carries a boot-config epoch, #177). |
| `CLAUDE_NO_CONTEXT` | `false` | Suppress CLAUDE.md and auto-memory injection (pure API mode) | | `CLAUDE_NO_CONTEXT` | `false` | Suppress CLAUDE.md and auto-memory injection (pure API mode) |
| `PROXY_API_KEY` | *(unset)* | Bearer token for shared-mode authentication | | `PROXY_API_KEY` | *(unset)* | Bearer token for shared-mode authentication |
@@ -382,6 +387,91 @@ ocp settings cacheTTL 0 # disable at runtime
Cache is **disabled by default** (`CLAUDE_CACHE_TTL=0`). All data is stored locally in `~/.ocp/ocp.db`. **Hash format upgrade in v3.13.0:** legacy `v1` cache rows don't match new `v2`-format lookups; they orphan and are reaped by the TTL cleanup interval within one window — no migration script required. Cache is **disabled by default** (`CLAUDE_CACHE_TTL=0`). All data is stored locally in `~/.ocp/ocp.db`. **Hash format upgrade in v3.13.0:** legacy `v1` cache rows don't match new `v2`-format lookups; they orphan and are reaped by the TTL cleanup interval within one window — no migration script required.
## Images / Multimodal (Vision)
`POST /v1/chat/completions` accepts OpenAI-style multimodal `content` parts, so a
message can carry images alongside text and Claude will actually see them. This
follows OpenAI's [vision](https://platform.openai.com/docs/guides/vision) /
[chat-completions `image_url`](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages)
request shape — no OCP-invented fields. (Class B.1 endpoint; see ADR 0006.)
Under the hood, when a request carries an image OCP feeds the conversation to the
Claude CLI as Anthropic image blocks over `--input-format stream-json`. Text-only
requests are completely unaffected (unchanged code path).
### Supported input
- **Base64 data URIs** (default, recommended):
`data:image/png;base64,<...>`. Media types: `image/jpeg`, `image/png`,
`image/gif`, `image/webp`.
- **Remote `http(s)` URLs** — **off by default**. Set `CLAUDE_IMAGE_ALLOW_URL=1`
to enable; the URL is passed through to Anthropic (OCP never fetches it itself,
so there is no OCP-side SSRF surface).
- Images may appear in **any** message in the history (multi-turn), not just the
last one.
- Non-image, non-text parts (audio, files) are **not** yet supported and are
replaced with a `[non-text content omitted]` placeholder (deferred to a future
version).
### Example (base64 data URI)
```bash
curl -X POST http://127.0.0.1:3456/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "claude-sonnet-4-6",
"messages": [{
"role": "user",
"content": [
{ "type": "text", "text": "What is in this image?" },
{ "type": "image_url",
"image_url": { "url": "data:image/png;base64,iVBORw0KGgoAAA..." } }
]
}]
}'
```
### Not supported in TUI mode
Multimodal images require the default `-p` spawn path. In **TUI / subscription-pool
mode** (`CLAUDE_TUI_MODE=true`) the CLI is driven interactively and cannot carry
image blocks, so a request with an `image_url` part returns **`400
images_unsupported_in_tui_mode`** rather than silently dropping the image and
answering about something the model never saw. Remove the images, or run OCP
without TUI mode, to use vision.
Images must also live in a **user or assistant** message, not a `system` message
(system content is not forwarded to the CLI as image blocks). An `image_url` part
present only in a system message returns **`400 images_unsupported_in_system_messages`**
for the same reason — fail loudly rather than answer about an unseen image. This matches
the OpenAI vision spec, which does not place images in the system role.
### Limits
Images bypass the text `CLAUDE_MAX_PROMPT_CHARS` budget and are instead bounded by
their own byte/count caps. The **text** in a multimodal request is still subject to
`CLAUDE_MAX_PROMPT_CHARS` (older text is truncated exactly as on the text-only
path — only the image bytes are exempt). All numeric caps are parsed **fail-closed**:
a malformed value (e.g. `CLAUDE_MAX_BODY_SIZE=unlimited` or `=5MB`) is rejected with
a startup warning and the safe default is kept — a misconfigured cap can never
silently disable the guard. Requests that violate a cap get a clear `4xx` (never a
silent drop):
| Cap | Env var | Default | Error |
|-----|---------|---------|-------|
| Request body | `CLAUDE_MAX_BODY_SIZE` | 5 MB | `413` request body too large |
| Per-image bytes | `CLAUDE_MAX_IMAGE_BYTES` | 5 MB | `413` `image_too_large` |
| Total image bytes | `CLAUDE_MAX_IMAGE_TOTAL_BYTES` | 20 MB | `413` `images_too_large` |
| Image count | `CLAUDE_MAX_IMAGES` | 20 | `413` `too_many_images` |
| Unsupported media type | — | — | `400` `unsupported_image_type` |
| Malformed data URI | — | — | `400` `invalid_data_uri` |
| Remote URL while disabled | `CLAUDE_IMAGE_ALLOW_URL` | off | `400` `remote_url_disabled` |
Base64 payloads are large: a 5 MB image is ~6.7 MB as a data URI, so raise
`CLAUDE_MAX_BODY_SIZE` (and, if needed, `CLAUDE_MAX_IMAGE_BYTES`) to admit big
images. Vision support depends on the target model — request a current
vision-capable Claude model.
## OpenClaw Integration ## OpenClaw Integration
OCP was originally built for [OpenClaw](https://github.com/openclaw/openclaw) and includes deep integration: OCP was originally built for [OpenClaw](https://github.com/openclaw/openclaw) and includes deep integration:
+29
View File
@@ -0,0 +1,29 @@
// OCP env-var parsing helpers.
//
// Fail-closed positive-integer parsing for numeric caps (body size, image
// byte/count limits). A misconfigured cap must NEVER silently disable a guard:
// `parseInt("unlimited", 10)` is NaN and `x > NaN` is always false, so a naive
// parse of CLAUDE_MAX_BODY_SIZE=unlimited would remove the body-size limit
// entirely (unbounded body → OOM). Likewise CLAUDE_MAX_BODY_SIZE=5MB naively
// parses to 5 (bytes) and bricks the proxy. So a present-but-invalid value is
// REJECTED (default kept, caller warns), not accepted. (PR #154 review F3.)
//
// Pure (no env access, no IO) so it is unit-testable without a live server.
// Parse `raw` as a strictly-positive base-10 integer of bytes/count (no unit
// suffix). Returns { value, ok, reason }:
// - missing/empty → { value: def, ok: true } (use default)
// - valid positive int → { value: n, ok: true }
// - anything else → { value: def, ok: false, reason } (fail closed)
// Rejects: NaN ("unlimited"), non-positive ("0", "-1"), unit-suffixed ("5MB"),
// and fractional/ambiguous ("20.5", "0x10") values — String(n) !== trimmed catches
// any input parseInt only partially consumed.
export function parsePositiveInt(raw, def) {
if (raw === undefined || raw === null || raw === "") return { value: def, ok: true };
const trimmed = String(raw).trim();
const n = parseInt(trimmed, 10);
if (!Number.isFinite(n) || n <= 0 || String(n) !== trimmed) {
return { value: def, ok: false, reason: "not a strictly-positive integer (bytes/count, no unit suffix)" };
}
return { value: n, ok: true };
}
+278
View File
@@ -0,0 +1,278 @@
// OCP multimodal helpers — OpenAI `image_url` content parts → Anthropic image
// blocks fed to `claude -p --input-format stream-json`. (issue #110)
//
// Class B.1 (OpenAI-compatibility surface). Protocol authority is OpenAI's
// chat/completions spec — the multimodal `content` parts shape
// (https://platform.openai.com/docs/guides/vision and
// https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages,
// `image_url` part with `image_url.url` = data URI or http(s) URL). Authorized
// by ADR 0006. This module introduces NO field beyond OpenAI's published shape:
// the OpenAI-side vocabulary read here is `type:"image_url"` +
// `image_url:{url, detail?}`; the Anthropic-side vocabulary written here
// (`type:"image", source:{type:"base64"|"url", ...}`) is the CLI's native
// stream-json input contract, not an OCP invention.
//
// Kept as a pure module (no I/O, no network, no process state) mirroring the
// lib/*.mjs pattern so it is unit-testable without a live server. server.mjs is
// the only consumer; it owns spawning, caps configuration, and HTTP status.
// Anthropic vision-supported image media types. A data URI whose media type is
// outside this set is rejected with a clear 4xx rather than forwarded (the API
// would reject it anyway; failing early gives a better error).
export const SUPPORTED_IMAGE_TYPES = new Set([
"image/jpeg",
"image/png",
"image/gif",
"image/webp",
]);
// Default caps. server.mjs overrides these from env; they live here so the
// pure transform is self-contained and testable.
export const DEFAULT_MULTIMODAL_OPTS = {
allowRemoteUrl: false, // http(s) image URLs are OFF by default (v1: data URIs only)
maxImageBytes: 5 * 1024 * 1024, // per-image decoded-byte cap
maxImages: 20, // max image parts across the whole request
maxTotalImageBytes: 20 * 1024 * 1024, // aggregate decoded-byte cap
maxTextChars: Infinity, // text-char budget (server passes MAX_PROMPT_CHARS); Infinity = no truncation
};
// Typed error so server.mjs can map to the right HTTP status + OpenAI-shaped
// error body. `status` is the HTTP code; `type` is the OpenAI error `type`.
export class MultimodalError extends Error {
constructor(code, status, message) {
super(message);
this.name = "MultimodalError";
this.code = code;
this.status = status;
this.type = "invalid_request_error"; // OpenAI error `type` for 4xx client errors
}
}
// True if any message carries an OpenAI `image_url` content part. Cheap guard so
// the byte-for-byte text path is only left when an image is genuinely present.
export function hasImageContent(messages) {
if (!Array.isArray(messages)) return false;
for (const m of messages) {
if (m && Array.isArray(m.content)) {
for (const part of m.content) {
if (part && part.type === "image_url") return true;
}
}
}
return false;
}
// Extract the URL string from an OpenAI image_url part. Spec form is
// `{type:"image_url", image_url:{url, detail?}}`; many OpenAI-compatible clients
// also send `image_url` as a bare string. Accept both (input leniency — no new
// output field). `detail` (auto|low|high) is OpenAI-only and has no Anthropic
// analogue, so it is read-and-ignored.
function imageUrlOf(part) {
const iu = part.image_url;
if (typeof iu === "string") return iu;
if (iu && typeof iu.url === "string") return iu.url;
return null;
}
// Parse a base64 data URI: `data:[<media_type>][;base64],<data>`.
// Returns { mediaType, data (base64), bytes (decoded size) } or throws MultimodalError.
function parseDataUri(uri) {
const comma = uri.indexOf(",");
if (comma === -1) {
throw new MultimodalError("invalid_data_uri", 400, "Malformed image data URI (no comma).");
}
const meta = uri.slice(5, comma); // strip leading "data:"
const segs = meta.split(";");
const mediaType = (segs[0] || "").trim().toLowerCase();
const isBase64 = segs.slice(1).some((s) => s.trim().toLowerCase() === "base64");
if (!isBase64) {
throw new MultimodalError("invalid_data_uri", 400, "Only base64-encoded image data URIs are supported.");
}
if (!SUPPORTED_IMAGE_TYPES.has(mediaType)) {
throw new MultimodalError(
"unsupported_image_type",
400,
`Unsupported image media type '${mediaType || "(none)"}'. Supported: ${[...SUPPORTED_IMAGE_TYPES].join(", ")}.`
);
}
// Strip incidental whitespace/newlines some encoders insert into data URIs.
const data = uri.slice(comma + 1).replace(/\s/g, "");
if (data.length === 0 || !/^[A-Za-z0-9+/]+={0,2}$/.test(data)) {
throw new MultimodalError("invalid_data_uri", 400, "Image data URI payload is not valid base64.");
}
// Decoded size from base64 length (minus padding); avoids decoding the buffer
// just to measure it.
const padding = data.endsWith("==") ? 2 : data.endsWith("=") ? 1 : 0;
const bytes = Math.floor((data.length * 3) / 4) - padding;
return { mediaType, data, bytes };
}
// Convert a single OpenAI image_url part to an Anthropic image block, enforcing
// caps via the mutable `acc` accumulator ({ images, bytes }). Throws MultimodalError.
function imagePartToBlock(part, opts, acc) {
const url = imageUrlOf(part);
if (!url) {
throw new MultimodalError("invalid_image_url", 400, "image_url part is missing a URL.");
}
acc.images += 1;
if (acc.images > opts.maxImages) {
throw new MultimodalError("too_many_images", 413, `Too many images in request (max ${opts.maxImages}).`);
}
if (url.startsWith("data:")) {
const { mediaType, data, bytes } = parseDataUri(url);
if (bytes > opts.maxImageBytes) {
throw new MultimodalError("image_too_large", 413, `Image exceeds per-image size limit (${opts.maxImageBytes} bytes).`);
}
acc.bytes += bytes;
if (acc.bytes > opts.maxTotalImageBytes) {
throw new MultimodalError("images_too_large", 413, `Total image payload exceeds limit (${opts.maxTotalImageBytes} bytes).`);
}
return { type: "image", source: { type: "base64", media_type: mediaType, data } };
}
if (/^https?:\/\//i.test(url)) {
if (!opts.allowRemoteUrl) {
throw new MultimodalError(
"remote_url_disabled",
400,
"Remote image URLs are disabled. Enable CLAUDE_IMAGE_ALLOW_URL=1 to allow http(s) image URLs, or pass the image as a base64 data URI."
);
}
// Passthrough as an Anthropic url-source block. OCP does NOT fetch the URL
// itself (no OCP-side SSRF surface); the fetch is performed upstream by the
// Anthropic API. Best-effort: unreachable/blocked URLs surface as an API error.
return { type: "image", source: { type: "url", url } };
}
throw new MultimodalError("unsupported_url_scheme", 400, "image_url must be a base64 data URI or an http(s) URL.");
}
// Role prefix mirrors messagesToPrompt()'s text-path labeling so a multi-turn
// conversation reads the same whether or not it carries images. System messages
// are handled by the caller via --system-prompt and never reach here.
function rolePrefix(role) {
if (role === "assistant") return "[Assistant] ";
return ""; // user / tool / anything else: verbatim, as in the text path
}
// Build the Anthropic content-block array for a single stream-json user
// envelope. Mirrors the text path's "collapse the whole conversation into one
// turn passed via stdin" model (OCP runs stateless, full context per spawn), but
// preserves image position relative to text and keeps images out of the text
// char budget entirely. Returns { blocks, stats } or throws MultimodalError.
export function buildImageBlocks(messages, opts = {}) {
const o = { ...DEFAULT_MULTIMODAL_OPTS, ...opts };
const blocks = [];
const acc = { images: 0, bytes: 0 };
let textChars = 0;
let firstMessage = true;
const pushText = (text) => {
if (!text) return;
blocks.push({ type: "text", text });
textChars += text.length;
};
for (const m of messages) {
const prefix = rolePrefix(m.role);
// Separate messages with a blank line, matching messagesToPrompt's "\n\n" join.
const sep = firstMessage ? "" : "\n\n";
firstMessage = false;
let prefixEmitted = false;
const emitPrefixWith = (t) => {
if (prefixEmitted) return t;
prefixEmitted = true;
return sep + prefix + t;
};
if (typeof m.content === "string") {
pushText(emitPrefixWith(m.content));
continue;
}
if (!Array.isArray(m.content)) {
// null / object content: mirror contentToText's fallback.
const t = m.content == null ? "" : JSON.stringify(m.content);
pushText(emitPrefixWith(t));
continue;
}
for (const part of m.content) {
if (part && part.type === "text" && typeof part.text === "string") {
pushText(emitPrefixWith(part.text));
} else if (part && part.type === "image_url") {
// Ensure the role prefix isn't lost when a message leads with an image.
if (!prefixEmitted && prefix) pushText(emitPrefixWith(""));
blocks.push(imagePartToBlock(part, o, acc));
} else {
// audio / file / unknown parts: preserve the existing placeholder
// behavior (issue #110) — deferred to a future version.
pushText(emitPrefixWith("[non-text content omitted]"));
}
}
}
// Defensive: a stream-json user turn must have at least one content block.
if (blocks.length === 0) blocks.push({ type: "text", text: "" });
// Enforce the text-char budget (PR #154 review F2). Without this, attaching a
// single tiny image would let unbounded text bypass the gateway's runaway-context
// guard entirely — messagesToPrompt truncates the text path, so the multimodal
// path must too. Image blocks are preserved and are NOT counted (they are bounded
// by the byte/count caps above).
const budgeted = enforceTextBudget(blocks, o.maxTextChars);
return {
blocks: budgeted.blocks,
stats: {
imageCount: acc.images,
totalImageBytes: acc.bytes,
textChars: budgeted.textChars,
originalTextChars: budgeted.originalTextChars,
truncated: budgeted.truncated,
},
};
}
// Enforce a text-char budget over already-built content blocks, mirroring
// messagesToPrompt's "keep the tail, drop the oldest" truncation. Image blocks are
// preserved in place; only text blocks count against the budget. A truncation note
// is prepended when anything is dropped. Returns
// { blocks, truncated, originalTextChars, textChars }.
function enforceTextBudget(blocks, maxTextChars) {
let originalTextChars = 0;
for (const b of blocks) if (b.type === "text") originalTextChars += b.text.length;
if (!(maxTextChars > 0) || originalTextChars <= maxTextChars) {
return { blocks, truncated: false, originalTextChars, textChars: originalTextChars };
}
// Keep the most recent text up to the budget; trim within the boundary block and
// drop older text blocks. Non-text (image) blocks always survive, in order.
let budget = maxTextChars;
const out = [];
for (let i = blocks.length - 1; i >= 0; i--) {
const b = blocks[i];
if (b.type !== "text") { out.unshift(b); continue; }
if (budget <= 0) continue; // older text fully dropped
if (b.text.length <= budget) {
out.unshift(b);
budget -= b.text.length;
} else {
out.unshift({ type: "text", text: b.text.slice(b.text.length - budget) });
budget = 0;
}
}
out.unshift({ type: "text", text: "[System] Note: older text content was truncated to fit the context limit." });
let textChars = 0;
for (const b of out) if (b.type === "text") textChars += b.text.length;
return { blocks: out, truncated: true, originalTextChars, textChars };
}
// Serialize the non-system conversation to a single newline-terminated
// stream-json user message for `claude -p --input-format stream-json` stdin.
// Returns { payload, stats } or throws MultimodalError.
export function buildStreamJsonInput(messages, opts = {}) {
const { blocks, stats } = buildImageBlocks(messages, opts);
const envelope = { type: "user", message: { role: "user", content: blocks } };
return { payload: JSON.stringify(envelope) + "\n", stats };
}
+148 -26
View File
@@ -49,7 +49,9 @@ import { TuiSemaphore, SemaphoreAbortError, recordTuiEntrypoint, buildTuiHealthB
import { TuiPanePool, resolvePoolSize, POOL_MAX_SIZE } from "./lib/tui/pool.mjs"; import { TuiPanePool, resolvePoolSize, POOL_MAX_SIZE } from "./lib/tui/pool.mjs";
import { TuiDeltaAssembler, DEFAULT_HOLDBACK_CHARS, resolveStreamHoldback } from "./lib/tui/stream.mjs"; import { TuiDeltaAssembler, DEFAULT_HOLDBACK_CHARS, resolveStreamHoldback } from "./lib/tui/stream.mjs";
import { createSerialMutex, createTtlCache, isTokenExpiring, orderLabelsLastGoodFirst } from "./lib/spawn-auth.mjs"; import { createSerialMutex, createTtlCache, isTokenExpiring, orderLabelsLastGoodFirst } from "./lib/spawn-auth.mjs";
import { appendOperatorPrompt, resolvePromptCharBudget } from "./lib/prompt.mjs"; import { hasImageContent, buildImageBlocks, buildStreamJsonInput, MultimodalError } from "./lib/multimodal.mjs";
import { parsePositiveInt } from "./lib/env.mjs";
import { appendOperatorPrompt, derivePromptCharBudget } from "./lib/prompt.mjs";
const __dirname = dirname(fileURLToPath(import.meta.url)); const __dirname = dirname(fileURLToPath(import.meta.url));
const _pkg = JSON.parse(readFileSync(join(__dirname, "package.json"), "utf8")); const _pkg = JSON.parse(readFileSync(join(__dirname, "package.json"), "utf8"));
@@ -1100,7 +1102,7 @@ const authCheckInterval = setInterval(checkAuth, 600000);
// CLAUDE_SYSTEM_PROMPT env var is absorbed into the system prompt via // CLAUDE_SYSTEM_PROMPT env var is absorbed into the system prompt via
// extractSystemPrompt() at the caller level; APPEND_SYSTEM_PROMPT no longer used. // extractSystemPrompt() at the caller level; APPEND_SYSTEM_PROMPT no longer used.
// Note: ALLOWED_TOOLS / SKIP_PERMISSIONS / MCP_CONFIG are preserved as before. // Note: ALLOWED_TOOLS / SKIP_PERMISSIONS / MCP_CONFIG are preserved as before.
function buildCliArgs(cliModel, systemPrompt) { function buildCliArgs(cliModel, systemPrompt, opts = {}) {
const args = [ const args = [
"--model", cliModel, "--model", cliModel,
"--output-format", "stream-json", "--output-format", "stream-json",
@@ -1109,6 +1111,14 @@ function buildCliArgs(cliModel, systemPrompt) {
"--system-prompt", systemPrompt, "--system-prompt", systemPrompt,
]; ];
// Multimodal path (issue #110): images are fed as Anthropic content blocks over
// a stream-json stdin stream. `--input-format stream-json` (§ --input-format,
// choices text|stream-json; realtime streaming input) is added ONLY when the
// request carries an image part; the default (text) input path is untouched.
if (opts.streamJsonInput) {
args.push("--input-format", "stream-json");
}
// Permissions // Permissions
// ADR 0007 B-path: in multi-tenant mode, suppress operator-FS tools so a guest // ADR 0007 B-path: in multi-tenant mode, suppress operator-FS tools so a guest
// prompt cannot drive Bash/Read/Write/Edit/etc. on the operator's filesystem. // prompt cannot drive Bash/Read/Write/Edit/etc. on the operator's filesystem.
@@ -1144,17 +1154,45 @@ function buildCliArgs(cliModel, systemPrompt) {
return args; return args;
} }
// Thin env wrapper over parsePositiveInt (lib/env.mjs): resolve `name` from the
// environment fail-closed, warning on a present-but-invalid value. Keeps the pure
// parse in a unit-testable module. (PR #154 review F3)
function parseIntEnv(name, def) {
const { value, ok } = parsePositiveInt(process.env[name], def);
if (!ok) console.warn(`${name}="${process.env[name]}" is not a valid positive integer (bytes/count, no unit suffix); ignoring and using default ${def}.`);
return value;
}
// ── Format messages to prompt text ────────────────────────────────────── // ── Format messages to prompt text ──────────────────────────────────────
// Truncation guard: if total chars exceed MAX_PROMPT_CHARS, keep the system // Truncation guard: if total chars exceed MAX_PROMPT_CHARS, keep the system
// message(s) + first user message + last N messages, dropping the middle. // message(s) + first user message + last N messages, dropping the middle.
// This prevents runaway context from gateway-side conversation accumulation. // This prevents runaway context from gateway-side conversation accumulation.
// // Routed through parseIntEnv so a misconfigured cap fails CLOSED to the default rather than
// Default is SPOT-DERIVED (ADR 0009): max(models.json contextWindow) × 3 chars/token — // NaN — CLAUDE_MAX_PROMPT_CHARS=unlimited previously → NaN → enforceTextBudget's `!(NaN > 0)`
// currently 200000 × 3 = 600,000 chars — instead of the old hand-set 150000 (≈37.5k // early-return → 500k chars passed unbounded, silently defeating F2's text-budget guarantee
// English tokens), which silently under-delivered the advertised window by ~5×. The env // (PR #154 round 2, gap (a)). The default itself is SPOT-DERIVED (ADR 0009, PR #179):
// var (and the runtime settings API below) remain absolute operator overrides. If // max(models.json contextWindow) × 3 chars/token — currently 600,000 — so the two fixes
// models.json ever advertises a bigger window, this budget follows automatically. // compose: parseIntEnv guards a SET-but-garbage value, the derivation supplies the honest
let MAX_PROMPT_CHARS = resolvePromptCharBudget(process.env.CLAUDE_MAX_PROMPT_CHARS, modelsConfig.models); // default when unset/empty. `let` is kept for the settings API.
let MAX_PROMPT_CHARS = parseIntEnv("CLAUDE_MAX_PROMPT_CHARS", derivePromptCharBudget(modelsConfig.models));
// ── Multimodal image caps (issue #110) ──────────────────────────────────
// OpenAI `image_url` parts are forwarded to claude as Anthropic image blocks via
// `--input-format stream-json`. Images deliberately BYPASS the text char budget
// (MAX_PROMPT_CHARS) — they are bounded by these byte/count caps instead, and by
// MAX_BODY_SIZE at the HTTP layer. Data URIs are supported by default; remote
// http(s) image URLs are OFF unless CLAUDE_IMAGE_ALLOW_URL is set (v1: data URIs
// only). See docs/adr/0006-openai-shim-scope.md (Class B.1) and README § "Images".
const IMAGE_ALLOW_URL = /^(1|true|yes|on)$/i.test(process.env.CLAUDE_IMAGE_ALLOW_URL || "");
const MAX_IMAGE_BYTES = parseIntEnv("CLAUDE_MAX_IMAGE_BYTES", 5 * 1024 * 1024);
const MAX_IMAGES = parseIntEnv("CLAUDE_MAX_IMAGES", 20);
const MAX_IMAGE_TOTAL_BYTES = parseIntEnv("CLAUDE_MAX_IMAGE_TOTAL_BYTES", 20 * 1024 * 1024);
const MULTIMODAL_OPTS = {
allowRemoteUrl: IMAGE_ALLOW_URL,
maxImageBytes: MAX_IMAGE_BYTES,
maxImages: MAX_IMAGES,
maxTotalImageBytes: MAX_IMAGE_TOTAL_BYTES,
};
// Flatten OpenAI content (string | array of parts) to plain text for the prompt. // Flatten OpenAI content (string | array of parts) to plain text for the prompt.
// Array content: concatenate text parts; replace non-text parts (e.g. image_url) // Array content: concatenate text parts; replace non-text parts (e.g. image_url)
@@ -1246,25 +1284,51 @@ function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlo
// Circuit breaker: disabled (see comment at top of breaker section) // Circuit breaker: disabled (see comment at top of breaker section)
stats.activeRequests++;
stats.totalRequests++;
// Phase 6c: always serialize full conversation via stdin (no session resume). // Phase 6c: always serialize full conversation via stdin (no session resume).
// System messages are extracted and passed via --system-prompt; the remaining // System messages are extracted and passed via --system-prompt; the remaining
// messages (user/assistant/tool) are serialized by messagesToPrompt. // messages (user/assistant/tool) are serialized for stdin.
const systemPrompt = extractSystemPrompt(messages); const systemPrompt = extractSystemPrompt(messages);
// messagesToPrompt skips system messages now that they go via --system-prompt. // messagesToPrompt / buildStreamJsonInput skip system messages (they go via
// Filter them out before calling to avoid double-injection. // --system-prompt). Filter them out first to avoid double-injection.
const nonSystemMessages = messages.filter(m => m.role !== "system"); const nonSystemMessages = messages.filter(m => m.role !== "system");
const prompt = messagesToPrompt(nonSystemMessages);
stats.oneOffRequests++; // Multimodal (issue #110): when any message carries an OpenAI image_url part,
if (conversationId) { // feed the conversation as Anthropic content blocks over --input-format
console.log(`[session] stateless conv=${conversationId.slice(0, 12)}... key=${keyName || "anon"} msgs=${messages.length} prompt_chars=${prompt.length}`); // stream-json (images preserved and kept OUT of the text char budget).
// Otherwise the text path is byte-for-byte unchanged. buildStreamJsonInput may
// throw MultimodalError on an invalid/oversized image; it runs BEFORE any stats
// mutation so a validation failure never leaks counters or the concurrency slot
// (handleChatCompletions validates first, so in practice it will not throw here).
const useStreamJson = hasImageContent(nonSystemMessages);
let stdinPayload, promptChars;
if (useStreamJson) {
// Pass MAX_PROMPT_CHARS so the multimodal text is bounded by the same
// runaway-context guard as the text path (PR #154 review F2). Images bypass it.
const built = buildStreamJsonInput(nonSystemMessages, { ...MULTIMODAL_OPTS, maxTextChars: MAX_PROMPT_CHARS });
stdinPayload = built.payload;
promptChars = built.stats.textChars;
if (built.stats.truncated) {
logEvent("warn", "prompt_truncated", {
originalChars: built.stats.originalTextChars,
maxChars: MAX_PROMPT_CHARS,
keptChars: built.stats.textChars,
path: "multimodal",
});
}
} else {
stdinPayload = messagesToPrompt(nonSystemMessages);
promptChars = stdinPayload.length;
} }
const cliArgs = buildCliArgs(cliModel, systemPrompt); stats.activeRequests++;
stats.totalRequests++;
stats.oneOffRequests++;
if (conversationId) {
console.log(`[session] stateless conv=${conversationId.slice(0, 12)}... key=${keyName || "anon"} msgs=${messages.length} prompt_chars=${promptChars}`);
}
const cliArgs = buildCliArgs(cliModel, systemPrompt, { streamJsonInput: useStreamJson });
const env = { ...process.env }; const env = { ...process.env };
delete env.CLAUDECODE; delete env.CLAUDECODE;
@@ -1350,12 +1414,13 @@ function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlo
// the spawned process, NOT on the stdin Writable — it does not catch this. // the spawned process, NOT on the stdin Writable — it does not catch this.
proc.stdin.on("error", (e) => logEvent("warn", "stdin_write_error", { error: e.message })); proc.stdin.on("error", (e) => logEvent("warn", "stdin_write_error", { error: e.message }));
// Write prompt to stdin immediately // Write the serialized turn to stdin immediately. Text path: the flat prompt.
proc.stdin.write(prompt); // Multimodal path: a single newline-terminated stream-json user envelope.
proc.stdin.write(stdinPayload);
proc.stdin.end(); proc.stdin.end();
recordModelRequest(cliModel, prompt.length); recordModelRequest(cliModel, promptChars);
logEvent("info", "claude_spawned", { model: cliModel, promptChars: prompt.length, timeout: TIMEOUT, tier: getModelTier(cliModel), session: conversationId ? conversationId.slice(0, 12) + "..." : "none" }); logEvent("info", "claude_spawned", { model: cliModel, promptChars, inputFormat: useStreamJson ? "stream-json" : "text", timeout: TIMEOUT, tier: getModelTier(cliModel), session: conversationId ? conversationId.slice(0, 12) + "..." : "none" });
// Single request timeout — no separate first-byte timer. // Single request timeout — no separate first-byte timer.
// Claude tool-use causes long pauses in the token stream (30s-5min), // Claude tool-use causes long pauses in the token stream (30s-5min),
@@ -2579,7 +2644,13 @@ async function handleSettings(req, res) {
} }
// ── Handle chat completions ───────────────────────────────────────────── // ── Handle chat completions ─────────────────────────────────────────────
const MAX_BODY_SIZE = 5 * 1024 * 1024; // 5 MB // Default 5 MB, byte-for-byte unchanged unless CLAUDE_MAX_BODY_SIZE is set. Base64
// image payloads inflate ~33%; operators enabling large images (issue #110) can
// raise this to admit bigger requests. Parsed fail-closed (PR #154 review F3): a
// bad value (`unlimited` → NaN, `5MB` → 5) must not disable the body cap or brick
// the proxy — parseIntEnv keeps the 5 MB default and warns instead.
const MAX_BODY_SIZE = parseIntEnv("CLAUDE_MAX_BODY_SIZE", 5 * 1024 * 1024);
const MAX_BODY_SIZE_LABEL = `${Math.round(MAX_BODY_SIZE / (1024 * 1024))}MB`;
// Set of all valid model identifiers (canonical IDs + aliases) // Set of all valid model identifiers (canonical IDs + aliases)
const VALID_MODELS = new Set(Object.keys(MODEL_MAP)); const VALID_MODELS = new Set(Object.keys(MODEL_MAP));
@@ -2590,7 +2661,7 @@ async function handleChatCompletions(req, res) {
for await (const chunk of req) { for await (const chunk of req) {
body += chunk; body += chunk;
if (body.length > MAX_BODY_SIZE) { if (body.length > MAX_BODY_SIZE) {
return jsonResponse(res, 413, { error: { message: "Request body too large (max 5MB)", type: "invalid_request_error" } }); return jsonResponse(res, 413, { error: { message: `Request body too large (max ${MAX_BODY_SIZE_LABEL})`, type: "invalid_request_error" } });
} }
} }
} catch (e) { } catch (e) {
@@ -2619,6 +2690,57 @@ async function handleChatCompletions(req, res) {
return jsonResponse(res, 400, { error: { message: "'messages' must be a non-empty array", type: "invalid_request_error" } }); return jsonResponse(res, 400, { error: { message: "'messages' must be a non-empty array", type: "invalid_request_error" } });
} }
// Multimodal validation (issue #110): when a request carries OpenAI `image_url`
// content parts, validate/parse them now so an invalid, unsupported, or oversized
// image returns a clean 4xx BEFORE the cache/spawn path (rather than a silent drop
// or an opaque 500). The stream-json transform itself runs at spawn time
// (spawnClaudeProcess → buildStreamJsonInput). Class B.1: authorized by ADR 0006;
// request shape per OpenAI vision spec (image_url content parts). buildImageBlocks
// validates without stringifying, so this early pass is cheap.
if (hasImageContent(messages)) {
// F1 (PR #154 review): the TUI path (callClaudeTui → messagesToPrompt) cannot
// carry image blocks — it renders every non-text part as "[non-text content
// omitted]". Forwarding here would let the model answer about an image it never
// saw and return 200, which is strictly worse than an honest error (the one
// outcome ALIGNMENT.md forbids: silently serving text the model did not mean).
// Stream-json image input requires the `claude -p` path, so in TUI_MODE we fail
// loudly instead of dropping. Documented in README § "Images".
if (TUI_MODE) {
return jsonResponse(res, 400, {
error: {
message: "Image inputs are not supported in TUI mode (CLAUDE_TUI_MODE=true). Images require the default -p spawn path; remove images or run OCP without TUI mode.",
type: "invalid_request_error",
code: "images_unsupported_in_tui_mode",
},
});
}
// Detection runs on the FULL message list, but extraction/spawn drop system messages
// (system role carries no image blocks to the CLI). So an image present ONLY in a
// system message would be detected as multimodal, survive no filter, fall to the text
// path, and render as "[non-text content omitted]" → 200 with a hallucinated answer —
// the exact silent-drop this guard exists to forbid. Fail loudly instead. OpenAI
// disallows images in the system role anyway, so no legitimate request is rejected.
// (PR #154 review round 2, gap (b).)
const nonSystem = messages.filter(m => m.role !== "system");
if (!hasImageContent(nonSystem)) {
return jsonResponse(res, 400, {
error: {
message: "Image inputs are only supported in user/assistant messages, not in system messages. Move the image_url part to a user message.",
type: "invalid_request_error",
code: "images_unsupported_in_system_messages",
},
});
}
try {
buildImageBlocks(nonSystem, { ...MULTIMODAL_OPTS, maxTextChars: MAX_PROMPT_CHARS });
} catch (e) {
if (e instanceof MultimodalError) {
return jsonResponse(res, e.status, { error: { message: e.message, type: e.type, code: e.code } });
}
throw e;
}
}
// NOTE: quota is best-effort / eventually-consistent. The gate reads the recorded count // NOTE: quota is best-effort / eventually-consistent. The gate reads the recorded count
// at entry and records only after the upstream completes, so concurrent requests at the // at entry and records only after the upstream completes, so concurrent requests at the
// boundary can overshoot the cap by up to MAX_CONCURRENT, and cache hits (served before // boundary can overshoot the cap by up to MAX_CONCURRENT, and cache hits (served before
+331
View File
@@ -3413,6 +3413,337 @@ test("contentToText: null returns empty string", () => {
assert.equal(contentToText(null), ""); assert.equal(contentToText(null), "");
}); });
// ── multimodal image transform (issue #110) ──────────────────────────────────
// OpenAI image_url parts → Anthropic image blocks for `claude -p --input-format
// stream-json`. lib/multimodal.mjs is a PURE module (no server.listen()), so it is
// imported directly here. Class B.1: shape per OpenAI vision spec, authorized by
// ADR 0006. Mechanism verified live: a base64 PNG fed as an Anthropic image block
// via --input-format stream-json is correctly described by the model.
import {
hasImageContent as mmHasImageContent,
buildImageBlocks as mmBuildImageBlocks,
buildStreamJsonInput as mmBuildStreamJsonInput,
MultimodalError as MmError,
SUPPORTED_IMAGE_TYPES as MM_SUPPORTED,
} from "./lib/multimodal.mjs";
import { parsePositiveInt } from "./lib/env.mjs";
console.log("\nmultimodal image transform (issue #110):");
// A short, valid base64 string (charset-valid; not decoded by the transform).
const MM_B64 = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAAAAAA6fptVAAAACklEQVR4nGP4DwABAQEAG7buVgAAAABJRU5ErkJggg==";
const dataUri = (mt = "image/png") => `data:${mt};base64,${MM_B64}`;
const imgPart = (mt) => ({ type: "image_url", image_url: { url: dataUri(mt) } });
const txtPart = (t) => ({ type: "text", text: t });
test("hasImageContent: plain string message → false (text path preserved)", () => {
assert.equal(mmHasImageContent([{ role: "user", content: "hello" }]), false);
});
test("hasImageContent: array of text-only parts → false", () => {
assert.equal(mmHasImageContent([{ role: "user", content: [txtPart("a"), txtPart("b")] }]), false);
});
test("hasImageContent: message with an image_url part → true", () => {
assert.equal(mmHasImageContent([{ role: "user", content: [txtPart("q"), imgPart()] }]), true);
});
test("hasImageContent: image anywhere in history (not just last) → true", () => {
const msgs = [
{ role: "user", content: [txtPart("look"), imgPart()] },
{ role: "assistant", content: "ok" },
{ role: "user", content: "and now?" },
];
assert.equal(mmHasImageContent(msgs), true);
});
// ── PR #154 review round 2, gap (b): image ONLY in a system message must not silently drop ──
// The handler detects multimodal on the FULL list but extraction/spawn filter system messages out.
// The guard fires exactly when the full list has an image but the non-system list does not — proven
// here against the same predicate the guard uses, so a system-only image is rejected (400) rather
// than falling to the text path and returning a 200 hallucinated answer.
test("hasImageContent: image ONLY in a system message → true on full list, false after system filter (guard fires)", () => {
const msgs = [
{ role: "system", content: [txtPart("context"), imgPart()] },
{ role: "user", content: "describe it" },
];
assert.equal(mmHasImageContent(msgs), true, "detected as multimodal on the full list");
assert.equal(mmHasImageContent(msgs.filter(m => m.role !== "system")), false, "no image survives the system filter → guard must 400");
});
test("hasImageContent: image in a USER message survives the system filter (legitimate request not rejected)", () => {
const msgs = [
{ role: "system", content: "you are helpful" },
{ role: "user", content: [txtPart("describe it"), imgPart()] },
];
assert.equal(mmHasImageContent(msgs.filter(m => m.role !== "system")), true, "user image survives → normal multimodal path");
});
test("buildImageBlocks: data-URI parsed into an Anthropic base64 image block", () => {
const { blocks, stats } = mmBuildImageBlocks([{ role: "user", content: [txtPart("what is this?"), imgPart("image/png")] }]);
assert.equal(blocks.length, 2);
assert.equal(blocks[0].type, "text");
assert.equal(blocks[0].text, "what is this?");
assert.deepEqual(blocks[1], { type: "image", source: { type: "base64", media_type: "image/png", data: MM_B64 } });
assert.equal(stats.imageCount, 1);
assert.ok(stats.totalImageBytes > 0);
});
test("buildImageBlocks: media_type carried through (jpeg/gif/webp)", () => {
for (const mt of ["image/jpeg", "image/gif", "image/webp"]) {
const { blocks } = mmBuildImageBlocks([{ role: "user", content: [imgPart(mt)] }]);
assert.equal(blocks.find(b => b.type === "image").source.media_type, mt);
}
});
test("buildImageBlocks: multiple images in one message both emitted", () => {
const { blocks, stats } = mmBuildImageBlocks([{ role: "user", content: [txtPart("compare"), imgPart(), imgPart()] }]);
const imgs = blocks.filter(b => b.type === "image");
assert.equal(imgs.length, 2);
assert.equal(stats.imageCount, 2);
});
test("buildImageBlocks: text/image/text ordering preserved", () => {
const { blocks } = mmBuildImageBlocks([{ role: "user", content: [txtPart("A"), imgPart(), txtPart("B")] }]);
assert.deepEqual(blocks.map(b => (b.type === "text" ? b.text : "IMG")), ["A", "IMG", "B"]);
});
test("buildImageBlocks: image-first message keeps ordering (image before text)", () => {
const { blocks } = mmBuildImageBlocks([{ role: "user", content: [imgPart(), txtPart("caption")] }]);
assert.deepEqual(blocks.map(b => (b.type === "text" ? b.text : "IMG")), ["IMG", "caption"]);
});
test("buildImageBlocks: multi-turn history — role prefixes + separators preserved", () => {
const msgs = [
{ role: "user", content: "first q" },
{ role: "assistant", content: "prior answer" },
{ role: "user", content: [txtPart("now this"), imgPart()] },
];
const { blocks } = mmBuildImageBlocks(msgs);
assert.equal(blocks[0].text, "first q");
assert.equal(blocks[1].text, "\n\n[Assistant] prior answer");
assert.equal(blocks[2].text, "\n\nnow this");
assert.equal(blocks[3].type, "image");
});
test("buildImageBlocks: image in an EARLIER turn is carried (history image)", () => {
const msgs = [
{ role: "user", content: [txtPart("here"), imgPart()] },
{ role: "assistant", content: "got it" },
{ role: "user", content: "thanks" },
];
const { blocks, stats } = mmBuildImageBlocks(msgs);
assert.equal(stats.imageCount, 1);
assert.equal(blocks.filter(b => b.type === "image").length, 1);
});
test("buildImageBlocks: image_url as bare string is accepted (client leniency)", () => {
const { blocks } = mmBuildImageBlocks([{ role: "user", content: [{ type: "image_url", image_url: dataUri() }] }]);
assert.equal(blocks.find(b => b.type === "image").source.data, MM_B64);
});
test("buildStreamJsonInput: emits one newline-terminated user envelope", () => {
const { payload } = mmBuildStreamJsonInput([{ role: "user", content: [txtPart("hi"), imgPart()] }]);
assert.ok(payload.endsWith("\n"));
const env = JSON.parse(payload.trim());
assert.equal(env.type, "user");
assert.equal(env.message.role, "user");
assert.equal(env.message.content[1].type, "image");
});
// ── malformed / policy / oversized handling (clean 4xx, never a silent drop) ──
test("buildImageBlocks: unsupported media type → 400 unsupported_image_type", () => {
assert.throws(
() => mmBuildImageBlocks([{ role: "user", content: [imgPart("image/tiff")] }]),
(e) => e instanceof MmError && e.code === "unsupported_image_type" && e.status === 400
);
});
test("buildImageBlocks: non-base64 data URI → 400 invalid_data_uri", () => {
assert.throws(
() => mmBuildImageBlocks([{ role: "user", content: [{ type: "image_url", image_url: { url: "data:image/png,notbase64" } }] }]),
(e) => e instanceof MmError && e.code === "invalid_data_uri" && e.status === 400
);
});
test("buildImageBlocks: malformed data URI (no comma) → 400 invalid_data_uri", () => {
assert.throws(
() => mmBuildImageBlocks([{ role: "user", content: [{ type: "image_url", image_url: { url: "data:image/png;base64" } }] }]),
(e) => e instanceof MmError && e.code === "invalid_data_uri"
);
});
test("buildImageBlocks: image_url part missing a URL → 400 invalid_image_url", () => {
assert.throws(
() => mmBuildImageBlocks([{ role: "user", content: [{ type: "image_url", image_url: {} }] }]),
(e) => e instanceof MmError && e.code === "invalid_image_url"
);
});
test("buildImageBlocks: oversized single image → 413 image_too_large", () => {
assert.throws(
() => mmBuildImageBlocks([{ role: "user", content: [imgPart()] }], { maxImageBytes: 4 }),
(e) => e instanceof MmError && e.code === "image_too_large" && e.status === 413
);
});
test("buildImageBlocks: too many images → 413 too_many_images", () => {
const many = Array.from({ length: 3 }, () => imgPart());
assert.throws(
() => mmBuildImageBlocks([{ role: "user", content: many }], { maxImages: 2 }),
(e) => e instanceof MmError && e.code === "too_many_images" && e.status === 413
);
});
test("buildImageBlocks: aggregate image bytes over cap → 413 images_too_large", () => {
assert.throws(
() => mmBuildImageBlocks([{ role: "user", content: [imgPart(), imgPart()] }], { maxTotalImageBytes: 100, maxImageBytes: 1000 }),
(e) => e instanceof MmError && e.code === "images_too_large" && e.status === 413
);
});
test("buildImageBlocks: remote http(s) URL disabled by default → 400 remote_url_disabled", () => {
assert.throws(
() => mmBuildImageBlocks([{ role: "user", content: [{ type: "image_url", image_url: { url: "https://example.com/a.png" } }] }]),
(e) => e instanceof MmError && e.code === "remote_url_disabled" && e.status === 400
);
});
test("buildImageBlocks: remote URL passthrough when allowRemoteUrl=true (url source, OCP does not fetch)", () => {
const { blocks } = mmBuildImageBlocks(
[{ role: "user", content: [{ type: "image_url", image_url: { url: "https://example.com/a.png" } }] }],
{ allowRemoteUrl: true }
);
assert.deepEqual(blocks.find(b => b.type === "image").source, { type: "url", url: "https://example.com/a.png" });
});
test("buildImageBlocks: unsupported URL scheme → 400 unsupported_url_scheme", () => {
assert.throws(
() => mmBuildImageBlocks([{ role: "user", content: [{ type: "image_url", image_url: { url: "ftp://x/y.png" } }] }], { allowRemoteUrl: true }),
(e) => e instanceof MmError && e.code === "unsupported_url_scheme"
);
});
test("buildImageBlocks: non-image parts (audio/file) fall back to placeholder text", () => {
const { blocks } = mmBuildImageBlocks([{ role: "user", content: [txtPart("hear this"), { type: "input_audio", input_audio: {} }] }]);
assert.deepEqual(blocks.map(b => b.text), ["hear this", "[non-text content omitted]"]);
});
test("SUPPORTED_IMAGE_TYPES: exactly the four Anthropic vision types", () => {
assert.deepEqual([...MM_SUPPORTED].sort(), ["image/gif", "image/jpeg", "image/png", "image/webp"]);
});
test("buildImageBlocks: pure-text conversation still yields text blocks (untouched-path parity)", () => {
// hasImageContent would be false for this input in server.mjs (text path taken),
// but the transform must still be well-defined for a text-only turn.
const { blocks, stats } = mmBuildImageBlocks([{ role: "user", content: "just text" }]);
assert.deepEqual(blocks, [{ type: "text", text: "just text" }]);
assert.equal(stats.imageCount, 0);
assert.equal(stats.truncated, false);
});
// ── F2 (PR #154 review): text char budget is enforced on the multimodal path ──
// Regression guard: without maxTextChars, attaching one tiny image let unbounded
// text bypass MAX_PROMPT_CHARS entirely (the text path truncates; the image path
// did not). server.mjs passes maxTextChars: MAX_PROMPT_CHARS into this transform.
console.log("\nmultimodal text-budget enforcement (PR #154 F2):");
test("buildImageBlocks: text under budget → not truncated, blocks unchanged", () => {
const { blocks, stats } = mmBuildImageBlocks(
[{ role: "user", content: [txtPart("short"), imgPart()] }],
{ maxTextChars: 1000 }
);
assert.equal(stats.truncated, false);
assert.equal(stats.textChars, "short".length);
assert.equal(blocks.filter(b => b.type === "image").length, 1);
});
test("buildImageBlocks: text over budget → truncated, keeps most-recent tail + note", () => {
const big = "A".repeat(300) + "TAIL_MARKER";
const { blocks, stats } = mmBuildImageBlocks(
[{ role: "user", content: [txtPart(big)] }],
{ maxTextChars: 50 }
);
assert.equal(stats.truncated, true);
assert.equal(stats.originalTextChars, big.length);
// The most recent characters (the tail) survive; the oldest 'A's are dropped.
const joined = blocks.filter(b => b.type === "text").map(b => b.text).join("");
assert.ok(joined.includes("TAIL_MARKER"), "tail text must be kept");
assert.ok(joined.includes("truncated to fit"), "a truncation note must be present");
assert.ok(stats.originalTextChars > stats.textChars, "post-truncation text is smaller");
});
test("buildImageBlocks: F2 exact scenario — 500k chars + one image → text bounded, image preserved", () => {
const { blocks, stats } = mmBuildImageBlocks(
[{ role: "user", content: [txtPart("Z".repeat(500000)), imgPart()] }],
{ maxTextChars: 150000 }
);
assert.equal(stats.truncated, true);
assert.ok(stats.textChars <= 150000 + 200, "text char count is bounded by the budget (+note)");
// The image bypasses the text budget and is NOT dropped by truncation.
assert.equal(blocks.filter(b => b.type === "image").length, 1);
});
test("buildImageBlocks: default (no maxTextChars) never truncates — pure module standalone", () => {
const { stats } = mmBuildImageBlocks([{ role: "user", content: [txtPart("x".repeat(10000))] }]);
assert.equal(stats.truncated, false);
assert.equal(stats.textChars, 10000);
});
// ── F3 (PR #154 review): fail-closed positive-int env parsing ────────────────
// A misconfigured numeric cap must NEVER silently disable a guard (`x > NaN` is
// always false) or brick the proxy with a nonsense value. parsePositiveInt keeps
// the default and reports ok:false so the caller can warn.
console.log("\nfail-closed env-cap parsing (PR #154 F3):");
test("parsePositiveInt: missing/empty → default, ok", () => {
assert.deepEqual(parsePositiveInt(undefined, 42), { value: 42, ok: true });
assert.deepEqual(parsePositiveInt("", 42), { value: 42, ok: true });
});
test("parsePositiveInt: valid positive integer → parsed value", () => {
assert.equal(parsePositiveInt("5000000", 42).value, 5000000);
assert.equal(parsePositiveInt("5000000", 42).ok, true);
});
test("parsePositiveInt: 'unlimited' → NaN rejected, default kept (would drop the cap)", () => {
const r = parsePositiveInt("unlimited", 5 * 1024 * 1024);
assert.equal(r.value, 5 * 1024 * 1024);
assert.equal(r.ok, false);
});
test("parsePositiveInt: '5MB' → unit suffix rejected (naive parseInt would give 5 bytes)", () => {
const r = parsePositiveInt("5MB", 5 * 1024 * 1024);
assert.equal(r.value, 5 * 1024 * 1024);
assert.equal(r.ok, false);
});
test("parsePositiveInt: '0' and '-1' → non-positive rejected", () => {
assert.equal(parsePositiveInt("0", 20).ok, false);
assert.equal(parsePositiveInt("0", 20).value, 20);
assert.equal(parsePositiveInt("-1", 20).ok, false);
});
test("parsePositiveInt: '20.5' → fractional/ambiguous rejected", () => {
assert.equal(parsePositiveInt("20.5", 20).ok, false);
});
test("parsePositiveInt: surrounding whitespace tolerated", () => {
assert.deepEqual(parsePositiveInt(" 20 ", 5), { value: 20, ok: true });
});
// ── PR #154 review round 2, gap (a): MAX_PROMPT_CHARS must fail closed like the other caps ──
// server.mjs now derives MAX_PROMPT_CHARS via parseIntEnv → parsePositiveInt (was a raw parseInt).
// CLAUDE_MAX_PROMPT_CHARS=unlimited previously → NaN → enforceTextBudget's `!(NaN > 0)` early-return
// → 500k chars passed unbounded, defeating F2's text-budget guarantee. The default must be kept.
test("parsePositiveInt: CLAUDE_MAX_PROMPT_CHARS='unlimited' → default kept, cap not lost to NaN (gap a)", () => {
const r = parsePositiveInt("unlimited", 150000);
assert.equal(r.ok, false);
assert.equal(r.value, 150000, "the 150k text budget must survive a bad config, not become NaN");
});
test("parsePositiveInt: CLAUDE_MAX_PROMPT_CHARS valid override honored", () => {
assert.deepEqual(parsePositiveInt("200000", 150000), { value: 200000, ok: true });
});
// ── messages guard predicate truth-table (issue #110) ──────────────────────── // ── messages guard predicate truth-table (issue #110) ────────────────────────
// Mirrors the guard at server.mjs line ~1650: Array.isArray(x) && x.length > 0 // Mirrors the guard at server.mjs line ~1650: Array.isArray(x) && x.length > 0
console.log("\nmessages guard predicate (issue #110):"); console.log("\nmessages guard predicate (issue #110):");