diff --git a/docs/adr/0005-no-multi-provider.md b/docs/adr/0005-no-multi-provider.md new file mode 100644 index 0000000..0477079 --- /dev/null +++ b/docs/adr/0005-no-multi-provider.md @@ -0,0 +1,79 @@ +# 0005 — OCP Stays Single-Provider; No Multi-Provider Refactor + +- **Date**: 2026-05-06 +- **Status**: Accepted +- **Authors**: project maintainer (with AI advisory drafting) +- **Related**: ADR 0002 (Alignment Constitution), ADR 0003 (`models.json` SPOT) + +## Context + +OCP's `server.mjs` reached 1667 lines and now provides response cache, per-key quota, session tracking, model-level stats, and SSE heartbeat — all targeting a single backend path: `spawn` the locally installed `cli.js` and let it transact with `api.anthropic.com`. This architecture is the source of OCP's only real differentiator: **`cli.js` behavior-level alignment** (session create-vs-resume semantics, tool_use id reuse, SSE quirks, etc.) — none of which a generic LLM gateway has, because none of them speak this protocol. + +The maintainer evaluated extending OCP to support OpenAI / Gemini / OpenRouter / Together / Groq / Ollama — i.e., turning OCP into a multi-provider gateway resembling Helicone, LiteLLM, OpenRouter, or Portkey. The motivation for that extension: reduce dependency on Anthropic, broaden OCP's commercial surface, and stop being grayscale-positioned (the `cli.js` spawn pattern depends on the local Pro/Max subscription, which Anthropic could fingerprint and disable). + +The honest engineering estimate for that extension: + +| Phase | Net New LOC | Calendar Time (part-time) | +|---|---|---| +| Provider abstraction + OpenAI | ~1230 + schema migration | 2 weeks | +| Add Gemini | ~550 | +1.5 weeks | +| OpenAI-compatible family (OpenRouter / Together / Groq) | ~300 | +1 week | +| Tests, docs, hardening | — | +1.5 weeks | +| **Multi-provider v1** | ~2080 | **~7 weeks focused** | + +That number is not the real cost. The real cost is **strategic**: + +1. **Loss of unique value.** `cli.js` behavior alignment is meaningless for OpenAI / Gemini / Ollama traffic. Going multi-provider means OCP's only moat applies to ~30% of its surface; the other 70% is generic gateway code already done better by Helicone / LiteLLM. + +2. **Hybrid architecture awkwardness.** A multi-provider OCP would have two paths: `spawn(cli.js)` for Claude (still grayscale, depends on Pro subscription), and direct API call for everyone else (clean, BYOK). Customers asking "what is OCP?" would hear two different answers depending on which model they pick. This is worse than either pure path. + +3. **Direct competition with funded incumbents.** Helicone (~$5M raised, YC W23), OpenRouter (~$1B valuation), LiteLLM (significant enterprise revenue), Portkey, Langfuse, Cloudflare AI Gateway — all already do multi-provider gateway with mature dashboards, audit logs, SOC2, and team features. OCP would enter that market 2+ years late with one engineer. + +4. **The grayscale problem isn't solved by adding providers.** As long as OCP keeps the `cli.js` spawn path for Anthropic, it remains grayscale for that path; adding OpenAI alongside doesn't make the Anthropic path any less dependent on a Pro/Max subscription that wasn't licensed for proxying. + +The maintainer's separate decision (recorded in personal notes, not this repo) is that **OCP itself will not be commercialized**; it will remain a personal power tool plus open-source contribution. Any commercial gateway work, if pursued, will start from a clean codebase with BYOK from day one — not from OCP. + +Given that, the multi-provider extension would buy OCP nothing: not a moat, not commercial readiness, not even meaningfully better personal utility (the maintainer overwhelmingly uses Claude). + +## Decision + +OCP stays single-provider. Specifically: + +1. **No new providers added to `server.mjs`.** The dispatch path remains `spawn(cli.js) → api.anthropic.com`. Pull requests that introduce a `providers/` directory or a model-to-provider router are declined on the basis of this ADR. + +2. **`models.json` schema stays Anthropic-only.** No `provider` field, no per-model cost/capability metadata that anticipates other providers. If non-Anthropic models ever need to be referenced (e.g., for OpenClaw provider list completeness), they live in a separate file or in OpenClaw's own config — not in OCP's SPOT. + +3. **Cache improvements are in scope.** The existing response cache (in `keys.mjs`: `cacheHash` / `getCachedResponse` / `setCachedResponse` / `clearCache`) is acceptable to upgrade with stream replay, stampede protection (singleflight), per-key isolation, and Anthropic `cache_control` awareness. These reinforce the single-provider position; they do not create provider-extension surface area. + +4. **Anthropic alignment work continues to be encouraged.** Anything that deepens `cli.js` behavior alignment — session lifetime, tool_use id semantics, SSE behavior, multi-account routing, model-tier observability — is the project's actual value and should be prioritized over generic-gateway features. + +5. **Commercial work, if pursued, starts elsewhere.** A separate repository, separate name, BYOK from day one, no `cli.js` spawn. That repo is out of scope for OCP and is not bound by this ADR. + +## Consequences + +**Positive** + +- Project scope stays bounded. The maintainer can keep evolving OCP at part-time pace without the multi-provider maintenance burden (every provider's API breaks at some point and demands attention). +- The unique value (`cli.js` alignment) is preserved and continues to compound — every new alignment fix increases OCP's distance from generic gateways. +- Future contributors reading the code see one architecture, not a hybrid; debugging stays tractable. +- Decisions about commercialization are decoupled from OCP's technical evolution. OCP can stay grayscale-personal-tool indefinitely without that being a blocker for any future commercial product. + +**Negative** + +- OCP cannot serve any user who needs OpenAI / Gemini / local LLM access. Those users must route through a different gateway (Helicone, LiteLLM, OpenRouter) or call providers directly. +- If Anthropic substantially changes `cli.js` (e.g., adds client attestation, removes the spawn-and-forward pattern, or migrates `claude` to a non-CLI form factor), OCP's core architecture breaks and there is no second backend to fall back to. +- The maintainer must resist a recurring temptation: "while I'm in here, let me just add OpenAI." The whole point of this ADR is to make that temptation cost a documented amendment, not a quiet PR. + +**Neutral** + +- This ADR records a non-decision in code: nothing in `server.mjs` changes today. Its purpose is to make future contributors (including the maintainer) explain themselves before going against it. Per the project's PR template and Iron Rule 11, an amendment to this ADR is the gating step before any provider-extension PR. + +## Trigger conditions for revisiting this ADR + +This ADR should be revisited (and possibly amended or superseded) if any of the following occur: + +1. Anthropic ships a feature that breaks the `cli.js` spawn pattern OCP depends on, and the maintainer wants to keep OCP useful. +2. The maintainer makes a deliberate decision to commercialize OCP (rather than start a separate codebase). This requires explicit re-scoping; "let me try" is not enough. +3. A genuine user need emerges — e.g., the maintainer themselves starts using OpenAI / Gemini frequently from Claude Code workflows — that single-provider OCP cannot serve. + +In all three cases, the response is **first amend this ADR**, then write code. Order is not optional. diff --git a/docs/adr/README.md b/docs/adr/README.md index 0f6cfc2..1f30a80 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -21,6 +21,7 @@ New ADRs increment from the highest existing number. Filenames are | [0002](0002-alignment-constitution.md) | Alignment Constitution | The `ALIGNMENT.md` constitution: why every `server.mjs` change requires `cli.js` citation + independent reviewer + CI blacklist pass. Background: the 2026-04-11 drift incident. | | [0003](0003-models-json-spot.md) | `models.json` as SPOT | Why model IDs / aliases / context windows live in a single JSON file (not duplicated in `server.mjs` and `setup.mjs` arrays). v3.11.0 refactor. | | [0004](0004-openclaw-auto-sync.md) | OpenClaw Auto-Sync | Why `scripts/sync-openclaw.mjs` runs on `ocp update`, what its scope boundary is (writes only `models.providers["claude-local"].models` and `agents.defaults.models["claude-local/*"]`), and the idempotency contract. | +| [0005](0005-no-multi-provider.md) | No Multi-Provider | Why OCP stays single-provider (Anthropic-via-cli.js) and does not extend to OpenAI / Gemini / OpenRouter. Cost estimate: ~7 weeks for a v1 that buys neither moat nor commercial readiness. Separate commercial work starts in a separate repo. | ## When to write a new ADR diff --git a/docs/superpowers/specs/2026-05-07-cache-upgrade-design.md b/docs/superpowers/specs/2026-05-07-cache-upgrade-design.md new file mode 100644 index 0000000..83f5f2e --- /dev/null +++ b/docs/superpowers/specs/2026-05-07-cache-upgrade-design.md @@ -0,0 +1,147 @@ +# Design: Response Cache Upgrade (Per-Key Isolation, cache_control Bypass, Chunked Stream Replay, Singleflight) + +**Date:** 2026-05-07 +**Status:** Draft (awaiting maintainer approval) +**Target version:** v3.13.0 (minor — internal correctness/concurrency improvements; no new public env vars or endpoints) +**Driving ADR:** [ADR 0005 — No Multi-Provider](../../adr/0005-no-multi-provider.md), decision §3 ("Cache improvements are in scope") + +--- + +## Overview + +OCP already has a response cache (`keys.mjs:296` `cacheHash` / `keys.mjs:311` `getCachedResponse` / `keys.mjs:324` `setCachedResponse`), wired into the proxy core at `server.mjs:1220` (non-streaming path read), `server.mjs:1227` (cache-hit-on-streaming-request replay), and `server.mjs:683` (streaming write-back). Today it has four functional gaps. This PR pair closes all four, in two minimum-reviewable units, **without changing the public API surface**. + +| Gap | Impact today | Fix lands in | +|---|---|---| +| All keys share one cache pool | Key A's cache hit can leak Key B's prompt response | PR-A | +| Anthropic `cache_control` markers not detected | OCP cache may interfere with Anthropic prompt caching that the user explicitly requested | PR-A | +| Stream cache hit replays whole content in one SSE chunk | Downstream renders all-at-once; some SDKs misbehave on huge single deltas | PR-A | +| Concurrent identical cache misses all spawn `cli.js` independently | Cache stampede: N requests → N spawns → N billable calls | PR-B | + +--- + +## Constitutional alignment (ALIGNMENT.md) + +**`cli.js` does not perform response caching at the proxy layer.** The OCP response cache is a value-add operation that exists only inside OCP, between the wire (clients ↔ OCP) and the spawn (OCP ↔ `cli.js`). It does not introduce, rename, or alter any endpoint, header, request field, or response field that `cli.js` emits or expects. Cache hits return content byte-identical to what `cli.js` returned on the original miss, with the same `chat.completion` / `chat.completion.chunk` shape — **no client-observable wire shape change**. + +This PR pair extends the existing cache (introduced in earlier commits) without expanding its surface. No new endpoints. No new headers. No new env vars exposed publicly (we add internal counters readable via the existing `/cache/stats` endpoint, but the response shape only gains numeric fields, not new structural fields). + +Per Rule 1 / Rule 5: every commit body in this PR pair will state the absence of `cli.js` reference explicitly and justify scope under Rule 2's value-add carve-out for non-wire-affecting proxy operations. + +--- + +## Key decisions (with rationale) + +### D1. Per-key isolation via hash input, not schema column + +`cacheHash` gains an optional `keyId` input. Distinct `keyId` values produce distinct hashes for the same prompt, so SQLite-level isolation falls out for free without a schema change. + +**Rationale.** Adding a `key_id` column to `response_cache` requires either (a) dropping the existing `hash UNIQUE` index and replacing with a composite `(hash, key_id) UNIQUE`, which SQLite cannot do via plain `ALTER TABLE` and would require a table-rebuild migration, or (b) tolerating duplicate `hash` rows, which contradicts the existing schema comment and breaks `setCachedResponse`'s `ON CONFLICT(hash)` upsert clause. + +The hash-input approach is reversible (we can switch to a schema column later if analytics across keys becomes a real need) and zero-risk on the SQL plane. The trade-off — losing the ability to query "which keys have cached this prompt?" — has no current consumer. + +**Hash input format.** `cacheHash` prepends a version tag and key tag before the existing inputs: + +``` +v2|k:||...rest as today +``` + +The `v2` prefix means existing v1-format rows in the cache table no longer hash-match any new request. They are abandoned, not deleted; the existing TTL-based `clearCache(CACHE_TTL)` cleanup interval at `server.mjs:185` reaps them within one TTL window. **No migration step is needed.** This is acceptable because the cache is by definition ephemeral and best-effort. + +**Anonymous fallback.** When the request has no authenticated key (`req._authKeyId === undefined`), `keyId` is `"anon"`. Anonymous-mode users (PROXY_ANONYMOUS_KEY or no auth) share one anonymous pool, which preserves the only legitimate today-multi-user use case (a household running OCP without per-user keys). If this becomes a problem we can add per-IP scoping later, but anonymous-pool sharing is acceptable for v1 because anonymous mode is fundamentally a trust-everyone-on-LAN posture. + +### D2. `cache_control` bypass: detect anywhere, skip OCP cache entirely + +If any element in `messages` (top-level or nested in `content` arrays) carries a `cache_control` field, OCP sets `req._cacheHash = null` and skips both lookup and write-back. + +**Rationale.** Anthropic's [prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) is opt-in by client-side annotation. A user who annotates `cache_control: { type: "ephemeral" }` is explicitly requesting that *Anthropic's* cache serve the call (and is paying the reduced cache-read pricing). Layering OCP's response cache on top in this case is wrong on two counts: + +1. The user's intent is "cache at provider, not at proxy." OCP overruling that intent silently is the same drift family as the 2026-04-11 incident — proxy invents behavior the upstream surface doesn't request. +2. OCP cache hits would make `usage.cache_read_input_tokens` (the client-observable signal that prompt caching worked) appear inconsistent — sometimes present, sometimes absent — depending on whether OCP cached. + +Detection is purely structural: walk `messages`, for each `m` check `m.cache_control` (rare top-level form) and if `m.content` is an array, check each part. No semantic interpretation; if the field is present, we bypass. + +**Implementation site.** A small helper `hasCacheControl(messages)` exported from `keys.mjs`, called in `handleChatCompletions` immediately before the existing `cacheHash` call. If it returns true, we skip the cache-lookup branch entirely. + +### D3. Chunked stream replay (80 chars/chunk, no artificial delay) + +Today's cache-hit-on-streaming-request branch (`server.mjs:1227–1237`) sends the entire cached content in a single `delta.content` chunk. This works for spec-compliant SSE clients but visibly degrades the UX (no incremental render) and has tripped at least one buggy SDK in the wild that assumes deltas are small. + +The fix splits cached content into ~80-character substrings, each sent as a separate `chat.completion.chunk` SSE event. **No artificial delay between chunks** — they ship as fast as `res.write` accepts. This preserves OCP's "ship as fast as possible" disposition; we are simulating *the chunk shape* of streaming, not the *latency*. + +**Why 80 chars?** Compromise: small enough that even a multi-paragraph cached response yields >5 chunks (visible incremental render), large enough that even a 4 KB response only produces 50 chunks (not 4000 single-char events). Tunable later via internal constant; not exposed as env var per scope-creep avoidance. + +**Boundary safety.** UTF-8 multibyte characters: we slice by `Array.from(content)` (so each iteration step is a full code point) and group every 80 code points. This avoids producing invalid UTF-8 mid-character. + +### D4. Singleflight stampede protection: in-process Map, all-or-nothing failure + +`keys.mjs` exports `singleflight(hash, fn)`. An in-memory `Map` deduplicates concurrent identical cache-miss flows. The first request executes `fn()`; concurrent requests with the same hash receive the same promise. When the promise settles (resolve or reject), the map entry is deleted. + +**Rationale (single-process scope).** OCP runs as a single Node.js process per host. A `Map` is sufficient. Adding Redis or another shared store would be the start of a multi-instance evolution, which is out of scope per ADR 0005 (OCP is a personal power tool, not a horizontally-scaled SaaS). + +**All-or-nothing failure semantics.** When the leader's `fn()` rejects, all followers receive the same rejection. The alternative — letting followers retry independently after a leader failure — risks N retries of an already-broken upstream, which is exactly what stampede protection was meant to prevent. Followers can retry at the *next* request, with idle backoff handled by the client. This matches Go's `golang.org/x/sync/singleflight` reference behavior. + +**Streaming caveat.** Singleflight wraps the *non-streaming* code path only in PR-B. For streaming, deduplicating concurrent identical streaming requests is materially harder (we'd need to fan out one upstream stream to N downstream connections in real time, with backpressure). It's also a less common case (cache stampedes typically come from non-streaming batch jobs hitting the proxy in parallel). Streaming dedup is **explicitly out of scope** for this PR pair; leave a TODO comment in `callClaudeStreaming` for a future ticket. + +**Map size unboundedness.** In normal operation the map is empty most of the time (entries delete on Promise settlement). Pathological case: an upstream call that hangs forever leaks one Map entry per stuck request. The existing `TIMEOUT` guard on `callClaude` (server.mjs spawn timeout) bounds this — the Promise will reject (timeout) within `TIMEOUT` ms, and the entry clears. No additional sweep needed. + +--- + +## PR boundaries + +### PR-A — Foundation (D1 + D2 + D3) + +**Files touched:** +- `keys.mjs`: extend `cacheHash` with optional `keyId`/version prefix; add `hasCacheControl(messages)` helper +- `server.mjs`: pass `req._authKeyId` to `cacheHash`; check `hasCacheControl` and bypass; chunk cache-hit replay at line 1227–1237 +- `test-features.mjs`: add cases for keyId isolation, cache_control bypass, chunked replay shape + +**LOC budget:** ~80 production + ~50 test +**Risk:** Low — all changes are additive or guard-clause; existing cache behavior preserved when `keyId` defaults to "anon" and no `cache_control` present. +**Backward compat:** v1-format hashes naturally orphan; TTL cleanup reaps within one window; no migration script. + +### PR-B — Concurrency (D4) + +**Files touched:** +- `keys.mjs`: add `singleflight(hash, fn)` and `getInflightStats()` exports +- `server.mjs`: wrap non-streaming cache-miss path through `singleflight`; add inflight count to `/cache/stats` response +- `test-features.mjs`: add concurrent-request test that asserts only 1 spawn occurs for N=10 simultaneous identical requests + +**LOC budget:** ~70 production + ~40 test +**Risk:** Medium — concurrency code is harder to reason about; mitigation is an explicit test case for the dedup behavior. +**Streaming explicitly out of scope:** TODO comment placed in `callClaudeStreaming` for follow-up ticket. + +--- + +## Testing strategy + +**Unit-ish (in `test-features.mjs`):** +1. `cacheHash` with two different `keyId` values → different hashes +2. `cacheHash` v2 prefix present in output (sanity check) +3. `hasCacheControl` returns true for top-level `cache_control` and for nested in `content[]` +4. `hasCacheControl` returns false for benign messages +5. Chunked replay: cached "abcdefgh..." (160 chars) produces 2 deltas + +**Integration (manual smoke before merge):** +1. Set `CLAUDE_CACHE_TTL=60000`; create key A and key B; identical prompt from each → both spawn fresh; second-call from same key → cache hit +2. Send a message with `cache_control` annotation → OCP logs `cache_skipped: cache_control_present`; no cache write +3. Streaming cache hit visibly produces multiple SSE deltas (`curl -N | grep "data: "` shows >1 lines) + +**Concurrent (PR-B only):** +1. Spawn 10 simultaneous identical non-streaming requests; assert (via `/cache/stats` inflight peak or via a process spawn counter) only 1 `cli.js` spawn occurred + +--- + +## Out of scope (deliberately deferred) + +- **Streaming singleflight** — see D4 streaming caveat. TODO in code. +- **Semantic cache** (embedding-based near-match) — needs an embedding provider + vector index. Punt to v3.14+ if there's user demand. +- **Cross-process cache** (Redis backend) — violates ADR 0005's "personal power tool" posture. +- **Cache versioning by model ID hash** — model upgrades currently invalidate cache organically because model is in the hash; if Anthropic ever silently changes a model's behavior without a model ID bump, that's a separate alignment problem. +- **Per-key cache TTL override** — single global TTL (existing `CLAUDE_CACHE_TTL`) is fine; per-key TTL is a knob no one has asked for. + +--- + +## Rollback plan + +If either PR introduces a regression, the rollback is a clean git revert. The cache layer is opt-in (default `CLAUDE_CACHE_TTL=0` = disabled), so users who never enabled the cache are unaffected by any cache-layer regression. Users who *had* enabled the cache lose only ephemeral state on revert. No persistent on-disk state is reshaped by this PR pair (we explicitly avoid schema migrations per D1 rationale). diff --git a/keys.mjs b/keys.mjs index f8ada0d..c597fe7 100644 --- a/keys.mjs +++ b/keys.mjs @@ -292,9 +292,13 @@ export function getKeyQuota(keyId) { // ── Response cache ── -// Generate a cache key from model + messages + request params that affect output +// Generate a cache key from model + messages + request params that affect output. +// opts.keyId isolates per-API-key cache pools (v2 hash format). +// When keyId is absent/null/empty, falls back to "anon" (shared anonymous pool). export function cacheHash(model, messages, opts = {}) { + const keyId = opts.keyId || "anon"; const h = createHash("sha256"); + h.update(`v2|k:${keyId}|`); h.update(model); if (opts.temperature != null) h.update(`t:${opts.temperature}`); if (opts.max_tokens != null) h.update(`mt:${opts.max_tokens}`); @@ -306,6 +310,22 @@ export function cacheHash(model, messages, opts = {}) { return h.digest("hex"); } +// Check whether any message (or content part) carries an Anthropic cache_control field. +// If true, OCP should skip its own cache to avoid interfering with prompt-caching intent. +export function hasCacheControl(messages) { + for (const m of messages || []) { + if (m && typeof m === "object") { + if (m.cache_control) return true; + if (Array.isArray(m.content)) { + for (const part of m.content) { + if (part && typeof part === "object" && part.cache_control) return true; + } + } + } + } + return false; +} + // Look up a cached response. Returns { response, hits } or null. // Also updates last_hit_at and increments hits counter on hit. export function getCachedResponse(hash, ttlMs) { diff --git a/server.mjs b/server.mjs index cd1f045..72c936d 100644 --- a/server.mjs +++ b/server.mjs @@ -34,7 +34,7 @@ import { readFileSync, accessSync, existsSync, constants } from "node:fs"; import { fileURLToPath } from "node:url"; import { dirname, join } from "node:path"; import { homedir } from "node:os"; -import { validateKey, recordUsage, getUsageByKey, getUsageTimeline, getRecentUsage, createKey, listKeys, revokeKey, closeDb, checkQuota, updateKeyQuota, getKeyQuota, findKey, cacheHash, getCachedResponse, setCachedResponse, clearCache, getCacheStats } from "./keys.mjs"; +import { validateKey, recordUsage, getUsageByKey, getUsageTimeline, getRecentUsage, createKey, listKeys, revokeKey, closeDb, checkQuota, updateKeyQuota, getKeyQuota, findKey, cacheHash, getCachedResponse, setCachedResponse, clearCache, getCacheStats, hasCacheControl } from "./keys.mjs"; const __dirname = dirname(fileURLToPath(import.meta.url)); const _pkg = JSON.parse(readFileSync(join(__dirname, "package.json"), "utf8")); @@ -1218,30 +1218,43 @@ async function handleChatCompletions(req, res) { // Cache check (only when cache is enabled and no active conversation/session) if (CACHE_TTL > 0 && !conversationId) { - const hash = cacheHash(model, messages, { temperature: parsed.temperature, max_tokens: parsed.max_tokens, top_p: parsed.top_p }); - req._cacheHash = hash; // store for later write-back - try { - const cached = getCachedResponse(hash, CACHE_TTL); - if (cached) { - logEvent("info", "cache_hit", { model, hash: hash.slice(0, 12), hits: cached.hits }); - if (stream) { - // Simulate streaming for cached response - const id = `chatcmpl-${randomUUID()}`; - const created = Math.floor(Date.now() / 1000); - res.writeHead(200, { "Content-Type": "text/event-stream", "Cache-Control": "no-cache", "Connection": "keep-alive", "X-Accel-Buffering": "no" }); - sendSSE(res, { id, object: "chat.completion.chunk", created, model, choices: [{ index: 0, delta: { role: "assistant" }, finish_reason: null }] }); - sendSSE(res, { id, object: "chat.completion.chunk", created, model, choices: [{ index: 0, delta: { content: cached.response }, finish_reason: null }] }); - sendSSE(res, { id, object: "chat.completion.chunk", created, model, choices: [{ index: 0, delta: {}, finish_reason: "stop" }] }); - res.write("data: [DONE]\n\n"); - res.end(); - return; - } else { - const id = `chatcmpl-${randomUUID()}`; - return completionResponse(res, id, model, cached.response); + // D2: skip OCP cache entirely when messages carry cache_control annotations; + // the client is requesting Anthropic-side prompt caching, not OCP-layer caching. + if (hasCacheControl(messages)) { + req._cacheHash = null; + logEvent("info", "cache_skipped", { reason: "cache_control_present" }); + } else { + // D1: include keyId in hash to isolate per-key cache pools (v2 format) + const hash = cacheHash(model, messages, { keyId: req._authKeyId, temperature: parsed.temperature, max_tokens: parsed.max_tokens, top_p: parsed.top_p }); + req._cacheHash = hash; // store for later write-back + try { + const cached = getCachedResponse(hash, CACHE_TTL); + if (cached) { + logEvent("info", "cache_hit", { model, hash: hash.slice(0, 12), hits: cached.hits }); + if (stream) { + // D3: replay cached content as chunked SSE stream (80 codepoints/chunk) + const CACHE_REPLAY_CHUNK_SIZE = 80; + const id = `chatcmpl-${randomUUID()}`; + const created = Math.floor(Date.now() / 1000); + res.writeHead(200, { "Content-Type": "text/event-stream", "Cache-Control": "no-cache", "Connection": "keep-alive", "X-Accel-Buffering": "no" }); + sendSSE(res, { id, object: "chat.completion.chunk", created, model, choices: [{ index: 0, delta: { role: "assistant" }, finish_reason: null }] }); + const codepoints = Array.from(cached.response); + for (let i = 0; i < codepoints.length; i += CACHE_REPLAY_CHUNK_SIZE) { + const chunk = codepoints.slice(i, i + CACHE_REPLAY_CHUNK_SIZE).join(""); + sendSSE(res, { id, object: "chat.completion.chunk", created, model, choices: [{ index: 0, delta: { content: chunk }, finish_reason: null }] }); + } + sendSSE(res, { id, object: "chat.completion.chunk", created, model, choices: [{ index: 0, delta: {}, finish_reason: "stop" }] }); + res.write("data: [DONE]\n\n"); + res.end(); + return; + } else { + const id = `chatcmpl-${randomUUID()}`; + return completionResponse(res, id, model, cached.response); + } } + } catch (e) { + logEvent("error", "cache_check_failed", { error: e.message }); } - } catch (e) { - logEvent("error", "cache_check_failed", { error: e.message }); } } diff --git a/test-features.mjs b/test-features.mjs index b3ad97e..5eccf93 100644 --- a/test-features.mjs +++ b/test-features.mjs @@ -3,7 +3,8 @@ * Integration test for Quota + Cache features. * Tests database layer functions directly — no server needed. */ -import { getDb, createKey, listKeys, validateKey, recordUsage, checkQuota, updateKeyQuota, getKeyQuota, findKey, cacheHash, getCachedResponse, setCachedResponse, clearCache, getCacheStats, closeDb } from "./keys.mjs"; +import { getDb, createKey, listKeys, validateKey, recordUsage, checkQuota, updateKeyQuota, getKeyQuota, findKey, cacheHash, getCachedResponse, setCachedResponse, clearCache, getCacheStats, closeDb, hasCacheControl } from "./keys.mjs"; +import { createHash } from "node:crypto"; import { strict as assert } from "node:assert"; import { unlinkSync } from "node:fs"; import { join } from "node:path"; @@ -253,6 +254,106 @@ test("clearCache with TTL only removes old entries", () => { clearCache(); }); +// ── PR-A: Per-key isolation (D1), cache_control bypass (D2), chunked replay (D3) ── +console.log("\nPR-A Cache Upgrade:"); + +const msgsBase = [{ role: "user", content: "Shared prompt text" }]; + +test("D1: cacheHash with two distinct keyIds produces different hashes", () => { + const h1 = cacheHash("sonnet", msgsBase, { keyId: "key-aaa" }); + const h2 = cacheHash("sonnet", msgsBase, { keyId: "key-bbb" }); + assert.notEqual(h1, h2); +}); + +test("D1: cacheHash with keyId=undefined and keyId='anon' produce the same hash", () => { + const hUndef = cacheHash("sonnet", msgsBase, { keyId: undefined }); + const hAnon = cacheHash("sonnet", msgsBase, { keyId: "anon" }); + assert.equal(hUndef, hAnon); +}); + +test("D1: cacheHash with keyId=null and keyId='anon' produce the same hash", () => { + const hNull = cacheHash("sonnet", msgsBase, { keyId: null }); + const hAnon = cacheHash("sonnet", msgsBase, { keyId: "anon" }); + assert.equal(hNull, hAnon); +}); + +test("D1: v2 prefix — hash differs from a v1-style baseline (no prefix)", () => { + // Reproduce a v1-style hash manually to confirm v2 differs + const v1 = createHash("sha256") + .update("sonnet") + .update(msgsBase[0].role) + .update(msgsBase[0].content) + .digest("hex"); + const v2 = cacheHash("sonnet", msgsBase, { keyId: "anon" }); + assert.notEqual(v1, v2); +}); + +test("D1: cacheHash is reproducible for same keyId (determinism)", () => { + const h1 = cacheHash("sonnet", msgsBase, { keyId: "key-xyz" }); + const h2 = cacheHash("sonnet", msgsBase, { keyId: "key-xyz" }); + assert.equal(h1, h2); +}); + +test("D2: hasCacheControl returns true for top-level cache_control on message", () => { + const msgs = [{ role: "user", cache_control: { type: "ephemeral" }, content: "hello" }]; + assert.equal(hasCacheControl(msgs), true); +}); + +test("D2: hasCacheControl returns true for nested cache_control in content array", () => { + const msgs = [{ role: "user", content: [{ type: "text", text: "x", cache_control: { type: "ephemeral" } }] }]; + assert.equal(hasCacheControl(msgs), true); +}); + +test("D2: hasCacheControl returns false for plain string content", () => { + const msgs = [{ role: "user", content: "plain string" }]; + assert.equal(hasCacheControl(msgs), false); +}); + +test("D2: hasCacheControl returns false for content array without cache_control", () => { + const msgs = [{ role: "user", content: [{ type: "text", text: "x" }] }]; + assert.equal(hasCacheControl(msgs), false); +}); + +test("D2: hasCacheControl handles null/empty input gracefully", () => { + assert.equal(hasCacheControl(null), false); + assert.equal(hasCacheControl([]), false); + assert.equal(hasCacheControl([null, undefined]), false); +}); + +// D3: chunked stream replay — verify the logic by simulating what server.mjs does +test("D3: 160-char cached response produces 2 chunks at 80 codepoints/chunk", () => { + const content = "a".repeat(160); + const CACHE_REPLAY_CHUNK_SIZE = 80; + const codepoints = Array.from(content); + const chunks = []; + for (let i = 0; i < codepoints.length; i += CACHE_REPLAY_CHUNK_SIZE) { + chunks.push(codepoints.slice(i, i + CACHE_REPLAY_CHUNK_SIZE).join("")); + } + assert.equal(chunks.length, 2); + assert.equal(chunks[0].length, 80); + assert.equal(chunks[1].length, 80); +}); + +test("D3: chunked replay uses Array.from — multibyte codepoints stay intact", () => { + // Each Chinese character is 1 codepoint but 3 UTF-8 bytes + const chinese = "你好世界".repeat(25); // 100 codepoints + const CACHE_REPLAY_CHUNK_SIZE = 80; + const codepoints = Array.from(chinese); + const chunks = []; + for (let i = 0; i < codepoints.length; i += CACHE_REPLAY_CHUNK_SIZE) { + chunks.push(codepoints.slice(i, i + CACHE_REPLAY_CHUNK_SIZE).join("")); + } + assert.equal(chunks.length, 2); + assert.equal(Array.from(chunks[0]).length, 80); + assert.equal(Array.from(chunks[1]).length, 20); + // Verify each character is a complete codepoint (no mojibake) + for (const chunk of chunks) { + for (const cp of Array.from(chunk)) { + assert.equal(cp.length <= 2, true); // surrogate pairs are length 2, single chars length 1 + } + } +}); + // ── Cleanup ── closeDb();