taodeng/ocp
1
0
Fork 0
mirror of https://github.com/dtzp555-max/ocp.git synced 2026-07-19 09:44:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(cache): per-key isolation, cache_control bypass, chunked stream replay (#65)

Browse Source 
* docs(governance): ADR 0005 (no multi-provider) + cache upgrade spec

Governance prelude for the cache upgrade work:

- docs/adr/0005-no-multi-provider.md — locks in the decision that
  OCP stays single-provider (Anthropic via cli.js spawn). Cache
  improvements are explicitly in scope (decision §3); multi-provider
  refactor is explicitly out of scope, with three documented trigger
  conditions for revisiting.

- docs/adr/README.md — index updated to reference 0005.

- docs/superpowers/specs/2026-05-07-cache-upgrade-design.md — design
  for the cache upgrade work split into PR-A (per-key isolation,
  cache_control bypass, chunked stream replay) and PR-B (singleflight
  stampede protection). Each design decision has a written rationale.

server.mjs is not modified; this commit is doc-only and therefore
exempt from the cli.js citation requirement (ALIGNMENT.md Rule 5
applies only to commits that touch server.mjs).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(cache): per-key isolation, cache_control bypass, chunked stream replay

cli.js does not perform response caching at the proxy layer. OCP's response
cache is a value-add operation internal to OCP, between the wire and the
cli.js spawn. It does not introduce, rename, or alter any endpoint, header,
request field, or response field that cli.js emits or expects — this change
qualifies under ALIGNMENT.md Rule 2's value-add carve-out for non-wire-
affecting proxy operations. no client-observable wire shape change.

Spec: docs/superpowers/specs/2026-05-07-cache-upgrade-design.md

D1 — Per-key cache isolation (cacheHash v2 format)
  keys.mjs: prepend `v2|k:<keyId or "anon">|` before existing hash fields.
  Backward-compatible: absent/null/empty keyId folds to "anon".
  v1-format rows in response_cache are abandoned naturally; TTL cleanup at
  server.mjs:185 reaps them within one window. No migration needed.
  server.mjs: pass keyId: req._authKeyId at the single cacheHash call site
  (line ~1221).

D2 — cache_control bypass
  keys.mjs: export hasCacheControl(messages) — walks messages and nested
  content arrays for presence of cache_control field.
  server.mjs: if hasCacheControl(messages) is true, set req._cacheHash = null
  and log cache_skipped{reason: cache_control_present}; existing
  `if (CACHE_TTL > 0 && req._cacheHash)` guards on write-back handle the skip.

D3 — Chunked stream replay (80 codepoints/chunk, no artificial delay)
  server.mjs: replace single-chunk cached.response emission with an
  Array.from(cached.response) loop in steps of CACHE_REPLAY_CHUNK_SIZE=80.
  Array.from ensures multibyte UTF-8 codepoints (e.g. CJK) are never split.

Tests: 12 new cases in test-features.mjs (36 total, 0 failed).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: dtzp555 <dtzp555@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
dtzp555-max
2026-05-07 13:50:50 +10:00
committed by GitHub
co-authored by taodeng Claude Opus 4.7
parent c998d21a4f
commit 16eeb66557
6 changed files with 386 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr/0005-no-multi-provider.md
+79
View File
@@ -0,0 +1,79 @@
# 0005 — OCP Stays Single-Provider; No Multi-Provider Refactor
- **Date**: 2026-05-06
- **Status**: Accepted
- **Authors**: project maintainer (with AI advisory drafting)
- **Related**: ADR 0002 (Alignment Constitution), ADR 0003 (`models.json` SPOT)
## Context
OCP's `server.mjs` reached 1667 lines and now provides response cache, per-key quota, session tracking, model-level stats, and SSE heartbeat — all targeting a single backend path: `spawn` the locally installed `cli.js` and let it transact with `api.anthropic.com`. This architecture is the source of OCP's only real differentiator: **`cli.js` behavior-level alignment** (session create-vs-resume semantics, tool_use id reuse, SSE quirks, etc.) — none of which a generic LLM gateway has, because none of them speak this protocol.
The maintainer evaluated extending OCP to support OpenAI / Gemini / OpenRouter / Together / Groq / Ollama — i.e., turning OCP into a multi-provider gateway resembling Helicone, LiteLLM, OpenRouter, or Portkey. The motivation for that extension: reduce dependency on Anthropic, broaden OCP's commercial surface, and stop being grayscale-positioned (the `cli.js` spawn pattern depends on the local Pro/Max subscription, which Anthropic could fingerprint and disable).
The honest engineering estimate for that extension:
| Phase | Net New LOC | Calendar Time (part-time) |
|---|---|---|
| Provider abstraction + OpenAI | ~1230 + schema migration | 2 weeks |
| Add Gemini | ~550 | +1.5 weeks |
| OpenAI-compatible family (OpenRouter / Together / Groq) | ~300 | +1 week |
| Tests, docs, hardening | — | +1.5 weeks |
| **Multi-provider v1** | ~2080 | **~7 weeks focused** |
That number is not the real cost. The real cost is **strategic**:
1. **Loss of unique value.** `cli.js` behavior alignment is meaningless for OpenAI / Gemini / Ollama traffic. Going multi-provider means OCP's only moat applies to ~30% of its surface; the other 70% is generic gateway code already done better by Helicone / LiteLLM.
2. **Hybrid architecture awkwardness.** A multi-provider OCP would have two paths: `spawn(cli.js)` for Claude (still grayscale, depends on Pro subscription), and direct API call for everyone else (clean, BYOK). Customers asking "what is OCP?" would hear two different answers depending on which model they pick. This is worse than either pure path.
3. **Direct competition with funded incumbents.** Helicone (~$5M raised, YC W23), OpenRouter (~$1B valuation), LiteLLM (significant enterprise revenue), Portkey, Langfuse, Cloudflare AI Gateway — all already do multi-provider gateway with mature dashboards, audit logs, SOC2, and team features. OCP would enter that market 2+ years late with one engineer.
4. **The grayscale problem isn't solved by adding providers.** As long as OCP keeps the `cli.js` spawn path for Anthropic, it remains grayscale for that path; adding OpenAI alongside doesn't make the Anthropic path any less dependent on a Pro/Max subscription that wasn't licensed for proxying.
The maintainer's separate decision (recorded in personal notes, not this repo) is that **OCP itself will not be commercialized**; it will remain a personal power tool plus open-source contribution. Any commercial gateway work, if pursued, will start from a clean codebase with BYOK from day one — not from OCP.
Given that, the multi-provider extension would buy OCP nothing: not a moat, not commercial readiness, not even meaningfully better personal utility (the maintainer overwhelmingly uses Claude).
## Decision
OCP stays single-provider. Specifically:
1. **No new providers added to `server.mjs`.** The dispatch path remains `spawn(cli.js) → api.anthropic.com`. Pull requests that introduce a `providers/` directory or a model-to-provider router are declined on the basis of this ADR.
2. **`models.json` schema stays Anthropic-only.** No `provider` field, no per-model cost/capability metadata that anticipates other providers. If non-Anthropic models ever need to be referenced (e.g., for OpenClaw provider list completeness), they live in a separate file or in OpenClaw's own config — not in OCP's SPOT.
3. **Cache improvements are in scope.** The existing response cache (in `keys.mjs`: `cacheHash` / `getCachedResponse` / `setCachedResponse` / `clearCache`) is acceptable to upgrade with stream replay, stampede protection (singleflight), per-key isolation, and Anthropic `cache_control` awareness. These reinforce the single-provider position; they do not create provider-extension surface area.
4. **Anthropic alignment work continues to be encouraged.** Anything that deepens `cli.js` behavior alignment — session lifetime, tool_use id semantics, SSE behavior, multi-account routing, model-tier observability — is the project's actual value and should be prioritized over generic-gateway features.
5. **Commercial work, if pursued, starts elsewhere.** A separate repository, separate name, BYOK from day one, no `cli.js` spawn. That repo is out of scope for OCP and is not bound by this ADR.
## Consequences
**Positive**
- Project scope stays bounded. The maintainer can keep evolving OCP at part-time pace without the multi-provider maintenance burden (every provider's API breaks at some point and demands attention).
- The unique value (`cli.js` alignment) is preserved and continues to compound — every new alignment fix increases OCP's distance from generic gateways.
- Future contributors reading the code see one architecture, not a hybrid; debugging stays tractable.
- Decisions about commercialization are decoupled from OCP's technical evolution. OCP can stay grayscale-personal-tool indefinitely without that being a blocker for any future commercial product.
**Negative**
- OCP cannot serve any user who needs OpenAI / Gemini / local LLM access. Those users must route through a different gateway (Helicone, LiteLLM, OpenRouter) or call providers directly.
- If Anthropic substantially changes `cli.js` (e.g., adds client attestation, removes the spawn-and-forward pattern, or migrates `claude` to a non-CLI form factor), OCP's core architecture breaks and there is no second backend to fall back to.
- The maintainer must resist a recurring temptation: "while I'm in here, let me just add OpenAI." The whole point of this ADR is to make that temptation cost a documented amendment, not a quiet PR.
**Neutral**
- This ADR records a non-decision in code: nothing in `server.mjs` changes today. Its purpose is to make future contributors (including the maintainer) explain themselves before going against it. Per the project's PR template and Iron Rule 11, an amendment to this ADR is the gating step before any provider-extension PR.
## Trigger conditions for revisiting this ADR
This ADR should be revisited (and possibly amended or superseded) if any of the following occur:
1. Anthropic ships a feature that breaks the `cli.js` spawn pattern OCP depends on, and the maintainer wants to keep OCP useful.
2. The maintainer makes a deliberate decision to commercialize OCP (rather than start a separate codebase). This requires explicit re-scoping; "let me try" is not enough.
3. A genuine user need emerges — e.g., the maintainer themselves starts using OpenAI / Gemini frequently from Claude Code workflows — that single-provider OCP cannot serve.
In all three cases, the response is **first amend this ADR**, then write code. Order is not optional.
docs/adr/README.md
+1
View File
@@ -21,6 +21,7 @@ New ADRs increment from the highest existing number. Filenames are
| [0002](0002-alignment-constitution.md) | Alignment Constitution | The `ALIGNMENT.md` constitution: why every `server.mjs` change requires `cli.js` citation + independent reviewer + CI blacklist pass. Background: the 2026-04-11 drift incident. |
| [0003](0003-models-json-spot.md) | `models.json` as SPOT | Why model IDs / aliases / context windows live in a single JSON file (not duplicated in `server.mjs` and `setup.mjs` arrays). v3.11.0 refactor. |
| [0004](0004-openclaw-auto-sync.md) | OpenClaw Auto-Sync | Why `scripts/sync-openclaw.mjs` runs on `ocp update`, what its scope boundary is (writes only `models.providers["claude-local"].models` and `agents.defaults.models["claude-local/*"]`), and the idempotency contract. |
| [0005](0005-no-multi-provider.md) | No Multi-Provider | Why OCP stays single-provider (Anthropic-via-cli.js) and does not extend to OpenAI / Gemini / OpenRouter. Cost estimate: ~7 weeks for a v1 that buys neither moat nor commercial readiness. Separate commercial work starts in a separate repo. |
## When to write a new ADR