diff --git a/PLAN-v4.md b/PLAN-v4.md new file mode 100644 index 0000000..1aa14de --- /dev/null +++ b/PLAN-v4.md @@ -0,0 +1,439 @@ +# OCP v4.0 开发计划 — Agent 成本控制器 + +## 背景 + +OCP 当前架构:`Tool → OCP (localhost:3456) → Claude CLI → Anthropic` + +核心风险:Claude CLI proxy 属于灰色地带,Anthropic 随时可能封堵。 + +核心资产(不依赖 CLI 的部分): +- OpenAI 兼容 API 层 +- Session 管理(会话隔离、TTL、复用) +- Usage 监控(plan limits、per-model stats、实时查询) +- 运行时调参(timeout、并发、prompt 限制) +- `/ocp` 命令集(Telegram/Discord/CLI) +- 多实例并发控制 + +## 定位 + +> OCP is a cost-control and model-routing layer for AI agents, optimized for subscription-based usage. + +**不是 proxy,不是 gateway,不是 LiteLLM clone。** +**是:让 Agent 用模型这件事变得可控 — 尤其是成本。** + +核心差异点:OCP 有 agent 上下文,知道是谁在请求、在做什么任务、已经花了多少。LiteLLM 只看到 HTTP 请求。 + +## 目标 + +将 OCP 从 "Claude CLI proxy" 重构为 "Agent 成本控制器",支持可扩展的多后端,保留全部管理能力。 + +--- + +## ⚠️ 第零原则:不动现有功能 + +OCP 作为 Claude CLI proxy 是当前的核心价值和用户吸引力。所有 v4 重构必须遵守: + +- **外部 API 不变** — `/v1/chat/completions`、`/usage`、`/health`、`/settings` 签名不变 +- **行为不变** — 现有用户升级后零感知,所有 session/timeout/并发逻辑保持原样 +- **逐步替换** — 先抽模块,跑稳了再切;任何时候回归测试不过就回滚 +- **新 backend 是增量** — 加 OpenAI/Ollama 是新能力,不改动 Claude CLI 路径 + +Phase 1 的定义就是:**拆完之后跟没拆一样。** 拆出问题就说明拆法有误。 + +--- + +## 架构关键决策(三轮 Design Review 定稿) + +### 决策 1: 路由分两层 — Model Registry + Agent Policy + +**三轮讨论结论:** +- 第一轮建议"删掉 model routing" → 错误,非 agent 客户端(Cline/Aider)只带 model 名 +- 反驳:model-to-backend mapping 是基础设施,不是 routing rule +- 校准:物理拆成两个模块,不要混在一起 + +**最终设计 — 两层分离:** + +``` +🧱 Layer 1: Model Registry(基础设施层) + model-registry.mjs + 静态映射,无条件,必须存在 + "claude-sonnet-4-6" → claude-cli backend + "gpt-4o" → openai backend + "qwen3" → ollama backend + +🧠 Layer 2: Agent Policy(策略层) + agent-router.mjs + 动态决策,可覆盖,可选 + agent "coder" → override to claude-cli + agent "interpreter" → override to ollama +``` + +**路由决策顺序(写死):** +``` +1. Agent policy override → 有 agent identity 且有配置?用它 +2. Model registry → 模型属于哪个 backend?路由过去 +3. Default backend → 都没匹配?全局默认 +4. Fallback chain → 上面选的挂了?降级 +``` + +### 决策 2: 统一 Streaming 协议 + +**所有 adapter 必须输出统一的内部 chunk 格式:** + +```javascript +{ + type: "delta" | "done" | "error", + content?: string, + model: string, + backend: string, + usage?: { promptTokens: number, completionTokens: number } +} +``` + +server.mjs 只处理统一协议 → 转换为 OpenAI SSE 输出。 + +### 决策 3: 成本标记 — actual / estimated / free + +```javascript +{ + cost: { + value: 0.23, + type: "actual" | "estimated" | "free" + } +} +``` + +| Backend | cost type | 来源 | +|---------|-----------|------| +| OpenAI API | actual | API 返回 usage | +| Claude CLI | estimated | prompt chars × 估价 | +| Ollama | free | 本地运行 | + +`/ocp cost` 输出明确标注类型,不误导用户。 + +### 决策 4: Fallback — 只在 First-Byte 前触发 + +```javascript +const allowFallback = !hasStartedStreaming; +``` + +Phase 3 严格限制:已输出 ≥1 个 delta → 不 fallback,直接报错。 +flag 预留,未来扩展 streaming fallback 只改判断条件。 + +### 决策 5: Agent Identity — 结构预留,实现从简 + +**优先级:** +``` +1. x-ocp-agent header → OpenClaw 自动附加 +2. Session metadata → 首次请求绑定,后续复用 +3. (future) Inferred → 按使用模式推断(预留接口) +4. default → 兜底 +``` + +**现在只实现 1 + 2 + 4。** 但接口预留 inferred 扩展点。 + +为什么重要:未来 Cline coding mode 和 chat mode 可能需要不同路由,如果没有 identity hook 以后很难加。 + +--- + +## Backend Adapter 设计 + +### 接口 + +```javascript +class BackendAdapter { + get id() // "claude-cli" | "openai-api" | "ollama" + get displayName() + get models() + get costType() // "actual" | "estimated" | "free" + get tier() // "core" | "community" + + async healthCheck() → { ok, message, latencyMs } + async *chatCompletion(request) → AsyncGenerator + async initialize(config) + async shutdown() +} +``` + +### Backend Tier 体系 + +OCP provides a pluggable backend adapter interface. Core adapters are maintained officially; community adapters are contributed by users. + +``` +/ocp backends 输出示例: + +claude-cli ✓ healthy 12ms (core) +openai ✓ healthy 89ms (core) +ollama ✓ healthy 5ms (core) +deepseek ✓ healthy 120ms (community) +``` + +架构不限制 backend 数量。我们主动维护 3 个 core adapter(Claude CLI、OpenAI 兼容、Ollama),接口开放,社区可贡献更多。 + +OpenAI 兼容的服务(DeepSeek、Groq、Together 等)直接复用 `OpenAiApiAdapter`,只需换 baseUrl + apiKey,不算新 adapter。 + +--- + +## 分阶段实施 + +### Phase 1: 后端抽象 + Model Registry(v4.0-alpha) +**目标:** 不改变外部行为,内部重构 + +- [ ] 定义 `BackendAdapter` 接口 + `UnifiedChunk` 类型 +- [ ] `model-registry.mjs` — 模型到 backend 的静态映射 + - `getBackendForModel(model) → backendId` +- [ ] `agent-router.mjs` — 路由决策引擎 + - `resolveBackend({ agent, model }) → backendId` + - 实现决策顺序:agent override → model registry → default → fallback +- [ ] 将 Claude CLI 逻辑封装为 `ClaudeCliAdapter` + - stdout → UnifiedChunk 转换 + - costType = "estimated" +- [ ] 抽取 `SessionManager`(从 server.mjs) +- [ ] 抽取 `StatsCollector`(per-agent, per-backend) +- [ ] Config v4 格式(带 version + validation): + +```json +{ + "version": "4.0", + "backends": { + "claude-cli": { + "type": "claude-cli", + "tier": "core", + "enabled": true, + "models": ["claude-sonnet-4-6", "claude-opus-4-6", "claude-haiku-4-5"], + "maxConcurrent": 4, + "timeout": { "firstByte": 120000, "overall": 300000 }, + "estimatedCostPerMTok": { "input": 3.0, "output": 15.0 } + } + }, + "agents": { + "default": { "preferred": "claude-cli", "fallback": null } + } +} +``` + +- [ ] 启动时 config validation +- [ ] `/ocp backends` 命令 +- [ ] 所有现有测试通过,行为不变 + +**验收标准:** 现有用户升级后零感知变化 + +### Phase 2: OpenAI API 后端(v4.0-beta) +**目标:** 第二个 backend,验证 adapter 抽象 + +- [ ] `OpenAiApiAdapter` + - SSE → UnifiedChunk + - costType = "actual" + - 支持任何 OpenAI 兼容端点(baseUrl + apiKey) +- [ ] Model registry 自动注册新 backend 的 models +- [ ] Agent routing 生效: + +```json +{ + "agents": { + "default": { "preferred": "claude-cli", "fallback": "openai" }, + "interpreter": { "preferred": "openai/gpt-4o-mini", "fallback": null } + } +} +``` + +- [ ] Non-streaming fallback(`allowFallback = !hasStartedStreaming`) +- [ ] Stats 按 backend + agent 分组 + +**验收标准:** claude-cli 挂了 → default agent 自动 fallback 到 openai + +### Phase 3: Ollama + Agent Routing 稳定化(v4.0-rc) +**目标:** 摆脱付费依赖,agent routing 跑稳 + +- [ ] `OllamaAdapter` + - 自动发现模型 + - costType = "free" +- [ ] Agent routing 完整覆盖所有 agent: + +```json +{ + "agents": { + "main": { "preferred": "claude-cli/opus", "fallback": "openai" }, + "tech_geek": { "preferred": "claude-cli/sonnet", "fallback": "ollama/qwen3" }, + "interpreter": { "preferred": "ollama/qwen3", "fallback": null } + } +} +``` + +- [ ] `/ocp routing` — 显示每个 agent 当前走哪个 backend +- [ ] 端到端测试:Claude CLI 下线 → agent 自动 fallback → 恢复后自动回切 + +**验收标准:** 三个 backend 同时运行,每个 agent 走不同路径,稳定无错 + +### Phase 4: Budget + Cost Control(v4.1) +**目标:** 核心差异化 — agent-aware 成本控制 + +**前置条件:** Phase 3 routing 稳定(先确定流量怎么走 → 再控制花多少钱) + +**安全原则:** 没有显式配置 budget 的 agent → 行为等同于 v3(不限制、不降级、不告警)。Budget 是 opt-in,不是 opt-out。 + +- [ ] Per-agent 预算: + +```json +{ + "agents": { + "main": { + "preferred": "claude-cli/opus", + "fallback": "openai", + "dailyBudget": { "limit": 3.00, "onExhausted": "downgrade" } + } + }, + "budget": { + "global": { "daily": 10.00, "onExhausted": "warn" } + } +} +``` + +- [ ] 超预算行为:downgrade | block | warn +- [ ] `/ocp cost` — per-agent 成本(标注 actual/estimated/free) +- [ ] Telegram 告警(预算 80%、backend 连续失败) + +### Phase 5: Anthropic 官方 API(v4.2,等时机) +- [ ] `AnthropicApiAdapter`(官方 SDK) +- [ ] 迁移指南:一行配置切换 +- [ ] costType = "actual" + +--- + +## 文件结构(目标) + +``` +claude-proxy/ +├── server.mjs → HTTP 层(统一 chunk → SSE 输出) +├── model-registry.mjs → 模型注册表(model → backend 映射) +├── agent-router.mjs → Agent 路由引擎(策略层) +├── session-manager.mjs → 会话管理 +├── stats-collector.mjs → 统计 + 成本追踪 +├── config.mjs → 配置加载 + 版本验证 +├── unified-chunk.mjs → UnifiedChunk 类型定义 +├── backends/ +│ ├── base.mjs → BackendAdapter 接口 +│ ├── claude-cli.mjs → Claude CLI(core) +│ ├── openai-api.mjs → OpenAI 兼容(core) +│ └── ollama.mjs → Ollama 本地(core) +├── setup.mjs → 安装向导 +└── ocp-plugin/ → Gateway 命令插件 +``` + +--- + +## 配置完整示例(目标状态) + +```json +{ + "version": "4.0", + "port": 3456, + + "backends": { + "claude-cli": { + "type": "claude-cli", + "tier": "core", + "enabled": true, + "models": ["claude-sonnet-4-6", "claude-opus-4-6"], + "maxConcurrent": 4, + "timeout": { "firstByte": 120000, "overall": 300000 }, + "estimatedCostPerMTok": { "input": 3.0, "output": 15.0 } + }, + "openai": { + "type": "openai-api", + "tier": "core", + "enabled": true, + "baseUrl": "https://openrouter.ai/api/v1", + "apiKey": "${OPENROUTER_API_KEY}", + "models": ["anthropic/claude-3.5-sonnet", "google/gemini-2.0-flash"], + "maxConcurrent": 10, + "timeout": { "firstByte": 30000, "overall": 120000 } + }, + "ollama": { + "type": "ollama", + "tier": "core", + "enabled": true, + "baseUrl": "http://localhost:11434", + "models": "auto" + } + }, + + "agents": { + "default": { + "preferred": "claude-cli", + "fallback": "openai" + }, + "main": { + "preferred": "claude-cli/claude-opus-4-6", + "fallback": "openai", + "dailyBudget": { "limit": 3.00, "onExhausted": "downgrade" } + }, + "tech_geek": { + "preferred": "claude-cli/claude-sonnet-4-6", + "fallback": "ollama/qwen3", + "dailyBudget": { "limit": 2.00, "onExhausted": "fallback" } + }, + "interpreter": { + "preferred": "ollama/qwen3", + "fallback": null + } + }, + + "fallback": { + "onlyBeforeFirstByte": true, + "triggers": ["timeout", "rate-limit", "connection-error"], + "maxRetries": 1 + }, + + "budget": { + "global": { "daily": 10.00, "onExhausted": "warn" }, + "alerts": { "threshold": 0.8, "channel": "telegram" } + }, + + "sessions": { "ttl": 3600000, "maxPromptChars": 150000 }, + "monitoring": { "retainHours": 72 } +} +``` + +--- + +## 风险与应对 + +| 风险 | 应对 | +|------|------| +| Claude CLI 被封 | 切换 backend,上层不变 | +| Anthropic 出订阅 API | 实现 anthropic-api adapter,成本控制层继续有价值 | +| LiteLLM 竞争 | 不竞争。OCP 做 agent-aware 成本控制,赛道不同 | +| 成本数据不准 | cost type 标记 actual/estimated/free | +| Streaming fallback 拼接 | Phase 3 只做 non-streaming fallback | +| 过度工程 | 只维护 3 个 core backend,phase 严格拆分 | + +## 优先级 + +| Phase | 时间 | 内容 | 交付物 | +|-------|------|------|--------| +| 1 | 本周 | 重构:adapter 接口 + model registry + agent router | 行为不变,架构就位 | +| 2 | 1-2 周 | OpenAI backend | 两个 backend 跑通 | +| 3 | 2-3 周 | Ollama + Agent routing 稳定化 | 三 backend + per-agent 路由 | +| 4 | 按需 | Budget + Cost control | 成本可控 | +| 5 | 等时机 | Anthropic 官方 API | 逃生通道 | + +## 成功标准 + +1. Claude CLI 被封的那天,改一行配置,所有 agent 继续运行 +2. `/ocp cost` 每个 agent 花了多少(actual/estimated/free) +3. 翻译 agent 自动用 Ollama($0),coding agent 用 Claude +4. 超预算自动降级,不需人工干预 +5. 新 backend 只需实现 BackendAdapter 接口 + 一个文件 + +## 设计审核记录 + +| 轮次 | 审核方 | 关键结论 | +|------|--------|----------| +| 第一轮 | AI-A | 定位从 proxy 转向成本控制器;识别 5 个 P0 问题 | +| 第二轮 | Claude(本项目开发者) | 反驳:不能删 model routing(基础设施层);backend 数量是运营决策不是架构约束;budget 拆到 Phase 4 | +| 第三轮 | AI-A | 校准:model registry 与 routing 物理拆开两个模块;agent identity 结构预留 inferred 扩展点;backend 加 tier 标记 | + +## 一句话总结 + +**短期靠套利活(Claude CLI proxy),长期靠控制力活(Agent 成本控制器)。两条腿走路,CLI 死了上层不死。** diff --git a/agent-router.mjs b/agent-router.mjs new file mode 100644 index 0000000..5fda1b4 --- /dev/null +++ b/agent-router.mjs @@ -0,0 +1,171 @@ +/** + * Agent Router — Layer 2 of OCP routing. + * + * Routing decision order (written in stone): + * 1. Agent policy override → agent has explicit backend config? Use it. + * 2. Model registry → model maps to a backend? Route there. + * 3. Default backend → global fallback. + * 4. Fallback chain → chosen backend is down? Try next. + * + * Agent identity sources (priority): + * 1. x-ocp-agent header + * 2. Session metadata (bound on first request) + * 3. (future) Inferred from usage patterns + * 4. "default" + */ + +export class AgentRouter { + /** + * @param {Object} options + * @param {import("./model-registry.mjs").ModelRegistry} options.registry + * @param {Map} options.backends + * @param {Object} options.agentConfig — agent routing rules from config + * @param {string} options.defaultBackend — fallback backend ID + */ + constructor({ registry, backends, agentConfig = {}, defaultBackend = "claude-cli" }) { + this._registry = registry; + this._backends = backends; + this._agentConfig = agentConfig; + this._defaultBackend = defaultBackend; + } + + /** + * Identify the agent from a request. + * + * @param {Object} req - HTTP request + * @param {Object} [sessionMeta] - Session metadata (may contain agent binding) + * @returns {string} Agent name + */ + identifyAgent(req, sessionMeta = null) { + // 1. Explicit header + const headerAgent = req.headers?.["x-ocp-agent"]; + if (headerAgent) return headerAgent; + + // 2. Session binding + if (sessionMeta?.agent) return sessionMeta.agent; + + // 3. (future) Inferred — not implemented yet + + // 4. Default + return "default"; + } + + /** + * Resolve which backend + model to use for a request. + * + * @param {Object} params + * @param {string} params.agent - Agent name (from identifyAgent) + * @param {string} params.model - Requested model ID + * @returns {{backendId: string, model: string, reason: string}} + */ + resolve({ agent, model }) { + // Step 1: Agent policy override + const agentRule = this._agentConfig[agent]; + if (agentRule?.preferred) { + const { backendId, modelId } = this._parsePreferred(agentRule.preferred); + const backend = this._backends.get(backendId); + if (backend?.enabled) { + const resolvedModel = modelId || model; + return { + backendId, + model: resolvedModel, + reason: `agent-policy(${agent})`, + }; + } + // Agent preferred backend is down — try fallback + if (agentRule.fallback) { + const fb = this._parseFallback(agentRule.fallback, model); + if (fb) return { ...fb, reason: `agent-fallback(${agent})` }; + } + } + + // Step 2: Model registry + const registryEntry = this._registry.resolve(model); + if (registryEntry) { + const backend = this._backends.get(registryEntry.backendId); + if (backend?.enabled) { + return { + backendId: registryEntry.backendId, + model: registryEntry.canonical, + reason: "model-registry", + }; + } + } + + // Step 2b: Check adapters for alias support + for (const [id, backend] of this._backends) { + if (backend.enabled && backend.supportsModel(model)) { + const resolved = typeof backend.resolveModel === "function" + ? backend.resolveModel(model) + : model; + return { + backendId: id, + model: resolved, + reason: `adapter-match(${id})`, + }; + } + } + + // Step 3: Default backend + const defaultAgent = this._agentConfig.default; + if (defaultAgent?.preferred) { + const { backendId } = this._parsePreferred(defaultAgent.preferred); + if (this._backends.get(backendId)?.enabled) { + return { backendId, model, reason: "default-agent-policy" }; + } + } + + if (this._backends.get(this._defaultBackend)?.enabled) { + return { + backendId: this._defaultBackend, + model, + reason: "default-backend", + }; + } + + // Step 4: Fallback — try any enabled backend + for (const [id, backend] of this._backends) { + if (backend.enabled) { + return { backendId: id, model, reason: `last-resort(${id})` }; + } + } + + return { backendId: null, model, reason: "no-backend-available" }; + } + + /** + * Parse a preferred string like "claude-cli" or "claude-cli/claude-opus-4-6" + */ + _parsePreferred(preferred) { + if (preferred.includes("/")) { + const [backendId, modelId] = preferred.split("/", 2); + return { backendId, modelId }; + } + return { backendId: preferred, modelId: null }; + } + + /** + * Parse a fallback value and return routing info. + */ + _parseFallback(fallback, model) { + if (!fallback) return null; + const { backendId, modelId } = this._parsePreferred(fallback); + const backend = this._backends.get(backendId); + if (!backend?.enabled) return null; + return { backendId, model: modelId || model }; + } + + /** + * Get routing info for display (e.g. /ocp routing command). + */ + getRoutingTable() { + const table = {}; + for (const [agent, config] of Object.entries(this._agentConfig)) { + table[agent] = { + preferred: config.preferred || null, + fallback: config.fallback || null, + }; + } + return table; + } +} diff --git a/backends/base.mjs b/backends/base.mjs new file mode 100644 index 0000000..aa4c3f5 --- /dev/null +++ b/backends/base.mjs @@ -0,0 +1,90 @@ +/** + * BackendAdapter — base class for all OCP backends. + * + * Each backend adapter MUST: + * 1. Implement chatCompletion() as an async generator yielding UnifiedChunks + * 2. Implement healthCheck() + * 3. Report its costType (actual | estimated | free) + * 4. Report its tier (core | community) + * + * Adapters do NOT handle: + * - Session management (SessionManager does that) + * - Stats collection (StatsCollector does that) + * - Routing decisions (agent-router does that) + */ + +export class BackendAdapter { + /** + * @param {Object} config - Backend-specific configuration + */ + constructor(config = {}) { + this._config = config; + this._enabled = config.enabled !== false; + } + + /** @returns {string} Unique backend identifier */ + get id() { throw new Error("BackendAdapter.id must be implemented"); } + + /** @returns {string} Human-readable name */ + get displayName() { throw new Error("BackendAdapter.displayName must be implemented"); } + + /** @returns {string[]} List of supported model IDs */ + get models() { throw new Error("BackendAdapter.models must be implemented"); } + + /** @returns {"actual"|"estimated"|"free"} */ + get costType() { throw new Error("BackendAdapter.costType must be implemented"); } + + /** @returns {"core"|"community"} */ + get tier() { return "core"; } + + /** @returns {boolean} */ + get enabled() { return this._enabled; } + + /** + * Initialize the backend (called once at startup). + * Override to do async setup (e.g. validate binary, check API key). + */ + async initialize() {} + + /** + * Shutdown the backend (called on graceful shutdown). + * Override to clean up resources. + */ + async shutdown() {} + + /** + * Check if this backend is healthy and reachable. + * @returns {Promise<{ok: boolean, message: string, latencyMs: number}>} + */ + async healthCheck() { + return { ok: false, message: "healthCheck not implemented", latencyMs: 0 }; + } + + /** + * Execute a chat completion request. + * + * MUST yield UnifiedChunk objects: + * { type: "delta", content: "...", model, backend } — streaming content + * { type: "done", model, backend, usage?, cost? } — stream finished + * { type: "error", error: "...", model, backend } — error occurred + * + * @param {Object} request + * @param {string} request.model - Canonical model ID + * @param {Array} request.messages - OpenAI-format messages array + * @param {Object} [request.session] - Session info { uuid, resume } + * @param {Object} [request.options] - Additional options (temperature, etc.) + * @yields {import("../unified-chunk.mjs").UnifiedChunk} + */ + async *chatCompletion(request) { + throw new Error("BackendAdapter.chatCompletion must be implemented"); + } + + /** + * Check if this backend supports a given model. + * Default: exact match against models list. + * Override for pattern matching (e.g. "gpt-*" → openai). + */ + supportsModel(model) { + return this.models.includes(model); + } +} diff --git a/backends/claude-cli.mjs b/backends/claude-cli.mjs new file mode 100644 index 0000000..366b699 --- /dev/null +++ b/backends/claude-cli.mjs @@ -0,0 +1,393 @@ +/** + * ClaudeCliAdapter — OCP backend for Claude via local CLI binary. + * + * Spawns `claude -p` processes for each request. Converts CLI stdout + * into UnifiedChunk streaming protocol. + * + * costType: "estimated" (no actual cost data from CLI) + */ + +import { spawn, execFileSync } from "node:child_process"; +import { accessSync, constants } from "node:fs"; +import { join } from "node:path"; +import { BackendAdapter } from "./base.mjs"; +import { delta, done, error } from "../unified-chunk.mjs"; + +// Default models exposed by this backend +const DEFAULT_MODELS = [ + "claude-opus-4-6", + "claude-sonnet-4-6", + "claude-haiku-4-5-20251001", +]; + +// Alias → canonical model mapping +const MODEL_ALIASES = { + "claude-opus-4": "claude-opus-4-6", + "claude-haiku-4": "claude-haiku-4-5-20251001", + "claude-haiku-4-5": "claude-haiku-4-5-20251001", + "opus": "claude-opus-4-6", + "sonnet": "claude-sonnet-4-6", + "haiku": "claude-haiku-4-5-20251001", +}; + +export class ClaudeCliAdapter extends BackendAdapter { + constructor(config = {}) { + super(config); + this._claudeBin = null; + this._activeProcesses = new Set(); + this._maxConcurrent = config.maxConcurrent || 8; + this._timeout = config.timeout?.overall || 300000; + this._skipPermissions = config.skipPermissions || false; + this._allowedTools = config.allowedTools || [ + "Bash", "Read", "Write", "Edit", "Glob", "Grep", + "WebSearch", "WebFetch", "Agent", + ]; + this._systemPrompt = config.systemPrompt || ""; + this._mcpConfig = config.mcpConfig || ""; + this._modelList = config.models || DEFAULT_MODELS; + + // Timeout tiers: per-model base + per-prompt-char scaling + this._timeoutTiers = config.timeoutTiers || { + opus: { base: 150000, perPromptChar: 0.00050 }, + sonnet: { base: 120000, perPromptChar: 0.00050 }, + haiku: { base: 45000, perPromptChar: 0.00010 }, + }; + + // Estimated cost per MTok (for cost tracking) + this._estimatedCost = config.estimatedCostPerMTok || { + input: 3.0, + output: 15.0, + }; + } + + get id() { return "claude-cli"; } + get displayName() { return "Claude CLI"; } + get models() { return this._modelList; } + get costType() { return "estimated"; } + get tier() { return "core"; } + get activeProcessCount() { return this._activeProcesses.size; } + + /** + * Resolve a model ID, handling aliases. + * Returns canonical CLI model ID or the input if no alias found. + */ + resolveModel(model) { + return MODEL_ALIASES[model] || model; + } + + supportsModel(model) { + const canonical = this.resolveModel(model); + return this._modelList.includes(canonical) || model in MODEL_ALIASES; + } + + async initialize() { + this._claudeBin = this._resolveBinary(); + } + + async shutdown() { + for (const proc of this._activeProcesses) { + try { proc.kill("SIGTERM"); } catch {} + } + // Force kill after 5s + if (this._activeProcesses.size > 0) { + await new Promise(r => setTimeout(r, 5000)); + for (const proc of this._activeProcesses) { + try { proc.kill("SIGKILL"); } catch {} + } + } + } + + async healthCheck() { + const t0 = Date.now(); + try { + const env = this._cleanEnv(); + execFileSync(this._claudeBin, ["auth", "status"], { + encoding: "utf8", timeout: 10000, env, + }); + return { ok: true, message: "authenticated", latencyMs: Date.now() - t0 }; + } catch (e) { + return { + ok: false, + message: (e.stderr || e.message || "").slice(0, 200), + latencyMs: Date.now() - t0, + }; + } + } + + /** + * Execute a chat completion via Claude CLI. + * Yields UnifiedChunks as stdout data arrives. + */ + async *chatCompletion(request) { + const { model, messages, session, promptText } = request; + const cliModel = this.resolveModel(model); + + if (this._activeProcesses.size >= this._maxConcurrent) { + yield error( + `concurrency limit reached (${this._activeProcesses.size}/${this._maxConcurrent})`, + cliModel, this.id, + ); + return; + } + + const prompt = promptText || this._messagesToPrompt(messages); + const cliArgs = this._buildCliArgs(cliModel, session); + const env = this._cleanEnv(); + + const proc = spawn(this._claudeBin, cliArgs, { + env, stdio: ["pipe", "pipe", "pipe"], + }); + this._activeProcesses.add(proc); + + const t0 = Date.now(); + const firstByteTimeoutMs = this._computeFirstByteTimeout(cliModel, prompt.length); + let gotFirstByte = false; + let totalChars = 0; + let stderr = ""; + let finished = false; + + // Create a promise-based wrapper for the process + try { + yield* await new Promise((resolve, reject) => { + const chunks = []; + let streamResolve = null; + let streamDone = false; + + // We can't directly yield from inside callbacks, so we collect + // and use a different approach — return an async iterable. + // Actually, let's use a simpler pattern with a queue. + reject(new Error("USE_QUEUE_PATTERN")); + }); + } catch { + // Fall through to queue-based pattern below + } + + // Queue-based async generator pattern + const queue = []; + let queueResolve = null; + let queueDone = false; + + function enqueue(chunk) { + if (queueDone) return; + if (queueResolve) { + const r = queueResolve; + queueResolve = null; + r(chunk); + } else { + queue.push(chunk); + } + } + + function waitForChunk() { + if (queue.length > 0) return Promise.resolve(queue.shift()); + if (queueDone) return Promise.resolve(null); + return new Promise(r => { queueResolve = r; }); + } + + // Write prompt to stdin + proc.stdin.write(prompt); + proc.stdin.end(); + + proc.stdout.on("data", (d) => { + if (!gotFirstByte) { + gotFirstByte = true; + clearTimeout(firstByteTimer); + } + const text = d.toString(); + totalChars += text.length; + enqueue(delta(text, cliModel, this.id)); + }); + + proc.stderr.on("data", (d) => { stderr += d; }); + + proc.on("close", (code, signal) => { + this._activeProcesses.delete(proc); + clearTimeout(firstByteTimer); + clearTimeout(overallTimer); + const elapsed = Date.now() - t0; + + if (code !== 0) { + const msg = stderr.slice(0, 300) || `claude exit ${code}`; + enqueue(error(msg, cliModel, this.id)); + } else { + // Estimate cost based on prompt + completion chars + const estimatedInputTokens = Math.ceil(prompt.length / 4); + const estimatedOutputTokens = Math.ceil(totalChars / 4); + enqueue(done(cliModel, this.id, { + promptTokens: estimatedInputTokens, + completionTokens: estimatedOutputTokens, + }, { + value: (estimatedInputTokens * this._estimatedCost.input + + estimatedOutputTokens * this._estimatedCost.output) / 1000000, + type: "estimated", + })); + } + queueDone = true; + if (queueResolve) { + const r = queueResolve; + queueResolve = null; + r(null); + } + }); + + proc.on("error", (err) => { + this._activeProcesses.delete(proc); + clearTimeout(firstByteTimer); + clearTimeout(overallTimer); + enqueue(error(err.message, cliModel, this.id)); + queueDone = true; + if (queueResolve) { + const r = queueResolve; + queueResolve = null; + r(null); + } + }); + + // First-byte timeout + const firstByteTimer = setTimeout(() => { + if (!gotFirstByte) { + try { proc.kill("SIGTERM"); } catch {} + setTimeout(() => { try { proc.kill("SIGKILL"); } catch {} }, 5000); + enqueue(error(`first-byte timeout after ${firstByteTimeoutMs}ms`, cliModel, this.id)); + queueDone = true; + } + }, firstByteTimeoutMs); + + // Overall timeout + const overallTimer = setTimeout(() => { + try { proc.kill("SIGTERM"); } catch {} + setTimeout(() => { try { proc.kill("SIGKILL"); } catch {} }, 5000); + enqueue(error(`request timeout after ${this._timeout}ms`, cliModel, this.id)); + queueDone = true; + }, this._timeout); + + // Yield chunks as they arrive + while (true) { + const chunk = await waitForChunk(); + if (chunk === null) break; + yield chunk; + if (chunk.type === "done" || chunk.type === "error") break; + } + } + + // ── Internal helpers ────────────────────────────────────────────────── + + _resolveBinary() { + if (process.env.CLAUDE_BIN) { + try { + accessSync(process.env.CLAUDE_BIN, constants.X_OK); + return process.env.CLAUDE_BIN; + } catch { + throw new Error(`CLAUDE_BIN="${process.env.CLAUDE_BIN}" is set but not executable.`); + } + } + + const candidates = [ + "/opt/homebrew/bin/claude", + "/usr/local/bin/claude", + "/usr/bin/claude", + join(process.env.HOME || "", ".local/bin/claude"), + ]; + for (const p of candidates) { + try { accessSync(p, constants.X_OK); return p; } catch {} + } + + try { + const resolved = execFileSync("which", ["claude"], { + encoding: "utf8", timeout: 5000, + }).trim(); + if (resolved) return resolved; + } catch {} + + throw new Error( + "claude binary not found. Set CLAUDE_BIN or ensure claude is in PATH." + ); + } + + _cleanEnv() { + const env = { ...process.env }; + delete env.CLAUDECODE; + delete env.ANTHROPIC_API_KEY; + delete env.ANTHROPIC_BASE_URL; + delete env.ANTHROPIC_AUTH_TOKEN; + return env; + } + + _buildCliArgs(cliModel, session) { + const args = ["-p", "--model", cliModel, "--output-format", "text"]; + + if (session?.resume) { + args.push("--resume", session.uuid); + } else if (session?.uuid) { + args.push("--session-id", session.uuid); + } else { + args.push("--no-session-persistence"); + } + + if (this._skipPermissions) { + args.push("--dangerously-skip-permissions"); + } else if (this._allowedTools.length > 0) { + args.push("--allowedTools", ...this._allowedTools); + } + + if (this._systemPrompt) { + args.push("--append-system-prompt", this._systemPrompt); + } + + if (this._mcpConfig) { + args.push("--mcp-config", this._mcpConfig); + } + + return args; + } + + _getModelTier(cliModel) { + if (cliModel.includes("opus")) return "opus"; + if (cliModel.includes("haiku")) return "haiku"; + return "sonnet"; + } + + _computeFirstByteTimeout(cliModel, promptLength) { + const tierName = this._getModelTier(cliModel); + const tier = this._timeoutTiers[tierName] || this._timeoutTiers.sonnet; + const timeout = tier.base + Math.floor(promptLength * tier.perPromptChar); + return Math.min(timeout, Math.max(this._timeout - 5000, 10000)); + } + + /** + * Convert OpenAI-format messages to a plain text prompt for Claude CLI. + * Includes truncation guard for oversized prompts. + */ + _messagesToPrompt(messages, maxChars = 150000) { + const full = messages.map((m) => { + const text = typeof m.content === "string" ? m.content : JSON.stringify(m.content); + if (m.role === "system") return `[System] ${text}`; + if (m.role === "assistant") return `[Assistant] ${text}`; + return text; + }); + + const joined = full.join("\n\n"); + if (joined.length <= maxChars) return joined; + + // Truncation: keep system + recent messages + const system = []; + const rest = []; + for (let i = 0; i < full.length; i++) { + if (messages[i].role === "system") system.push(full[i]); + else rest.push(full[i]); + } + + const systemText = system.join("\n\n"); + const budget = maxChars - systemText.length - 200; + const kept = []; + let used = 0; + for (let i = rest.length - 1; i >= 0; i--) { + if (used + rest[i].length + 2 > budget) break; + kept.unshift(rest[i]); + used += rest[i].length + 2; + } + + const truncNote = `[System] Note: ${rest.length - kept.length} older messages were truncated to fit context limit.`; + return [systemText, truncNote, ...kept].filter(Boolean).join("\n\n"); + } +} diff --git a/model-registry.mjs b/model-registry.mjs new file mode 100644 index 0000000..9ef8b26 --- /dev/null +++ b/model-registry.mjs @@ -0,0 +1,99 @@ +/** + * Model Registry — Layer 1 of OCP routing. + * + * Static mapping: model ID → backend ID. + * This is infrastructure, not policy. Every model must map to exactly one backend. + * + * Non-agent callers (Cline, Aider, etc.) only send a model name — this module + * is the only thing that tells OCP where to route the request. + */ + +export class ModelRegistry { + constructor() { + /** @type {Map} */ + this._models = new Map(); + + /** @type {Map} model id → display metadata */ + this._metadata = new Map(); + } + + /** + * Register models from a backend adapter. + * Called during startup for each enabled backend. + * + * @param {import("./backends/base.mjs").BackendAdapter} adapter + */ + registerBackend(adapter) { + for (const modelId of adapter.models) { + this._models.set(modelId, { + backendId: adapter.id, + canonical: modelId, + }); + this._metadata.set(modelId, { + backendId: adapter.id, + backendName: adapter.displayName, + tier: adapter.tier, + costType: adapter.costType, + }); + } + + // Also register aliases if the adapter supports resolveModel + if (typeof adapter.resolveModel === "function") { + // We don't auto-register aliases here — they're handled by + // supportsModel() on the adapter. The registry only knows canonical IDs. + } + } + + /** + * Look up which backend serves a given model. + * + * @param {string} model — model ID (could be canonical or alias) + * @returns {{backendId: string, canonical: string} | null} + */ + resolve(model) { + // Direct lookup first + const direct = this._models.get(model); + if (direct) return direct; + + // No alias resolution here — that's the adapter's job. + // The router will ask each adapter if it supports the model. + return null; + } + + /** + * Get all registered models with metadata. + * Used by GET /v1/models endpoint. + */ + listModels() { + const result = []; + for (const [id, meta] of this._metadata) { + result.push({ id, ...meta }); + } + return result; + } + + /** + * Check if a model is known to the registry. + */ + has(model) { + return this._models.has(model); + } + + /** + * Get the backend ID for a model, or null. + */ + getBackendId(model) { + return this._models.get(model)?.backendId || null; + } + + /** + * Get all models for a specific backend. + */ + getModelsForBackend(backendId) { + const models = []; + for (const [id, entry] of this._models) { + if (entry.backendId === backendId) models.push(id); + } + return models; + } +} diff --git a/server.mjs b/server.mjs index 354ebf1..6697952 100644 --- a/server.mjs +++ b/server.mjs @@ -1,11 +1,17 @@ #!/usr/bin/env node /** - * openclaw-claude-proxy v2.5.0 — OpenAI-compatible proxy for Claude CLI + * OCP (OpenClaw Control Plane) v4.0-alpha * - * Translates OpenAI chat/completions requests into `claude -p` CLI calls, - * letting you use your Claude Pro/Max subscription as an OpenClaw model provider. + * Cost-control and model-routing layer for AI agents. + * Currently supports Claude CLI backend; architecture supports multiple backends. * - * v2.5.0: + * v4.0-alpha: + * - Internal modular architecture: BackendAdapter, ModelRegistry, AgentRouter, + * SessionManager, StatsCollector + * - External API unchanged from v2.5.0 + * - Claude CLI backend extracted to ClaudeCliAdapter + * + * v2.5.0 (legacy notes): * - Sliding-window circuit breaker: uses time-windowed failure rate instead of * consecutive-count, preventing multi-agent burst scenarios from tripping the * breaker too aggressively. Half-open state allows configurable probe requests. @@ -47,6 +53,54 @@ import { dirname, join } from "node:path"; const __dirname = dirname(fileURLToPath(import.meta.url)); const _pkg = JSON.parse(readFileSync(join(__dirname, "package.json"), "utf8")); +// ── v4 Modular Architecture (Step 1: load + initialize, hot path unchanged) ── +import { ClaudeCliAdapter } from "./backends/claude-cli.mjs"; +import { ModelRegistry } from "./model-registry.mjs"; +import { AgentRouter } from "./agent-router.mjs"; +import { SessionManager as SessionMgr } from "./session-manager.mjs"; +import { StatsCollector } from "./stats-collector.mjs"; + +// These modules are initialized at startup but NOT yet wired into the request +// path. Step 2 will replace the legacy callClaude/callClaudeStreaming with +// adapter-based calls. For now, they run in parallel for validation. +let _v4Backend = null; +let _v4Registry = null; +let _v4Router = null; +let _v4Sessions = null; +let _v4Stats = null; + +async function initV4Modules() { + try { + _v4Stats = new StatsCollector(); + _v4Sessions = new SessionMgr({ ttl: SESSION_TTL }); + _v4Registry = new ModelRegistry(); + + _v4Backend = new ClaudeCliAdapter({ + maxConcurrent: MAX_CONCURRENT, + timeout: { overall: TIMEOUT }, + skipPermissions: SKIP_PERMISSIONS, + allowedTools: ALLOWED_TOOLS, + systemPrompt: SYSTEM_PROMPT, + mcpConfig: MCP_CONFIG, + timeoutTiers: MODEL_TIMEOUT_TIERS, + }); + await _v4Backend.initialize(); + _v4Registry.registerBackend(_v4Backend); + + const backends = new Map([["claude-cli", _v4Backend]]); + _v4Router = new AgentRouter({ + registry: _v4Registry, + backends, + agentConfig: { default: { preferred: "claude-cli", fallback: null } }, + defaultBackend: "claude-cli", + }); + + console.log(`[v4] Modules initialized: backend=${_v4Backend.id}, models=${_v4Backend.models.length}, registry=${_v4Registry.listModels().length} models`); + } catch (err) { + console.error(`[v4] Module init failed (non-fatal, legacy path active): ${err.message}`); + } +} + // ── Resolve claude binary ─────────────────────────────────────────────── // Priority: CLAUDE_BIN env > well-known paths > which lookup // Fail-fast if not found — never start with an unresolvable binary. @@ -1178,7 +1232,36 @@ const server = createServer(async (req, res) => { return handleSettings(req, res); } - jsonResponse(res, 404, { error: "Not found. Endpoints: GET /v1/models, POST /v1/chat/completions, GET /health, GET /usage, GET /status, GET /logs, GET|PATCH /settings, GET|DELETE /sessions" }); + // GET /backends — v4 backend status (new endpoint) + if (req.url === "/backends" && req.method === "GET") { + if (!_v4Backend) { + return jsonResponse(res, 503, { error: "v4 modules not initialized" }); + } + const health = await _v4Backend.healthCheck(); + return jsonResponse(res, 200, { + backends: [{ + id: _v4Backend.id, + displayName: _v4Backend.displayName, + tier: _v4Backend.tier, + costType: _v4Backend.costType, + enabled: _v4Backend.enabled, + models: _v4Backend.models, + activeProcesses: _v4Backend.activeProcessCount, + health, + }], + routing: _v4Router?.getRoutingTable() || {}, + }); + } + + // GET /routing — v4 agent routing table (new endpoint) + if (req.url === "/routing" && req.method === "GET") { + return jsonResponse(res, 200, { + table: _v4Router?.getRoutingTable() || {}, + registry: _v4Registry?.listModels() || [], + }); + } + + jsonResponse(res, 404, { error: "Not found. Endpoints: GET /v1/models, POST /v1/chat/completions, GET /health, GET /usage, GET /status, GET /logs, GET|PATCH /settings, GET|DELETE /sessions, GET /backends, GET /routing" }); }); @@ -1199,6 +1282,10 @@ function gracefulShutdown(signal) { clearInterval(sessionCleanupInterval); clearInterval(authCheckInterval); + // 2b. Shutdown v4 modules + if (_v4Sessions) _v4Sessions.shutdown(); + if (_v4Backend) _v4Backend.shutdown().catch(() => {}); + // 3. Kill all active child processes for (const proc of activeProcesses) { try { proc.kill("SIGTERM"); } catch {} @@ -1235,8 +1322,11 @@ process.on("SIGTERM", () => gracefulShutdown("SIGTERM")); process.on("SIGINT", () => gracefulShutdown("SIGINT")); // ── Start ─────────────────────────────────────────────────────────────── +// Initialize v4 modules before listening +initV4Modules().catch(err => console.error(`[v4] init error: ${err.message}`)); + server.listen(PORT, "127.0.0.1", () => { - console.log(`openclaw-claude-proxy v${VERSION} listening on http://127.0.0.1:${PORT}`); + console.log(`OCP (OpenClaw Control Plane) v${VERSION} listening on http://127.0.0.1:${PORT}`); console.log(`Architecture: on-demand spawning (no pool)`); console.log(`Models: ${MODELS.map((m) => m.id).join(", ")}`); console.log(`Claude binary: ${CLAUDE}`); diff --git a/session-manager.mjs b/session-manager.mjs new file mode 100644 index 0000000..68e3458 --- /dev/null +++ b/session-manager.mjs @@ -0,0 +1,123 @@ +/** + * SessionManager — manages conversation sessions across requests. + * + * Maps caller-provided conversation IDs to backend-specific session state. + * For Claude CLI, this means mapping to CLI session UUIDs for --resume. + */ + +import { randomUUID } from "node:crypto"; + +export class SessionManager { + /** + * @param {Object} options + * @param {number} [options.ttl=3600000] — Session TTL in ms (default 1h) + */ + constructor({ ttl = 3600000 } = {}) { + this._ttl = ttl; + /** @type {Map} */ + this._sessions = new Map(); + + // Cleanup expired sessions every 60s + this._cleanupInterval = setInterval(() => this._cleanup(), 60000); + } + + get ttl() { return this._ttl; } + set ttl(val) { this._ttl = val; } + get size() { return this._sessions.size; } + + /** + * Get or create a session for a conversation. + * + * @param {string|null} conversationId + * @param {string} model — model being used + * @param {string} [agent] — agent identity + * @returns {{session: {uuid: string, resume: boolean}, isNew: boolean, isOneOff: boolean}} + */ + resolve(conversationId, model, agent = "default") { + if (!conversationId) { + return { session: null, isNew: false, isOneOff: true }; + } + + if (this._sessions.has(conversationId)) { + const entry = this._sessions.get(conversationId); + entry.lastUsed = Date.now(); + entry.messageCount++; + return { + session: { uuid: entry.uuid, resume: true }, + isNew: false, + isOneOff: false, + }; + } + + // New session + const uuid = randomUUID(); + this._sessions.set(conversationId, { + uuid, + model, + agent, + messageCount: 1, + lastUsed: Date.now(), + createdAt: Date.now(), + }); + + return { + session: { uuid, resume: false }, + isNew: true, + isOneOff: false, + }; + } + + /** + * Remove a session (e.g. on resume failure). + */ + delete(conversationId) { + this._sessions.delete(conversationId); + } + + /** + * Clear all sessions. + * @returns {number} Number of sessions cleared + */ + clear() { + const count = this._sessions.size; + this._sessions.clear(); + return count; + } + + /** + * List all active sessions (for /sessions endpoint). + */ + list() { + const result = []; + for (const [id, s] of this._sessions) { + result.push({ + id, + uuid: s.uuid, + model: s.model, + agent: s.agent, + messages: s.messageCount, + lastUsed: new Date(s.lastUsed).toISOString(), + idleMs: Date.now() - s.lastUsed, + }); + } + return result; + } + + /** + * Shutdown — clear interval. + */ + shutdown() { + clearInterval(this._cleanupInterval); + } + + // ── Internal ────────────────────────────────────────────────────────── + + _cleanup() { + const now = Date.now(); + for (const [id, s] of this._sessions) { + if (now - s.lastUsed > this._ttl) { + this._sessions.delete(id); + } + } + } +} diff --git a/stats-collector.mjs b/stats-collector.mjs new file mode 100644 index 0000000..0cee712 --- /dev/null +++ b/stats-collector.mjs @@ -0,0 +1,186 @@ +/** + * StatsCollector — tracks request metrics per-model, per-backend, per-agent. + * + * Replaces the flat stats + modelStats from server.mjs with structured, + * multi-dimensional tracking. Backward compatible: still exposes the same + * data for /health, /usage, /status endpoints. + */ + +export class StatsCollector { + constructor() { + // Global counters (backward compatible) + this.totalRequests = 0; + this.activeRequests = 0; + this.errors = 0; + this.timeouts = 0; + this.sessionHits = 0; + this.sessionMisses = 0; + this.oneOffRequests = 0; + + // Recent errors ring buffer + this._recentErrors = []; + this._maxErrors = 20; + + /** @type {Map} cliModel → stats */ + this._modelStats = new Map(); + + /** @type {Map} backendId → stats */ + this._backendStats = new Map(); + + /** @type {Map} agentName → stats */ + this._agentStats = new Map(); + } + + /** + * Record a new request starting. + */ + recordRequest(model, backend, agent, promptChars) { + this.totalRequests++; + this.activeRequests++; + this._getOrCreate(this._modelStats, model).recordRequest(promptChars); + this._getOrCreate(this._backendStats, backend).recordRequest(promptChars); + this._getOrCreate(this._agentStats, agent).recordRequest(promptChars); + } + + /** + * Record a successful completion. + */ + recordSuccess(model, backend, agent, elapsedMs, cost = null) { + this.activeRequests = Math.max(0, this.activeRequests - 1); + this._getOrCreate(this._modelStats, model).recordSuccess(elapsedMs, cost); + this._getOrCreate(this._backendStats, backend).recordSuccess(elapsedMs, cost); + this._getOrCreate(this._agentStats, agent).recordSuccess(elapsedMs, cost); + } + + /** + * Record an error. + */ + recordError(model, backend, agent, isTimeout, message = "") { + this.activeRequests = Math.max(0, this.activeRequests - 1); + this.errors++; + if (isTimeout) this.timeouts++; + + this._getOrCreate(this._modelStats, model).recordError(isTimeout); + this._getOrCreate(this._backendStats, backend).recordError(isTimeout); + this._getOrCreate(this._agentStats, agent).recordError(isTimeout); + + this._recentErrors.push({ + time: new Date().toISOString(), + message: String(message).slice(0, 200), + model, backend, agent, + }); + if (this._recentErrors.length > this._maxErrors) this._recentErrors.shift(); + } + + /** + * Record session hit/miss/one-off. + */ + recordSessionHit() { this.sessionHits++; } + recordSessionMiss() { this.sessionMisses++; } + recordOneOff() { this.oneOffRequests++; } + + /** + * Get per-model stats snapshot (backward compatible with v2). + */ + getModelStatsSnapshot() { + return this._snapshot(this._modelStats); + } + + /** + * Get per-backend stats snapshot. + */ + getBackendStatsSnapshot() { + return this._snapshot(this._backendStats); + } + + /** + * Get per-agent stats snapshot. + */ + getAgentStatsSnapshot() { + return this._snapshot(this._agentStats); + } + + /** + * Get recent errors. + */ + getRecentErrors(n = 5) { + return this._recentErrors.slice(-n); + } + + /** + * Get global stats (backward compatible). + */ + getGlobalStats() { + return { + totalRequests: this.totalRequests, + activeRequests: this.activeRequests, + errors: this.errors, + timeouts: this.timeouts, + sessionHits: this.sessionHits, + sessionMisses: this.sessionMisses, + oneOffRequests: this.oneOffRequests, + }; + } + + // ── Internal ────────────────────────────────────────────────────────── + + _getOrCreate(map, key) { + if (!map.has(key)) map.set(key, new ModelStats()); + return map.get(key); + } + + _snapshot(map) { + const result = {}; + for (const [key, stats] of map) { + result[key] = stats.toJSON(); + } + return result; + } +} + +/** Per-entity stats tracker */ +class ModelStats { + constructor() { + this.requests = 0; + this.successes = 0; + this.errors = 0; + this.timeouts = 0; + this.totalElapsed = 0; + this.maxElapsed = 0; + this.totalPromptChars = 0; + this.maxPromptChars = 0; + this.totalCost = 0; // accumulated estimated/actual cost + } + + recordRequest(promptChars) { + this.requests++; + this.totalPromptChars += promptChars; + if (promptChars > this.maxPromptChars) this.maxPromptChars = promptChars; + } + + recordSuccess(elapsedMs, cost = null) { + this.successes++; + this.totalElapsed += elapsedMs; + if (elapsedMs > this.maxElapsed) this.maxElapsed = elapsedMs; + if (cost?.value) this.totalCost += cost.value; + } + + recordError(isTimeout) { + this.errors++; + if (isTimeout) this.timeouts++; + } + + toJSON() { + return { + requests: this.requests, + successes: this.successes, + errors: this.errors, + timeouts: this.timeouts, + avgElapsed: this.successes > 0 ? Math.round(this.totalElapsed / this.successes) : 0, + maxElapsed: this.maxElapsed, + avgPromptChars: this.requests > 0 ? Math.round(this.totalPromptChars / this.requests) : 0, + maxPromptChars: this.maxPromptChars, + totalCost: Math.round(this.totalCost * 10000) / 10000, // 4 decimal places + }; + } +} diff --git a/unified-chunk.mjs b/unified-chunk.mjs new file mode 100644 index 0000000..00db874 --- /dev/null +++ b/unified-chunk.mjs @@ -0,0 +1,42 @@ +/** + * UnifiedChunk — the internal streaming protocol for OCP. + * + * All backend adapters MUST convert their native output into this format. + * server.mjs only handles UnifiedChunks → OpenAI SSE conversion. + */ + +/** + * @typedef {Object} UnifiedChunk + * @property {"delta"|"done"|"error"} type + * @property {string} [content] — text content (for "delta") + * @property {string} model — canonical model id + * @property {string} backend — backend id (e.g. "claude-cli") + * @property {Object} [usage] — token usage (for "done") + * @property {number} [usage.promptTokens] + * @property {number} [usage.completionTokens] + * @property {Object} [cost] — cost info (for "done") + * @property {number} [cost.value] + * @property {"actual"|"estimated"|"free"} [cost.type] + * @property {string} [error] — error message (for "error") + */ + +/** + * Create a delta chunk (streaming content) + */ +export function delta(content, model, backend) { + return { type: "delta", content, model, backend }; +} + +/** + * Create a done chunk (stream finished) + */ +export function done(model, backend, usage = null, cost = null) { + return { type: "done", model, backend, ...(usage && { usage }), ...(cost && { cost }) }; +} + +/** + * Create an error chunk + */ +export function error(message, model, backend) { + return { type: "error", error: message, model, backend }; +}