mirror of
https://github.com/dtzp555-max/ocp.git
synced 2026-07-19 09:44:07 +00:00
feat: Phase 1 — modular backend architecture (v4.0-alpha)
Internal refactor: extract BackendAdapter, ModelRegistry, AgentRouter, SessionManager, and StatsCollector as standalone modules. Claude CLI logic encapsulated in ClaudeCliAdapter. External API completely unchanged — all existing endpoints behave identically. Legacy request path (callClaude/callClaudeStreaming) untouched. v4 modules load in parallel for validation. New endpoints: GET /backends — backend health and status GET /routing — agent routing table Architecture: unified-chunk.mjs — streaming protocol (delta/done/error) backends/base.mjs — BackendAdapter interface backends/claude-cli.mjs — Claude CLI adapter (core) model-registry.mjs — model → backend mapping (Layer 1) agent-router.mjs — agent policy routing (Layer 2) session-manager.mjs — conversation session management stats-collector.mjs — per-model/backend/agent metrics Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.6
+439
@@ -0,0 +1,439 @@
|
|||||||
|
# OCP v4.0 开发计划 — Agent 成本控制器
|
||||||
|
|
||||||
|
## 背景
|
||||||
|
|
||||||
|
OCP 当前架构：`Tool → OCP (localhost:3456) → Claude CLI → Anthropic`
|
||||||
|
|
||||||
|
核心风险：Claude CLI proxy 属于灰色地带，Anthropic 随时可能封堵。
|
||||||
|
|
||||||
|
核心资产（不依赖 CLI 的部分）：
|
||||||
|
- OpenAI 兼容 API 层
|
||||||
|
- Session 管理（会话隔离、TTL、复用）
|
||||||
|
- Usage 监控（plan limits、per-model stats、实时查询）
|
||||||
|
- 运行时调参（timeout、并发、prompt 限制）
|
||||||
|
- `/ocp` 命令集（Telegram/Discord/CLI）
|
||||||
|
- 多实例并发控制
|
||||||
|
|
||||||
|
## 定位
|
||||||
|
|
||||||
|
> OCP is a cost-control and model-routing layer for AI agents, optimized for subscription-based usage.
|
||||||
|
|
||||||
|
**不是 proxy，不是 gateway，不是 LiteLLM clone。**
|
||||||
|
**是：让 Agent 用模型这件事变得可控 — 尤其是成本。**
|
||||||
|
|
||||||
|
核心差异点：OCP 有 agent 上下文，知道是谁在请求、在做什么任务、已经花了多少。LiteLLM 只看到 HTTP 请求。
|
||||||
|
|
||||||
|
## 目标
|
||||||
|
|
||||||
|
将 OCP 从 "Claude CLI proxy" 重构为 "Agent 成本控制器"，支持可扩展的多后端，保留全部管理能力。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚠️ 第零原则：不动现有功能
|
||||||
|
|
||||||
|
OCP 作为 Claude CLI proxy 是当前的核心价值和用户吸引力。所有 v4 重构必须遵守：
|
||||||
|
|
||||||
|
- **外部 API 不变** — `/v1/chat/completions`、`/usage`、`/health`、`/settings` 签名不变
|
||||||
|
- **行为不变** — 现有用户升级后零感知，所有 session/timeout/并发逻辑保持原样
|
||||||
|
- **逐步替换** — 先抽模块，跑稳了再切；任何时候回归测试不过就回滚
|
||||||
|
- **新 backend 是增量** — 加 OpenAI/Ollama 是新能力，不改动 Claude CLI 路径
|
||||||
|
|
||||||
|
Phase 1 的定义就是：**拆完之后跟没拆一样。** 拆出问题就说明拆法有误。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 架构关键决策（三轮 Design Review 定稿）
|
||||||
|
|
||||||
|
### 决策 1: 路由分两层 — Model Registry + Agent Policy
|
||||||
|
|
||||||
|
**三轮讨论结论：**
|
||||||
|
- 第一轮建议"删掉 model routing" → 错误，非 agent 客户端（Cline/Aider）只带 model 名
|
||||||
|
- 反驳：model-to-backend mapping 是基础设施，不是 routing rule
|
||||||
|
- 校准：物理拆成两个模块，不要混在一起
|
||||||
|
|
||||||
|
**最终设计 — 两层分离：**
|
||||||
|
|
||||||
|
```
|
||||||
|
🧱 Layer 1: Model Registry（基础设施层）
|
||||||
|
model-registry.mjs
|
||||||
|
静态映射，无条件，必须存在
|
||||||
|
"claude-sonnet-4-6" → claude-cli backend
|
||||||
|
"gpt-4o" → openai backend
|
||||||
|
"qwen3" → ollama backend
|
||||||
|
|
||||||
|
🧠 Layer 2: Agent Policy（策略层）
|
||||||
|
agent-router.mjs
|
||||||
|
动态决策，可覆盖，可选
|
||||||
|
agent "coder" → override to claude-cli
|
||||||
|
agent "interpreter" → override to ollama
|
||||||
|
```
|
||||||
|
|
||||||
|
**路由决策顺序（写死）：**
|
||||||
|
```
|
||||||
|
1. Agent policy override → 有 agent identity 且有配置？用它
|
||||||
|
2. Model registry → 模型属于哪个 backend？路由过去
|
||||||
|
3. Default backend → 都没匹配？全局默认
|
||||||
|
4. Fallback chain → 上面选的挂了？降级
|
||||||
|
```
|
||||||
|
|
||||||
|
### 决策 2: 统一 Streaming 协议
|
||||||
|
|
||||||
|
**所有 adapter 必须输出统一的内部 chunk 格式：**
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
{
|
||||||
|
type: "delta" | "done" | "error",
|
||||||
|
content?: string,
|
||||||
|
model: string,
|
||||||
|
backend: string,
|
||||||
|
usage?: { promptTokens: number, completionTokens: number }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
server.mjs 只处理统一协议 → 转换为 OpenAI SSE 输出。
|
||||||
|
|
||||||
|
### 决策 3: 成本标记 — actual / estimated / free
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
{
|
||||||
|
cost: {
|
||||||
|
value: 0.23,
|
||||||
|
type: "actual" | "estimated" | "free"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Backend | cost type | 来源 |
|
||||||
|
|---------|-----------|------|
|
||||||
|
| OpenAI API | actual | API 返回 usage |
|
||||||
|
| Claude CLI | estimated | prompt chars × 估价 |
|
||||||
|
| Ollama | free | 本地运行 |
|
||||||
|
|
||||||
|
`/ocp cost` 输出明确标注类型，不误导用户。
|
||||||
|
|
||||||
|
### 决策 4: Fallback — 只在 First-Byte 前触发
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const allowFallback = !hasStartedStreaming;
|
||||||
|
```
|
||||||
|
|
||||||
|
Phase 3 严格限制：已输出 ≥1 个 delta → 不 fallback，直接报错。
|
||||||
|
flag 预留，未来扩展 streaming fallback 只改判断条件。
|
||||||
|
|
||||||
|
### 决策 5: Agent Identity — 结构预留，实现从简
|
||||||
|
|
||||||
|
**优先级：**
|
||||||
|
```
|
||||||
|
1. x-ocp-agent header → OpenClaw 自动附加
|
||||||
|
2. Session metadata → 首次请求绑定，后续复用
|
||||||
|
3. (future) Inferred → 按使用模式推断（预留接口）
|
||||||
|
4. default → 兜底
|
||||||
|
```
|
||||||
|
|
||||||
|
**现在只实现 1 + 2 + 4。** 但接口预留 inferred 扩展点。
|
||||||
|
|
||||||
|
为什么重要：未来 Cline coding mode 和 chat mode 可能需要不同路由，如果没有 identity hook 以后很难加。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Backend Adapter 设计
|
||||||
|
|
||||||
|
### 接口
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
class BackendAdapter {
|
||||||
|
get id() // "claude-cli" | "openai-api" | "ollama"
|
||||||
|
get displayName()
|
||||||
|
get models()
|
||||||
|
get costType() // "actual" | "estimated" | "free"
|
||||||
|
get tier() // "core" | "community"
|
||||||
|
|
||||||
|
async healthCheck() → { ok, message, latencyMs }
|
||||||
|
async *chatCompletion(request) → AsyncGenerator<UnifiedChunk>
|
||||||
|
async initialize(config)
|
||||||
|
async shutdown()
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Backend Tier 体系
|
||||||
|
|
||||||
|
OCP provides a pluggable backend adapter interface. Core adapters are maintained officially; community adapters are contributed by users.
|
||||||
|
|
||||||
|
```
|
||||||
|
/ocp backends 输出示例：
|
||||||
|
|
||||||
|
claude-cli ✓ healthy 12ms (core)
|
||||||
|
openai ✓ healthy 89ms (core)
|
||||||
|
ollama ✓ healthy 5ms (core)
|
||||||
|
deepseek ✓ healthy 120ms (community)
|
||||||
|
```
|
||||||
|
|
||||||
|
架构不限制 backend 数量。我们主动维护 3 个 core adapter（Claude CLI、OpenAI 兼容、Ollama），接口开放，社区可贡献更多。
|
||||||
|
|
||||||
|
OpenAI 兼容的服务（DeepSeek、Groq、Together 等）直接复用 `OpenAiApiAdapter`，只需换 baseUrl + apiKey，不算新 adapter。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 分阶段实施
|
||||||
|
|
||||||
|
### Phase 1: 后端抽象 + Model Registry（v4.0-alpha）
|
||||||
|
**目标：** 不改变外部行为，内部重构
|
||||||
|
|
||||||
|
- [ ] 定义 `BackendAdapter` 接口 + `UnifiedChunk` 类型
|
||||||
|
- [ ] `model-registry.mjs` — 模型到 backend 的静态映射
|
||||||
|
- `getBackendForModel(model) → backendId`
|
||||||
|
- [ ] `agent-router.mjs` — 路由决策引擎
|
||||||
|
- `resolveBackend({ agent, model }) → backendId`
|
||||||
|
- 实现决策顺序：agent override → model registry → default → fallback
|
||||||
|
- [ ] 将 Claude CLI 逻辑封装为 `ClaudeCliAdapter`
|
||||||
|
- stdout → UnifiedChunk 转换
|
||||||
|
- costType = "estimated"
|
||||||
|
- [ ] 抽取 `SessionManager`（从 server.mjs）
|
||||||
|
- [ ] 抽取 `StatsCollector`（per-agent, per-backend）
|
||||||
|
- [ ] Config v4 格式（带 version + validation）：
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"version": "4.0",
|
||||||
|
"backends": {
|
||||||
|
"claude-cli": {
|
||||||
|
"type": "claude-cli",
|
||||||
|
"tier": "core",
|
||||||
|
"enabled": true,
|
||||||
|
"models": ["claude-sonnet-4-6", "claude-opus-4-6", "claude-haiku-4-5"],
|
||||||
|
"maxConcurrent": 4,
|
||||||
|
"timeout": { "firstByte": 120000, "overall": 300000 },
|
||||||
|
"estimatedCostPerMTok": { "input": 3.0, "output": 15.0 }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"agents": {
|
||||||
|
"default": { "preferred": "claude-cli", "fallback": null }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] 启动时 config validation
|
||||||
|
- [ ] `/ocp backends` 命令
|
||||||
|
- [ ] 所有现有测试通过，行为不变
|
||||||
|
|
||||||
|
**验收标准：** 现有用户升级后零感知变化
|
||||||
|
|
||||||
|
### Phase 2: OpenAI API 后端（v4.0-beta）
|
||||||
|
**目标：** 第二个 backend，验证 adapter 抽象
|
||||||
|
|
||||||
|
- [ ] `OpenAiApiAdapter`
|
||||||
|
- SSE → UnifiedChunk
|
||||||
|
- costType = "actual"
|
||||||
|
- 支持任何 OpenAI 兼容端点（baseUrl + apiKey）
|
||||||
|
- [ ] Model registry 自动注册新 backend 的 models
|
||||||
|
- [ ] Agent routing 生效：
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"agents": {
|
||||||
|
"default": { "preferred": "claude-cli", "fallback": "openai" },
|
||||||
|
"interpreter": { "preferred": "openai/gpt-4o-mini", "fallback": null }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] Non-streaming fallback（`allowFallback = !hasStartedStreaming`）
|
||||||
|
- [ ] Stats 按 backend + agent 分组
|
||||||
|
|
||||||
|
**验收标准：** claude-cli 挂了 → default agent 自动 fallback 到 openai
|
||||||
|
|
||||||
|
### Phase 3: Ollama + Agent Routing 稳定化（v4.0-rc）
|
||||||
|
**目标：** 摆脱付费依赖，agent routing 跑稳
|
||||||
|
|
||||||
|
- [ ] `OllamaAdapter`
|
||||||
|
- 自动发现模型
|
||||||
|
- costType = "free"
|
||||||
|
- [ ] Agent routing 完整覆盖所有 agent：
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"agents": {
|
||||||
|
"main": { "preferred": "claude-cli/opus", "fallback": "openai" },
|
||||||
|
"tech_geek": { "preferred": "claude-cli/sonnet", "fallback": "ollama/qwen3" },
|
||||||
|
"interpreter": { "preferred": "ollama/qwen3", "fallback": null }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] `/ocp routing` — 显示每个 agent 当前走哪个 backend
|
||||||
|
- [ ] 端到端测试：Claude CLI 下线 → agent 自动 fallback → 恢复后自动回切
|
||||||
|
|
||||||
|
**验收标准：** 三个 backend 同时运行，每个 agent 走不同路径，稳定无错
|
||||||
|
|
||||||
|
### Phase 4: Budget + Cost Control（v4.1）
|
||||||
|
**目标：** 核心差异化 — agent-aware 成本控制
|
||||||
|
|
||||||
|
**前置条件：** Phase 3 routing 稳定（先确定流量怎么走 → 再控制花多少钱）
|
||||||
|
|
||||||
|
**安全原则：** 没有显式配置 budget 的 agent → 行为等同于 v3（不限制、不降级、不告警）。Budget 是 opt-in，不是 opt-out。
|
||||||
|
|
||||||
|
- [ ] Per-agent 预算：
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"agents": {
|
||||||
|
"main": {
|
||||||
|
"preferred": "claude-cli/opus",
|
||||||
|
"fallback": "openai",
|
||||||
|
"dailyBudget": { "limit": 3.00, "onExhausted": "downgrade" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"budget": {
|
||||||
|
"global": { "daily": 10.00, "onExhausted": "warn" }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] 超预算行为：downgrade | block | warn
|
||||||
|
- [ ] `/ocp cost` — per-agent 成本（标注 actual/estimated/free）
|
||||||
|
- [ ] Telegram 告警（预算 80%、backend 连续失败）
|
||||||
|
|
||||||
|
### Phase 5: Anthropic 官方 API（v4.2，等时机）
|
||||||
|
- [ ] `AnthropicApiAdapter`（官方 SDK）
|
||||||
|
- [ ] 迁移指南：一行配置切换
|
||||||
|
- [ ] costType = "actual"
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 文件结构（目标）
|
||||||
|
|
||||||
|
```
|
||||||
|
claude-proxy/
|
||||||
|
├── server.mjs → HTTP 层（统一 chunk → SSE 输出）
|
||||||
|
├── model-registry.mjs → 模型注册表（model → backend 映射）
|
||||||
|
├── agent-router.mjs → Agent 路由引擎（策略层）
|
||||||
|
├── session-manager.mjs → 会话管理
|
||||||
|
├── stats-collector.mjs → 统计 + 成本追踪
|
||||||
|
├── config.mjs → 配置加载 + 版本验证
|
||||||
|
├── unified-chunk.mjs → UnifiedChunk 类型定义
|
||||||
|
├── backends/
|
||||||
|
│ ├── base.mjs → BackendAdapter 接口
|
||||||
|
│ ├── claude-cli.mjs → Claude CLI（core）
|
||||||
|
│ ├── openai-api.mjs → OpenAI 兼容（core）
|
||||||
|
│ └── ollama.mjs → Ollama 本地（core）
|
||||||
|
├── setup.mjs → 安装向导
|
||||||
|
└── ocp-plugin/ → Gateway 命令插件
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 配置完整示例（目标状态）
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"version": "4.0",
|
||||||
|
"port": 3456,
|
||||||
|
|
||||||
|
"backends": {
|
||||||
|
"claude-cli": {
|
||||||
|
"type": "claude-cli",
|
||||||
|
"tier": "core",
|
||||||
|
"enabled": true,
|
||||||
|
"models": ["claude-sonnet-4-6", "claude-opus-4-6"],
|
||||||
|
"maxConcurrent": 4,
|
||||||
|
"timeout": { "firstByte": 120000, "overall": 300000 },
|
||||||
|
"estimatedCostPerMTok": { "input": 3.0, "output": 15.0 }
|
||||||
|
},
|
||||||
|
"openai": {
|
||||||
|
"type": "openai-api",
|
||||||
|
"tier": "core",
|
||||||
|
"enabled": true,
|
||||||
|
"baseUrl": "https://openrouter.ai/api/v1",
|
||||||
|
"apiKey": "${OPENROUTER_API_KEY}",
|
||||||
|
"models": ["anthropic/claude-3.5-sonnet", "google/gemini-2.0-flash"],
|
||||||
|
"maxConcurrent": 10,
|
||||||
|
"timeout": { "firstByte": 30000, "overall": 120000 }
|
||||||
|
},
|
||||||
|
"ollama": {
|
||||||
|
"type": "ollama",
|
||||||
|
"tier": "core",
|
||||||
|
"enabled": true,
|
||||||
|
"baseUrl": "http://localhost:11434",
|
||||||
|
"models": "auto"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
"agents": {
|
||||||
|
"default": {
|
||||||
|
"preferred": "claude-cli",
|
||||||
|
"fallback": "openai"
|
||||||
|
},
|
||||||
|
"main": {
|
||||||
|
"preferred": "claude-cli/claude-opus-4-6",
|
||||||
|
"fallback": "openai",
|
||||||
|
"dailyBudget": { "limit": 3.00, "onExhausted": "downgrade" }
|
||||||
|
},
|
||||||
|
"tech_geek": {
|
||||||
|
"preferred": "claude-cli/claude-sonnet-4-6",
|
||||||
|
"fallback": "ollama/qwen3",
|
||||||
|
"dailyBudget": { "limit": 2.00, "onExhausted": "fallback" }
|
||||||
|
},
|
||||||
|
"interpreter": {
|
||||||
|
"preferred": "ollama/qwen3",
|
||||||
|
"fallback": null
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
"fallback": {
|
||||||
|
"onlyBeforeFirstByte": true,
|
||||||
|
"triggers": ["timeout", "rate-limit", "connection-error"],
|
||||||
|
"maxRetries": 1
|
||||||
|
},
|
||||||
|
|
||||||
|
"budget": {
|
||||||
|
"global": { "daily": 10.00, "onExhausted": "warn" },
|
||||||
|
"alerts": { "threshold": 0.8, "channel": "telegram" }
|
||||||
|
},
|
||||||
|
|
||||||
|
"sessions": { "ttl": 3600000, "maxPromptChars": 150000 },
|
||||||
|
"monitoring": { "retainHours": 72 }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 风险与应对
|
||||||
|
|
||||||
|
| 风险 | 应对 |
|
||||||
|
|------|------|
|
||||||
|
| Claude CLI 被封 | 切换 backend，上层不变 |
|
||||||
|
| Anthropic 出订阅 API | 实现 anthropic-api adapter，成本控制层继续有价值 |
|
||||||
|
| LiteLLM 竞争 | 不竞争。OCP 做 agent-aware 成本控制，赛道不同 |
|
||||||
|
| 成本数据不准 | cost type 标记 actual/estimated/free |
|
||||||
|
| Streaming fallback 拼接 | Phase 3 只做 non-streaming fallback |
|
||||||
|
| 过度工程 | 只维护 3 个 core backend，phase 严格拆分 |
|
||||||
|
|
||||||
|
## 优先级
|
||||||
|
|
||||||
|
| Phase | 时间 | 内容 | 交付物 |
|
||||||
|
|-------|------|------|--------|
|
||||||
|
| 1 | 本周 | 重构：adapter 接口 + model registry + agent router | 行为不变，架构就位 |
|
||||||
|
| 2 | 1-2 周 | OpenAI backend | 两个 backend 跑通 |
|
||||||
|
| 3 | 2-3 周 | Ollama + Agent routing 稳定化 | 三 backend + per-agent 路由 |
|
||||||
|
| 4 | 按需 | Budget + Cost control | 成本可控 |
|
||||||
|
| 5 | 等时机 | Anthropic 官方 API | 逃生通道 |
|
||||||
|
|
||||||
|
## 成功标准
|
||||||
|
|
||||||
|
1. Claude CLI 被封的那天，改一行配置，所有 agent 继续运行
|
||||||
|
2. `/ocp cost` 每个 agent 花了多少（actual/estimated/free）
|
||||||
|
3. 翻译 agent 自动用 Ollama（$0），coding agent 用 Claude
|
||||||
|
4. 超预算自动降级，不需人工干预
|
||||||
|
5. 新 backend 只需实现 BackendAdapter 接口 + 一个文件
|
||||||
|
|
||||||
|
## 设计审核记录
|
||||||
|
|
||||||
|
| 轮次 | 审核方 | 关键结论 |
|
||||||
|
|------|--------|----------|
|
||||||
|
| 第一轮 | AI-A | 定位从 proxy 转向成本控制器；识别 5 个 P0 问题 |
|
||||||
|
| 第二轮 | Claude（本项目开发者） | 反驳：不能删 model routing（基础设施层）；backend 数量是运营决策不是架构约束；budget 拆到 Phase 4 |
|
||||||
|
| 第三轮 | AI-A | 校准：model registry 与 routing 物理拆开两个模块；agent identity 结构预留 inferred 扩展点；backend 加 tier 标记 |
|
||||||
|
|
||||||
|
## 一句话总结
|
||||||
|
|
||||||
|
**短期靠套利活（Claude CLI proxy），长期靠控制力活（Agent 成本控制器）。两条腿走路，CLI 死了上层不死。**
|
||||||
@@ -0,0 +1,171 @@
|
|||||||
|
/**
|
||||||
|
* Agent Router — Layer 2 of OCP routing.
|
||||||
|
*
|
||||||
|
* Routing decision order (written in stone):
|
||||||
|
* 1. Agent policy override → agent has explicit backend config? Use it.
|
||||||
|
* 2. Model registry → model maps to a backend? Route there.
|
||||||
|
* 3. Default backend → global fallback.
|
||||||
|
* 4. Fallback chain → chosen backend is down? Try next.
|
||||||
|
*
|
||||||
|
* Agent identity sources (priority):
|
||||||
|
* 1. x-ocp-agent header
|
||||||
|
* 2. Session metadata (bound on first request)
|
||||||
|
* 3. (future) Inferred from usage patterns
|
||||||
|
* 4. "default"
|
||||||
|
*/
|
||||||
|
|
||||||
|
export class AgentRouter {
|
||||||
|
/**
|
||||||
|
* @param {Object} options
|
||||||
|
* @param {import("./model-registry.mjs").ModelRegistry} options.registry
|
||||||
|
* @param {Map<string, import("./backends/base.mjs").BackendAdapter>} options.backends
|
||||||
|
* @param {Object} options.agentConfig — agent routing rules from config
|
||||||
|
* @param {string} options.defaultBackend — fallback backend ID
|
||||||
|
*/
|
||||||
|
constructor({ registry, backends, agentConfig = {}, defaultBackend = "claude-cli" }) {
|
||||||
|
this._registry = registry;
|
||||||
|
this._backends = backends;
|
||||||
|
this._agentConfig = agentConfig;
|
||||||
|
this._defaultBackend = defaultBackend;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Identify the agent from a request.
|
||||||
|
*
|
||||||
|
* @param {Object} req - HTTP request
|
||||||
|
* @param {Object} [sessionMeta] - Session metadata (may contain agent binding)
|
||||||
|
* @returns {string} Agent name
|
||||||
|
*/
|
||||||
|
identifyAgent(req, sessionMeta = null) {
|
||||||
|
// 1. Explicit header
|
||||||
|
const headerAgent = req.headers?.["x-ocp-agent"];
|
||||||
|
if (headerAgent) return headerAgent;
|
||||||
|
|
||||||
|
// 2. Session binding
|
||||||
|
if (sessionMeta?.agent) return sessionMeta.agent;
|
||||||
|
|
||||||
|
// 3. (future) Inferred — not implemented yet
|
||||||
|
|
||||||
|
// 4. Default
|
||||||
|
return "default";
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Resolve which backend + model to use for a request.
|
||||||
|
*
|
||||||
|
* @param {Object} params
|
||||||
|
* @param {string} params.agent - Agent name (from identifyAgent)
|
||||||
|
* @param {string} params.model - Requested model ID
|
||||||
|
* @returns {{backendId: string, model: string, reason: string}}
|
||||||
|
*/
|
||||||
|
resolve({ agent, model }) {
|
||||||
|
// Step 1: Agent policy override
|
||||||
|
const agentRule = this._agentConfig[agent];
|
||||||
|
if (agentRule?.preferred) {
|
||||||
|
const { backendId, modelId } = this._parsePreferred(agentRule.preferred);
|
||||||
|
const backend = this._backends.get(backendId);
|
||||||
|
if (backend?.enabled) {
|
||||||
|
const resolvedModel = modelId || model;
|
||||||
|
return {
|
||||||
|
backendId,
|
||||||
|
model: resolvedModel,
|
||||||
|
reason: `agent-policy(${agent})`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
// Agent preferred backend is down — try fallback
|
||||||
|
if (agentRule.fallback) {
|
||||||
|
const fb = this._parseFallback(agentRule.fallback, model);
|
||||||
|
if (fb) return { ...fb, reason: `agent-fallback(${agent})` };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 2: Model registry
|
||||||
|
const registryEntry = this._registry.resolve(model);
|
||||||
|
if (registryEntry) {
|
||||||
|
const backend = this._backends.get(registryEntry.backendId);
|
||||||
|
if (backend?.enabled) {
|
||||||
|
return {
|
||||||
|
backendId: registryEntry.backendId,
|
||||||
|
model: registryEntry.canonical,
|
||||||
|
reason: "model-registry",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 2b: Check adapters for alias support
|
||||||
|
for (const [id, backend] of this._backends) {
|
||||||
|
if (backend.enabled && backend.supportsModel(model)) {
|
||||||
|
const resolved = typeof backend.resolveModel === "function"
|
||||||
|
? backend.resolveModel(model)
|
||||||
|
: model;
|
||||||
|
return {
|
||||||
|
backendId: id,
|
||||||
|
model: resolved,
|
||||||
|
reason: `adapter-match(${id})`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 3: Default backend
|
||||||
|
const defaultAgent = this._agentConfig.default;
|
||||||
|
if (defaultAgent?.preferred) {
|
||||||
|
const { backendId } = this._parsePreferred(defaultAgent.preferred);
|
||||||
|
if (this._backends.get(backendId)?.enabled) {
|
||||||
|
return { backendId, model, reason: "default-agent-policy" };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (this._backends.get(this._defaultBackend)?.enabled) {
|
||||||
|
return {
|
||||||
|
backendId: this._defaultBackend,
|
||||||
|
model,
|
||||||
|
reason: "default-backend",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 4: Fallback — try any enabled backend
|
||||||
|
for (const [id, backend] of this._backends) {
|
||||||
|
if (backend.enabled) {
|
||||||
|
return { backendId: id, model, reason: `last-resort(${id})` };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return { backendId: null, model, reason: "no-backend-available" };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Parse a preferred string like "claude-cli" or "claude-cli/claude-opus-4-6"
|
||||||
|
*/
|
||||||
|
_parsePreferred(preferred) {
|
||||||
|
if (preferred.includes("/")) {
|
||||||
|
const [backendId, modelId] = preferred.split("/", 2);
|
||||||
|
return { backendId, modelId };
|
||||||
|
}
|
||||||
|
return { backendId: preferred, modelId: null };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Parse a fallback value and return routing info.
|
||||||
|
*/
|
||||||
|
_parseFallback(fallback, model) {
|
||||||
|
if (!fallback) return null;
|
||||||
|
const { backendId, modelId } = this._parsePreferred(fallback);
|
||||||
|
const backend = this._backends.get(backendId);
|
||||||
|
if (!backend?.enabled) return null;
|
||||||
|
return { backendId, model: modelId || model };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get routing info for display (e.g. /ocp routing command).
|
||||||
|
*/
|
||||||
|
getRoutingTable() {
|
||||||
|
const table = {};
|
||||||
|
for (const [agent, config] of Object.entries(this._agentConfig)) {
|
||||||
|
table[agent] = {
|
||||||
|
preferred: config.preferred || null,
|
||||||
|
fallback: config.fallback || null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return table;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
/**
|
||||||
|
* BackendAdapter — base class for all OCP backends.
|
||||||
|
*
|
||||||
|
* Each backend adapter MUST:
|
||||||
|
* 1. Implement chatCompletion() as an async generator yielding UnifiedChunks
|
||||||
|
* 2. Implement healthCheck()
|
||||||
|
* 3. Report its costType (actual | estimated | free)
|
||||||
|
* 4. Report its tier (core | community)
|
||||||
|
*
|
||||||
|
* Adapters do NOT handle:
|
||||||
|
* - Session management (SessionManager does that)
|
||||||
|
* - Stats collection (StatsCollector does that)
|
||||||
|
* - Routing decisions (agent-router does that)
|
||||||
|
*/
|
||||||
|
|
||||||
|
export class BackendAdapter {
|
||||||
|
/**
|
||||||
|
* @param {Object} config - Backend-specific configuration
|
||||||
|
*/
|
||||||
|
constructor(config = {}) {
|
||||||
|
this._config = config;
|
||||||
|
this._enabled = config.enabled !== false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @returns {string} Unique backend identifier */
|
||||||
|
get id() { throw new Error("BackendAdapter.id must be implemented"); }
|
||||||
|
|
||||||
|
/** @returns {string} Human-readable name */
|
||||||
|
get displayName() { throw new Error("BackendAdapter.displayName must be implemented"); }
|
||||||
|
|
||||||
|
/** @returns {string[]} List of supported model IDs */
|
||||||
|
get models() { throw new Error("BackendAdapter.models must be implemented"); }
|
||||||
|
|
||||||
|
/** @returns {"actual"|"estimated"|"free"} */
|
||||||
|
get costType() { throw new Error("BackendAdapter.costType must be implemented"); }
|
||||||
|
|
||||||
|
/** @returns {"core"|"community"} */
|
||||||
|
get tier() { return "core"; }
|
||||||
|
|
||||||
|
/** @returns {boolean} */
|
||||||
|
get enabled() { return this._enabled; }
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Initialize the backend (called once at startup).
|
||||||
|
* Override to do async setup (e.g. validate binary, check API key).
|
||||||
|
*/
|
||||||
|
async initialize() {}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Shutdown the backend (called on graceful shutdown).
|
||||||
|
* Override to clean up resources.
|
||||||
|
*/
|
||||||
|
async shutdown() {}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if this backend is healthy and reachable.
|
||||||
|
* @returns {Promise<{ok: boolean, message: string, latencyMs: number}>}
|
||||||
|
*/
|
||||||
|
async healthCheck() {
|
||||||
|
return { ok: false, message: "healthCheck not implemented", latencyMs: 0 };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Execute a chat completion request.
|
||||||
|
*
|
||||||
|
* MUST yield UnifiedChunk objects:
|
||||||
|
* { type: "delta", content: "...", model, backend } — streaming content
|
||||||
|
* { type: "done", model, backend, usage?, cost? } — stream finished
|
||||||
|
* { type: "error", error: "...", model, backend } — error occurred
|
||||||
|
*
|
||||||
|
* @param {Object} request
|
||||||
|
* @param {string} request.model - Canonical model ID
|
||||||
|
* @param {Array} request.messages - OpenAI-format messages array
|
||||||
|
* @param {Object} [request.session] - Session info { uuid, resume }
|
||||||
|
* @param {Object} [request.options] - Additional options (temperature, etc.)
|
||||||
|
* @yields {import("../unified-chunk.mjs").UnifiedChunk}
|
||||||
|
*/
|
||||||
|
async *chatCompletion(request) {
|
||||||
|
throw new Error("BackendAdapter.chatCompletion must be implemented");
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if this backend supports a given model.
|
||||||
|
* Default: exact match against models list.
|
||||||
|
* Override for pattern matching (e.g. "gpt-*" → openai).
|
||||||
|
*/
|
||||||
|
supportsModel(model) {
|
||||||
|
return this.models.includes(model);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,393 @@
|
|||||||
|
/**
|
||||||
|
* ClaudeCliAdapter — OCP backend for Claude via local CLI binary.
|
||||||
|
*
|
||||||
|
* Spawns `claude -p` processes for each request. Converts CLI stdout
|
||||||
|
* into UnifiedChunk streaming protocol.
|
||||||
|
*
|
||||||
|
* costType: "estimated" (no actual cost data from CLI)
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { spawn, execFileSync } from "node:child_process";
|
||||||
|
import { accessSync, constants } from "node:fs";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { BackendAdapter } from "./base.mjs";
|
||||||
|
import { delta, done, error } from "../unified-chunk.mjs";
|
||||||
|
|
||||||
|
// Default models exposed by this backend
|
||||||
|
const DEFAULT_MODELS = [
|
||||||
|
"claude-opus-4-6",
|
||||||
|
"claude-sonnet-4-6",
|
||||||
|
"claude-haiku-4-5-20251001",
|
||||||
|
];
|
||||||
|
|
||||||
|
// Alias → canonical model mapping
|
||||||
|
const MODEL_ALIASES = {
|
||||||
|
"claude-opus-4": "claude-opus-4-6",
|
||||||
|
"claude-haiku-4": "claude-haiku-4-5-20251001",
|
||||||
|
"claude-haiku-4-5": "claude-haiku-4-5-20251001",
|
||||||
|
"opus": "claude-opus-4-6",
|
||||||
|
"sonnet": "claude-sonnet-4-6",
|
||||||
|
"haiku": "claude-haiku-4-5-20251001",
|
||||||
|
};
|
||||||
|
|
||||||
|
export class ClaudeCliAdapter extends BackendAdapter {
|
||||||
|
constructor(config = {}) {
|
||||||
|
super(config);
|
||||||
|
this._claudeBin = null;
|
||||||
|
this._activeProcesses = new Set();
|
||||||
|
this._maxConcurrent = config.maxConcurrent || 8;
|
||||||
|
this._timeout = config.timeout?.overall || 300000;
|
||||||
|
this._skipPermissions = config.skipPermissions || false;
|
||||||
|
this._allowedTools = config.allowedTools || [
|
||||||
|
"Bash", "Read", "Write", "Edit", "Glob", "Grep",
|
||||||
|
"WebSearch", "WebFetch", "Agent",
|
||||||
|
];
|
||||||
|
this._systemPrompt = config.systemPrompt || "";
|
||||||
|
this._mcpConfig = config.mcpConfig || "";
|
||||||
|
this._modelList = config.models || DEFAULT_MODELS;
|
||||||
|
|
||||||
|
// Timeout tiers: per-model base + per-prompt-char scaling
|
||||||
|
this._timeoutTiers = config.timeoutTiers || {
|
||||||
|
opus: { base: 150000, perPromptChar: 0.00050 },
|
||||||
|
sonnet: { base: 120000, perPromptChar: 0.00050 },
|
||||||
|
haiku: { base: 45000, perPromptChar: 0.00010 },
|
||||||
|
};
|
||||||
|
|
||||||
|
// Estimated cost per MTok (for cost tracking)
|
||||||
|
this._estimatedCost = config.estimatedCostPerMTok || {
|
||||||
|
input: 3.0,
|
||||||
|
output: 15.0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
get id() { return "claude-cli"; }
|
||||||
|
get displayName() { return "Claude CLI"; }
|
||||||
|
get models() { return this._modelList; }
|
||||||
|
get costType() { return "estimated"; }
|
||||||
|
get tier() { return "core"; }
|
||||||
|
get activeProcessCount() { return this._activeProcesses.size; }
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Resolve a model ID, handling aliases.
|
||||||
|
* Returns canonical CLI model ID or the input if no alias found.
|
||||||
|
*/
|
||||||
|
resolveModel(model) {
|
||||||
|
return MODEL_ALIASES[model] || model;
|
||||||
|
}
|
||||||
|
|
||||||
|
supportsModel(model) {
|
||||||
|
const canonical = this.resolveModel(model);
|
||||||
|
return this._modelList.includes(canonical) || model in MODEL_ALIASES;
|
||||||
|
}
|
||||||
|
|
||||||
|
async initialize() {
|
||||||
|
this._claudeBin = this._resolveBinary();
|
||||||
|
}
|
||||||
|
|
||||||
|
async shutdown() {
|
||||||
|
for (const proc of this._activeProcesses) {
|
||||||
|
try { proc.kill("SIGTERM"); } catch {}
|
||||||
|
}
|
||||||
|
// Force kill after 5s
|
||||||
|
if (this._activeProcesses.size > 0) {
|
||||||
|
await new Promise(r => setTimeout(r, 5000));
|
||||||
|
for (const proc of this._activeProcesses) {
|
||||||
|
try { proc.kill("SIGKILL"); } catch {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async healthCheck() {
|
||||||
|
const t0 = Date.now();
|
||||||
|
try {
|
||||||
|
const env = this._cleanEnv();
|
||||||
|
execFileSync(this._claudeBin, ["auth", "status"], {
|
||||||
|
encoding: "utf8", timeout: 10000, env,
|
||||||
|
});
|
||||||
|
return { ok: true, message: "authenticated", latencyMs: Date.now() - t0 };
|
||||||
|
} catch (e) {
|
||||||
|
return {
|
||||||
|
ok: false,
|
||||||
|
message: (e.stderr || e.message || "").slice(0, 200),
|
||||||
|
latencyMs: Date.now() - t0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Execute a chat completion via Claude CLI.
|
||||||
|
* Yields UnifiedChunks as stdout data arrives.
|
||||||
|
*/
|
||||||
|
async *chatCompletion(request) {
|
||||||
|
const { model, messages, session, promptText } = request;
|
||||||
|
const cliModel = this.resolveModel(model);
|
||||||
|
|
||||||
|
if (this._activeProcesses.size >= this._maxConcurrent) {
|
||||||
|
yield error(
|
||||||
|
`concurrency limit reached (${this._activeProcesses.size}/${this._maxConcurrent})`,
|
||||||
|
cliModel, this.id,
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const prompt = promptText || this._messagesToPrompt(messages);
|
||||||
|
const cliArgs = this._buildCliArgs(cliModel, session);
|
||||||
|
const env = this._cleanEnv();
|
||||||
|
|
||||||
|
const proc = spawn(this._claudeBin, cliArgs, {
|
||||||
|
env, stdio: ["pipe", "pipe", "pipe"],
|
||||||
|
});
|
||||||
|
this._activeProcesses.add(proc);
|
||||||
|
|
||||||
|
const t0 = Date.now();
|
||||||
|
const firstByteTimeoutMs = this._computeFirstByteTimeout(cliModel, prompt.length);
|
||||||
|
let gotFirstByte = false;
|
||||||
|
let totalChars = 0;
|
||||||
|
let stderr = "";
|
||||||
|
let finished = false;
|
||||||
|
|
||||||
|
// Create a promise-based wrapper for the process
|
||||||
|
try {
|
||||||
|
yield* await new Promise((resolve, reject) => {
|
||||||
|
const chunks = [];
|
||||||
|
let streamResolve = null;
|
||||||
|
let streamDone = false;
|
||||||
|
|
||||||
|
// We can't directly yield from inside callbacks, so we collect
|
||||||
|
// and use a different approach — return an async iterable.
|
||||||
|
// Actually, let's use a simpler pattern with a queue.
|
||||||
|
reject(new Error("USE_QUEUE_PATTERN"));
|
||||||
|
});
|
||||||
|
} catch {
|
||||||
|
// Fall through to queue-based pattern below
|
||||||
|
}
|
||||||
|
|
||||||
|
// Queue-based async generator pattern
|
||||||
|
const queue = [];
|
||||||
|
let queueResolve = null;
|
||||||
|
let queueDone = false;
|
||||||
|
|
||||||
|
function enqueue(chunk) {
|
||||||
|
if (queueDone) return;
|
||||||
|
if (queueResolve) {
|
||||||
|
const r = queueResolve;
|
||||||
|
queueResolve = null;
|
||||||
|
r(chunk);
|
||||||
|
} else {
|
||||||
|
queue.push(chunk);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function waitForChunk() {
|
||||||
|
if (queue.length > 0) return Promise.resolve(queue.shift());
|
||||||
|
if (queueDone) return Promise.resolve(null);
|
||||||
|
return new Promise(r => { queueResolve = r; });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Write prompt to stdin
|
||||||
|
proc.stdin.write(prompt);
|
||||||
|
proc.stdin.end();
|
||||||
|
|
||||||
|
proc.stdout.on("data", (d) => {
|
||||||
|
if (!gotFirstByte) {
|
||||||
|
gotFirstByte = true;
|
||||||
|
clearTimeout(firstByteTimer);
|
||||||
|
}
|
||||||
|
const text = d.toString();
|
||||||
|
totalChars += text.length;
|
||||||
|
enqueue(delta(text, cliModel, this.id));
|
||||||
|
});
|
||||||
|
|
||||||
|
proc.stderr.on("data", (d) => { stderr += d; });
|
||||||
|
|
||||||
|
proc.on("close", (code, signal) => {
|
||||||
|
this._activeProcesses.delete(proc);
|
||||||
|
clearTimeout(firstByteTimer);
|
||||||
|
clearTimeout(overallTimer);
|
||||||
|
const elapsed = Date.now() - t0;
|
||||||
|
|
||||||
|
if (code !== 0) {
|
||||||
|
const msg = stderr.slice(0, 300) || `claude exit ${code}`;
|
||||||
|
enqueue(error(msg, cliModel, this.id));
|
||||||
|
} else {
|
||||||
|
// Estimate cost based on prompt + completion chars
|
||||||
|
const estimatedInputTokens = Math.ceil(prompt.length / 4);
|
||||||
|
const estimatedOutputTokens = Math.ceil(totalChars / 4);
|
||||||
|
enqueue(done(cliModel, this.id, {
|
||||||
|
promptTokens: estimatedInputTokens,
|
||||||
|
completionTokens: estimatedOutputTokens,
|
||||||
|
}, {
|
||||||
|
value: (estimatedInputTokens * this._estimatedCost.input +
|
||||||
|
estimatedOutputTokens * this._estimatedCost.output) / 1000000,
|
||||||
|
type: "estimated",
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
queueDone = true;
|
||||||
|
if (queueResolve) {
|
||||||
|
const r = queueResolve;
|
||||||
|
queueResolve = null;
|
||||||
|
r(null);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
proc.on("error", (err) => {
|
||||||
|
this._activeProcesses.delete(proc);
|
||||||
|
clearTimeout(firstByteTimer);
|
||||||
|
clearTimeout(overallTimer);
|
||||||
|
enqueue(error(err.message, cliModel, this.id));
|
||||||
|
queueDone = true;
|
||||||
|
if (queueResolve) {
|
||||||
|
const r = queueResolve;
|
||||||
|
queueResolve = null;
|
||||||
|
r(null);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// First-byte timeout
|
||||||
|
const firstByteTimer = setTimeout(() => {
|
||||||
|
if (!gotFirstByte) {
|
||||||
|
try { proc.kill("SIGTERM"); } catch {}
|
||||||
|
setTimeout(() => { try { proc.kill("SIGKILL"); } catch {} }, 5000);
|
||||||
|
enqueue(error(`first-byte timeout after ${firstByteTimeoutMs}ms`, cliModel, this.id));
|
||||||
|
queueDone = true;
|
||||||
|
}
|
||||||
|
}, firstByteTimeoutMs);
|
||||||
|
|
||||||
|
// Overall timeout
|
||||||
|
const overallTimer = setTimeout(() => {
|
||||||
|
try { proc.kill("SIGTERM"); } catch {}
|
||||||
|
setTimeout(() => { try { proc.kill("SIGKILL"); } catch {} }, 5000);
|
||||||
|
enqueue(error(`request timeout after ${this._timeout}ms`, cliModel, this.id));
|
||||||
|
queueDone = true;
|
||||||
|
}, this._timeout);
|
||||||
|
|
||||||
|
// Yield chunks as they arrive
|
||||||
|
while (true) {
|
||||||
|
const chunk = await waitForChunk();
|
||||||
|
if (chunk === null) break;
|
||||||
|
yield chunk;
|
||||||
|
if (chunk.type === "done" || chunk.type === "error") break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Internal helpers ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
_resolveBinary() {
|
||||||
|
if (process.env.CLAUDE_BIN) {
|
||||||
|
try {
|
||||||
|
accessSync(process.env.CLAUDE_BIN, constants.X_OK);
|
||||||
|
return process.env.CLAUDE_BIN;
|
||||||
|
} catch {
|
||||||
|
throw new Error(`CLAUDE_BIN="${process.env.CLAUDE_BIN}" is set but not executable.`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const candidates = [
|
||||||
|
"/opt/homebrew/bin/claude",
|
||||||
|
"/usr/local/bin/claude",
|
||||||
|
"/usr/bin/claude",
|
||||||
|
join(process.env.HOME || "", ".local/bin/claude"),
|
||||||
|
];
|
||||||
|
for (const p of candidates) {
|
||||||
|
try { accessSync(p, constants.X_OK); return p; } catch {}
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const resolved = execFileSync("which", ["claude"], {
|
||||||
|
encoding: "utf8", timeout: 5000,
|
||||||
|
}).trim();
|
||||||
|
if (resolved) return resolved;
|
||||||
|
} catch {}
|
||||||
|
|
||||||
|
throw new Error(
|
||||||
|
"claude binary not found. Set CLAUDE_BIN or ensure claude is in PATH."
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
_cleanEnv() {
|
||||||
|
const env = { ...process.env };
|
||||||
|
delete env.CLAUDECODE;
|
||||||
|
delete env.ANTHROPIC_API_KEY;
|
||||||
|
delete env.ANTHROPIC_BASE_URL;
|
||||||
|
delete env.ANTHROPIC_AUTH_TOKEN;
|
||||||
|
return env;
|
||||||
|
}
|
||||||
|
|
||||||
|
_buildCliArgs(cliModel, session) {
|
||||||
|
const args = ["-p", "--model", cliModel, "--output-format", "text"];
|
||||||
|
|
||||||
|
if (session?.resume) {
|
||||||
|
args.push("--resume", session.uuid);
|
||||||
|
} else if (session?.uuid) {
|
||||||
|
args.push("--session-id", session.uuid);
|
||||||
|
} else {
|
||||||
|
args.push("--no-session-persistence");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (this._skipPermissions) {
|
||||||
|
args.push("--dangerously-skip-permissions");
|
||||||
|
} else if (this._allowedTools.length > 0) {
|
||||||
|
args.push("--allowedTools", ...this._allowedTools);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (this._systemPrompt) {
|
||||||
|
args.push("--append-system-prompt", this._systemPrompt);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (this._mcpConfig) {
|
||||||
|
args.push("--mcp-config", this._mcpConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
return args;
|
||||||
|
}
|
||||||
|
|
||||||
|
_getModelTier(cliModel) {
|
||||||
|
if (cliModel.includes("opus")) return "opus";
|
||||||
|
if (cliModel.includes("haiku")) return "haiku";
|
||||||
|
return "sonnet";
|
||||||
|
}
|
||||||
|
|
||||||
|
_computeFirstByteTimeout(cliModel, promptLength) {
|
||||||
|
const tierName = this._getModelTier(cliModel);
|
||||||
|
const tier = this._timeoutTiers[tierName] || this._timeoutTiers.sonnet;
|
||||||
|
const timeout = tier.base + Math.floor(promptLength * tier.perPromptChar);
|
||||||
|
return Math.min(timeout, Math.max(this._timeout - 5000, 10000));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert OpenAI-format messages to a plain text prompt for Claude CLI.
|
||||||
|
* Includes truncation guard for oversized prompts.
|
||||||
|
*/
|
||||||
|
_messagesToPrompt(messages, maxChars = 150000) {
|
||||||
|
const full = messages.map((m) => {
|
||||||
|
const text = typeof m.content === "string" ? m.content : JSON.stringify(m.content);
|
||||||
|
if (m.role === "system") return `[System] ${text}`;
|
||||||
|
if (m.role === "assistant") return `[Assistant] ${text}`;
|
||||||
|
return text;
|
||||||
|
});
|
||||||
|
|
||||||
|
const joined = full.join("\n\n");
|
||||||
|
if (joined.length <= maxChars) return joined;
|
||||||
|
|
||||||
|
// Truncation: keep system + recent messages
|
||||||
|
const system = [];
|
||||||
|
const rest = [];
|
||||||
|
for (let i = 0; i < full.length; i++) {
|
||||||
|
if (messages[i].role === "system") system.push(full[i]);
|
||||||
|
else rest.push(full[i]);
|
||||||
|
}
|
||||||
|
|
||||||
|
const systemText = system.join("\n\n");
|
||||||
|
const budget = maxChars - systemText.length - 200;
|
||||||
|
const kept = [];
|
||||||
|
let used = 0;
|
||||||
|
for (let i = rest.length - 1; i >= 0; i--) {
|
||||||
|
if (used + rest[i].length + 2 > budget) break;
|
||||||
|
kept.unshift(rest[i]);
|
||||||
|
used += rest[i].length + 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
const truncNote = `[System] Note: ${rest.length - kept.length} older messages were truncated to fit context limit.`;
|
||||||
|
return [systemText, truncNote, ...kept].filter(Boolean).join("\n\n");
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
/**
|
||||||
|
* Model Registry — Layer 1 of OCP routing.
|
||||||
|
*
|
||||||
|
* Static mapping: model ID → backend ID.
|
||||||
|
* This is infrastructure, not policy. Every model must map to exactly one backend.
|
||||||
|
*
|
||||||
|
* Non-agent callers (Cline, Aider, etc.) only send a model name — this module
|
||||||
|
* is the only thing that tells OCP where to route the request.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export class ModelRegistry {
|
||||||
|
constructor() {
|
||||||
|
/** @type {Map<string, {backendId: string, canonical: string}>} */
|
||||||
|
this._models = new Map();
|
||||||
|
|
||||||
|
/** @type {Map<string, Object>} model id → display metadata */
|
||||||
|
this._metadata = new Map();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Register models from a backend adapter.
|
||||||
|
* Called during startup for each enabled backend.
|
||||||
|
*
|
||||||
|
* @param {import("./backends/base.mjs").BackendAdapter} adapter
|
||||||
|
*/
|
||||||
|
registerBackend(adapter) {
|
||||||
|
for (const modelId of adapter.models) {
|
||||||
|
this._models.set(modelId, {
|
||||||
|
backendId: adapter.id,
|
||||||
|
canonical: modelId,
|
||||||
|
});
|
||||||
|
this._metadata.set(modelId, {
|
||||||
|
backendId: adapter.id,
|
||||||
|
backendName: adapter.displayName,
|
||||||
|
tier: adapter.tier,
|
||||||
|
costType: adapter.costType,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Also register aliases if the adapter supports resolveModel
|
||||||
|
if (typeof adapter.resolveModel === "function") {
|
||||||
|
// We don't auto-register aliases here — they're handled by
|
||||||
|
// supportsModel() on the adapter. The registry only knows canonical IDs.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Look up which backend serves a given model.
|
||||||
|
*
|
||||||
|
* @param {string} model — model ID (could be canonical or alias)
|
||||||
|
* @returns {{backendId: string, canonical: string} | null}
|
||||||
|
*/
|
||||||
|
resolve(model) {
|
||||||
|
// Direct lookup first
|
||||||
|
const direct = this._models.get(model);
|
||||||
|
if (direct) return direct;
|
||||||
|
|
||||||
|
// No alias resolution here — that's the adapter's job.
|
||||||
|
// The router will ask each adapter if it supports the model.
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get all registered models with metadata.
|
||||||
|
* Used by GET /v1/models endpoint.
|
||||||
|
*/
|
||||||
|
listModels() {
|
||||||
|
const result = [];
|
||||||
|
for (const [id, meta] of this._metadata) {
|
||||||
|
result.push({ id, ...meta });
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if a model is known to the registry.
|
||||||
|
*/
|
||||||
|
has(model) {
|
||||||
|
return this._models.has(model);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get the backend ID for a model, or null.
|
||||||
|
*/
|
||||||
|
getBackendId(model) {
|
||||||
|
return this._models.get(model)?.backendId || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get all models for a specific backend.
|
||||||
|
*/
|
||||||
|
getModelsForBackend(backendId) {
|
||||||
|
const models = [];
|
||||||
|
for (const [id, entry] of this._models) {
|
||||||
|
if (entry.backendId === backendId) models.push(id);
|
||||||
|
}
|
||||||
|
return models;
|
||||||
|
}
|
||||||
|
}
|
||||||
+96
-6
@@ -1,11 +1,17 @@
|
|||||||
#!/usr/bin/env node
|
#!/usr/bin/env node
|
||||||
/**
|
/**
|
||||||
* openclaw-claude-proxy v2.5.0 — OpenAI-compatible proxy for Claude CLI
|
* OCP (OpenClaw Control Plane) v4.0-alpha
|
||||||
*
|
*
|
||||||
* Translates OpenAI chat/completions requests into `claude -p` CLI calls,
|
* Cost-control and model-routing layer for AI agents.
|
||||||
* letting you use your Claude Pro/Max subscription as an OpenClaw model provider.
|
* Currently supports Claude CLI backend; architecture supports multiple backends.
|
||||||
*
|
*
|
||||||
* v2.5.0:
|
* v4.0-alpha:
|
||||||
|
* - Internal modular architecture: BackendAdapter, ModelRegistry, AgentRouter,
|
||||||
|
* SessionManager, StatsCollector
|
||||||
|
* - External API unchanged from v2.5.0
|
||||||
|
* - Claude CLI backend extracted to ClaudeCliAdapter
|
||||||
|
*
|
||||||
|
* v2.5.0 (legacy notes):
|
||||||
* - Sliding-window circuit breaker: uses time-windowed failure rate instead of
|
* - Sliding-window circuit breaker: uses time-windowed failure rate instead of
|
||||||
* consecutive-count, preventing multi-agent burst scenarios from tripping the
|
* consecutive-count, preventing multi-agent burst scenarios from tripping the
|
||||||
* breaker too aggressively. Half-open state allows configurable probe requests.
|
* breaker too aggressively. Half-open state allows configurable probe requests.
|
||||||
@@ -47,6 +53,54 @@ import { dirname, join } from "node:path";
|
|||||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||||
const _pkg = JSON.parse(readFileSync(join(__dirname, "package.json"), "utf8"));
|
const _pkg = JSON.parse(readFileSync(join(__dirname, "package.json"), "utf8"));
|
||||||
|
|
||||||
|
// ── v4 Modular Architecture (Step 1: load + initialize, hot path unchanged) ──
|
||||||
|
import { ClaudeCliAdapter } from "./backends/claude-cli.mjs";
|
||||||
|
import { ModelRegistry } from "./model-registry.mjs";
|
||||||
|
import { AgentRouter } from "./agent-router.mjs";
|
||||||
|
import { SessionManager as SessionMgr } from "./session-manager.mjs";
|
||||||
|
import { StatsCollector } from "./stats-collector.mjs";
|
||||||
|
|
||||||
|
// These modules are initialized at startup but NOT yet wired into the request
|
||||||
|
// path. Step 2 will replace the legacy callClaude/callClaudeStreaming with
|
||||||
|
// adapter-based calls. For now, they run in parallel for validation.
|
||||||
|
let _v4Backend = null;
|
||||||
|
let _v4Registry = null;
|
||||||
|
let _v4Router = null;
|
||||||
|
let _v4Sessions = null;
|
||||||
|
let _v4Stats = null;
|
||||||
|
|
||||||
|
async function initV4Modules() {
|
||||||
|
try {
|
||||||
|
_v4Stats = new StatsCollector();
|
||||||
|
_v4Sessions = new SessionMgr({ ttl: SESSION_TTL });
|
||||||
|
_v4Registry = new ModelRegistry();
|
||||||
|
|
||||||
|
_v4Backend = new ClaudeCliAdapter({
|
||||||
|
maxConcurrent: MAX_CONCURRENT,
|
||||||
|
timeout: { overall: TIMEOUT },
|
||||||
|
skipPermissions: SKIP_PERMISSIONS,
|
||||||
|
allowedTools: ALLOWED_TOOLS,
|
||||||
|
systemPrompt: SYSTEM_PROMPT,
|
||||||
|
mcpConfig: MCP_CONFIG,
|
||||||
|
timeoutTiers: MODEL_TIMEOUT_TIERS,
|
||||||
|
});
|
||||||
|
await _v4Backend.initialize();
|
||||||
|
_v4Registry.registerBackend(_v4Backend);
|
||||||
|
|
||||||
|
const backends = new Map([["claude-cli", _v4Backend]]);
|
||||||
|
_v4Router = new AgentRouter({
|
||||||
|
registry: _v4Registry,
|
||||||
|
backends,
|
||||||
|
agentConfig: { default: { preferred: "claude-cli", fallback: null } },
|
||||||
|
defaultBackend: "claude-cli",
|
||||||
|
});
|
||||||
|
|
||||||
|
console.log(`[v4] Modules initialized: backend=${_v4Backend.id}, models=${_v4Backend.models.length}, registry=${_v4Registry.listModels().length} models`);
|
||||||
|
} catch (err) {
|
||||||
|
console.error(`[v4] Module init failed (non-fatal, legacy path active): ${err.message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ── Resolve claude binary ───────────────────────────────────────────────
|
// ── Resolve claude binary ───────────────────────────────────────────────
|
||||||
// Priority: CLAUDE_BIN env > well-known paths > which lookup
|
// Priority: CLAUDE_BIN env > well-known paths > which lookup
|
||||||
// Fail-fast if not found — never start with an unresolvable binary.
|
// Fail-fast if not found — never start with an unresolvable binary.
|
||||||
@@ -1178,7 +1232,36 @@ const server = createServer(async (req, res) => {
|
|||||||
return handleSettings(req, res);
|
return handleSettings(req, res);
|
||||||
}
|
}
|
||||||
|
|
||||||
jsonResponse(res, 404, { error: "Not found. Endpoints: GET /v1/models, POST /v1/chat/completions, GET /health, GET /usage, GET /status, GET /logs, GET|PATCH /settings, GET|DELETE /sessions" });
|
// GET /backends — v4 backend status (new endpoint)
|
||||||
|
if (req.url === "/backends" && req.method === "GET") {
|
||||||
|
if (!_v4Backend) {
|
||||||
|
return jsonResponse(res, 503, { error: "v4 modules not initialized" });
|
||||||
|
}
|
||||||
|
const health = await _v4Backend.healthCheck();
|
||||||
|
return jsonResponse(res, 200, {
|
||||||
|
backends: [{
|
||||||
|
id: _v4Backend.id,
|
||||||
|
displayName: _v4Backend.displayName,
|
||||||
|
tier: _v4Backend.tier,
|
||||||
|
costType: _v4Backend.costType,
|
||||||
|
enabled: _v4Backend.enabled,
|
||||||
|
models: _v4Backend.models,
|
||||||
|
activeProcesses: _v4Backend.activeProcessCount,
|
||||||
|
health,
|
||||||
|
}],
|
||||||
|
routing: _v4Router?.getRoutingTable() || {},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// GET /routing — v4 agent routing table (new endpoint)
|
||||||
|
if (req.url === "/routing" && req.method === "GET") {
|
||||||
|
return jsonResponse(res, 200, {
|
||||||
|
table: _v4Router?.getRoutingTable() || {},
|
||||||
|
registry: _v4Registry?.listModels() || [],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
jsonResponse(res, 404, { error: "Not found. Endpoints: GET /v1/models, POST /v1/chat/completions, GET /health, GET /usage, GET /status, GET /logs, GET|PATCH /settings, GET|DELETE /sessions, GET /backends, GET /routing" });
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
||||||
@@ -1199,6 +1282,10 @@ function gracefulShutdown(signal) {
|
|||||||
clearInterval(sessionCleanupInterval);
|
clearInterval(sessionCleanupInterval);
|
||||||
clearInterval(authCheckInterval);
|
clearInterval(authCheckInterval);
|
||||||
|
|
||||||
|
// 2b. Shutdown v4 modules
|
||||||
|
if (_v4Sessions) _v4Sessions.shutdown();
|
||||||
|
if (_v4Backend) _v4Backend.shutdown().catch(() => {});
|
||||||
|
|
||||||
// 3. Kill all active child processes
|
// 3. Kill all active child processes
|
||||||
for (const proc of activeProcesses) {
|
for (const proc of activeProcesses) {
|
||||||
try { proc.kill("SIGTERM"); } catch {}
|
try { proc.kill("SIGTERM"); } catch {}
|
||||||
@@ -1235,8 +1322,11 @@ process.on("SIGTERM", () => gracefulShutdown("SIGTERM"));
|
|||||||
process.on("SIGINT", () => gracefulShutdown("SIGINT"));
|
process.on("SIGINT", () => gracefulShutdown("SIGINT"));
|
||||||
|
|
||||||
// ── Start ───────────────────────────────────────────────────────────────
|
// ── Start ───────────────────────────────────────────────────────────────
|
||||||
|
// Initialize v4 modules before listening
|
||||||
|
initV4Modules().catch(err => console.error(`[v4] init error: ${err.message}`));
|
||||||
|
|
||||||
server.listen(PORT, "127.0.0.1", () => {
|
server.listen(PORT, "127.0.0.1", () => {
|
||||||
console.log(`openclaw-claude-proxy v${VERSION} listening on http://127.0.0.1:${PORT}`);
|
console.log(`OCP (OpenClaw Control Plane) v${VERSION} listening on http://127.0.0.1:${PORT}`);
|
||||||
console.log(`Architecture: on-demand spawning (no pool)`);
|
console.log(`Architecture: on-demand spawning (no pool)`);
|
||||||
console.log(`Models: ${MODELS.map((m) => m.id).join(", ")}`);
|
console.log(`Models: ${MODELS.map((m) => m.id).join(", ")}`);
|
||||||
console.log(`Claude binary: ${CLAUDE}`);
|
console.log(`Claude binary: ${CLAUDE}`);
|
||||||
|
|||||||
@@ -0,0 +1,123 @@
|
|||||||
|
/**
|
||||||
|
* SessionManager — manages conversation sessions across requests.
|
||||||
|
*
|
||||||
|
* Maps caller-provided conversation IDs to backend-specific session state.
|
||||||
|
* For Claude CLI, this means mapping to CLI session UUIDs for --resume.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { randomUUID } from "node:crypto";
|
||||||
|
|
||||||
|
export class SessionManager {
|
||||||
|
/**
|
||||||
|
* @param {Object} options
|
||||||
|
* @param {number} [options.ttl=3600000] — Session TTL in ms (default 1h)
|
||||||
|
*/
|
||||||
|
constructor({ ttl = 3600000 } = {}) {
|
||||||
|
this._ttl = ttl;
|
||||||
|
/** @type {Map<string, SessionEntry>} */
|
||||||
|
this._sessions = new Map();
|
||||||
|
|
||||||
|
// Cleanup expired sessions every 60s
|
||||||
|
this._cleanupInterval = setInterval(() => this._cleanup(), 60000);
|
||||||
|
}
|
||||||
|
|
||||||
|
get ttl() { return this._ttl; }
|
||||||
|
set ttl(val) { this._ttl = val; }
|
||||||
|
get size() { return this._sessions.size; }
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get or create a session for a conversation.
|
||||||
|
*
|
||||||
|
* @param {string|null} conversationId
|
||||||
|
* @param {string} model — model being used
|
||||||
|
* @param {string} [agent] — agent identity
|
||||||
|
* @returns {{session: {uuid: string, resume: boolean}, isNew: boolean, isOneOff: boolean}}
|
||||||
|
*/
|
||||||
|
resolve(conversationId, model, agent = "default") {
|
||||||
|
if (!conversationId) {
|
||||||
|
return { session: null, isNew: false, isOneOff: true };
|
||||||
|
}
|
||||||
|
|
||||||
|
if (this._sessions.has(conversationId)) {
|
||||||
|
const entry = this._sessions.get(conversationId);
|
||||||
|
entry.lastUsed = Date.now();
|
||||||
|
entry.messageCount++;
|
||||||
|
return {
|
||||||
|
session: { uuid: entry.uuid, resume: true },
|
||||||
|
isNew: false,
|
||||||
|
isOneOff: false,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// New session
|
||||||
|
const uuid = randomUUID();
|
||||||
|
this._sessions.set(conversationId, {
|
||||||
|
uuid,
|
||||||
|
model,
|
||||||
|
agent,
|
||||||
|
messageCount: 1,
|
||||||
|
lastUsed: Date.now(),
|
||||||
|
createdAt: Date.now(),
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
session: { uuid, resume: false },
|
||||||
|
isNew: true,
|
||||||
|
isOneOff: false,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Remove a session (e.g. on resume failure).
|
||||||
|
*/
|
||||||
|
delete(conversationId) {
|
||||||
|
this._sessions.delete(conversationId);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Clear all sessions.
|
||||||
|
* @returns {number} Number of sessions cleared
|
||||||
|
*/
|
||||||
|
clear() {
|
||||||
|
const count = this._sessions.size;
|
||||||
|
this._sessions.clear();
|
||||||
|
return count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* List all active sessions (for /sessions endpoint).
|
||||||
|
*/
|
||||||
|
list() {
|
||||||
|
const result = [];
|
||||||
|
for (const [id, s] of this._sessions) {
|
||||||
|
result.push({
|
||||||
|
id,
|
||||||
|
uuid: s.uuid,
|
||||||
|
model: s.model,
|
||||||
|
agent: s.agent,
|
||||||
|
messages: s.messageCount,
|
||||||
|
lastUsed: new Date(s.lastUsed).toISOString(),
|
||||||
|
idleMs: Date.now() - s.lastUsed,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Shutdown — clear interval.
|
||||||
|
*/
|
||||||
|
shutdown() {
|
||||||
|
clearInterval(this._cleanupInterval);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Internal ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
_cleanup() {
|
||||||
|
const now = Date.now();
|
||||||
|
for (const [id, s] of this._sessions) {
|
||||||
|
if (now - s.lastUsed > this._ttl) {
|
||||||
|
this._sessions.delete(id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,186 @@
|
|||||||
|
/**
|
||||||
|
* StatsCollector — tracks request metrics per-model, per-backend, per-agent.
|
||||||
|
*
|
||||||
|
* Replaces the flat stats + modelStats from server.mjs with structured,
|
||||||
|
* multi-dimensional tracking. Backward compatible: still exposes the same
|
||||||
|
* data for /health, /usage, /status endpoints.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export class StatsCollector {
|
||||||
|
constructor() {
|
||||||
|
// Global counters (backward compatible)
|
||||||
|
this.totalRequests = 0;
|
||||||
|
this.activeRequests = 0;
|
||||||
|
this.errors = 0;
|
||||||
|
this.timeouts = 0;
|
||||||
|
this.sessionHits = 0;
|
||||||
|
this.sessionMisses = 0;
|
||||||
|
this.oneOffRequests = 0;
|
||||||
|
|
||||||
|
// Recent errors ring buffer
|
||||||
|
this._recentErrors = [];
|
||||||
|
this._maxErrors = 20;
|
||||||
|
|
||||||
|
/** @type {Map<string, ModelStats>} cliModel → stats */
|
||||||
|
this._modelStats = new Map();
|
||||||
|
|
||||||
|
/** @type {Map<string, ModelStats>} backendId → stats */
|
||||||
|
this._backendStats = new Map();
|
||||||
|
|
||||||
|
/** @type {Map<string, ModelStats>} agentName → stats */
|
||||||
|
this._agentStats = new Map();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Record a new request starting.
|
||||||
|
*/
|
||||||
|
recordRequest(model, backend, agent, promptChars) {
|
||||||
|
this.totalRequests++;
|
||||||
|
this.activeRequests++;
|
||||||
|
this._getOrCreate(this._modelStats, model).recordRequest(promptChars);
|
||||||
|
this._getOrCreate(this._backendStats, backend).recordRequest(promptChars);
|
||||||
|
this._getOrCreate(this._agentStats, agent).recordRequest(promptChars);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Record a successful completion.
|
||||||
|
*/
|
||||||
|
recordSuccess(model, backend, agent, elapsedMs, cost = null) {
|
||||||
|
this.activeRequests = Math.max(0, this.activeRequests - 1);
|
||||||
|
this._getOrCreate(this._modelStats, model).recordSuccess(elapsedMs, cost);
|
||||||
|
this._getOrCreate(this._backendStats, backend).recordSuccess(elapsedMs, cost);
|
||||||
|
this._getOrCreate(this._agentStats, agent).recordSuccess(elapsedMs, cost);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Record an error.
|
||||||
|
*/
|
||||||
|
recordError(model, backend, agent, isTimeout, message = "") {
|
||||||
|
this.activeRequests = Math.max(0, this.activeRequests - 1);
|
||||||
|
this.errors++;
|
||||||
|
if (isTimeout) this.timeouts++;
|
||||||
|
|
||||||
|
this._getOrCreate(this._modelStats, model).recordError(isTimeout);
|
||||||
|
this._getOrCreate(this._backendStats, backend).recordError(isTimeout);
|
||||||
|
this._getOrCreate(this._agentStats, agent).recordError(isTimeout);
|
||||||
|
|
||||||
|
this._recentErrors.push({
|
||||||
|
time: new Date().toISOString(),
|
||||||
|
message: String(message).slice(0, 200),
|
||||||
|
model, backend, agent,
|
||||||
|
});
|
||||||
|
if (this._recentErrors.length > this._maxErrors) this._recentErrors.shift();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Record session hit/miss/one-off.
|
||||||
|
*/
|
||||||
|
recordSessionHit() { this.sessionHits++; }
|
||||||
|
recordSessionMiss() { this.sessionMisses++; }
|
||||||
|
recordOneOff() { this.oneOffRequests++; }
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get per-model stats snapshot (backward compatible with v2).
|
||||||
|
*/
|
||||||
|
getModelStatsSnapshot() {
|
||||||
|
return this._snapshot(this._modelStats);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get per-backend stats snapshot.
|
||||||
|
*/
|
||||||
|
getBackendStatsSnapshot() {
|
||||||
|
return this._snapshot(this._backendStats);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get per-agent stats snapshot.
|
||||||
|
*/
|
||||||
|
getAgentStatsSnapshot() {
|
||||||
|
return this._snapshot(this._agentStats);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get recent errors.
|
||||||
|
*/
|
||||||
|
getRecentErrors(n = 5) {
|
||||||
|
return this._recentErrors.slice(-n);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get global stats (backward compatible).
|
||||||
|
*/
|
||||||
|
getGlobalStats() {
|
||||||
|
return {
|
||||||
|
totalRequests: this.totalRequests,
|
||||||
|
activeRequests: this.activeRequests,
|
||||||
|
errors: this.errors,
|
||||||
|
timeouts: this.timeouts,
|
||||||
|
sessionHits: this.sessionHits,
|
||||||
|
sessionMisses: this.sessionMisses,
|
||||||
|
oneOffRequests: this.oneOffRequests,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Internal ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
_getOrCreate(map, key) {
|
||||||
|
if (!map.has(key)) map.set(key, new ModelStats());
|
||||||
|
return map.get(key);
|
||||||
|
}
|
||||||
|
|
||||||
|
_snapshot(map) {
|
||||||
|
const result = {};
|
||||||
|
for (const [key, stats] of map) {
|
||||||
|
result[key] = stats.toJSON();
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Per-entity stats tracker */
|
||||||
|
class ModelStats {
|
||||||
|
constructor() {
|
||||||
|
this.requests = 0;
|
||||||
|
this.successes = 0;
|
||||||
|
this.errors = 0;
|
||||||
|
this.timeouts = 0;
|
||||||
|
this.totalElapsed = 0;
|
||||||
|
this.maxElapsed = 0;
|
||||||
|
this.totalPromptChars = 0;
|
||||||
|
this.maxPromptChars = 0;
|
||||||
|
this.totalCost = 0; // accumulated estimated/actual cost
|
||||||
|
}
|
||||||
|
|
||||||
|
recordRequest(promptChars) {
|
||||||
|
this.requests++;
|
||||||
|
this.totalPromptChars += promptChars;
|
||||||
|
if (promptChars > this.maxPromptChars) this.maxPromptChars = promptChars;
|
||||||
|
}
|
||||||
|
|
||||||
|
recordSuccess(elapsedMs, cost = null) {
|
||||||
|
this.successes++;
|
||||||
|
this.totalElapsed += elapsedMs;
|
||||||
|
if (elapsedMs > this.maxElapsed) this.maxElapsed = elapsedMs;
|
||||||
|
if (cost?.value) this.totalCost += cost.value;
|
||||||
|
}
|
||||||
|
|
||||||
|
recordError(isTimeout) {
|
||||||
|
this.errors++;
|
||||||
|
if (isTimeout) this.timeouts++;
|
||||||
|
}
|
||||||
|
|
||||||
|
toJSON() {
|
||||||
|
return {
|
||||||
|
requests: this.requests,
|
||||||
|
successes: this.successes,
|
||||||
|
errors: this.errors,
|
||||||
|
timeouts: this.timeouts,
|
||||||
|
avgElapsed: this.successes > 0 ? Math.round(this.totalElapsed / this.successes) : 0,
|
||||||
|
maxElapsed: this.maxElapsed,
|
||||||
|
avgPromptChars: this.requests > 0 ? Math.round(this.totalPromptChars / this.requests) : 0,
|
||||||
|
maxPromptChars: this.maxPromptChars,
|
||||||
|
totalCost: Math.round(this.totalCost * 10000) / 10000, // 4 decimal places
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
/**
|
||||||
|
* UnifiedChunk — the internal streaming protocol for OCP.
|
||||||
|
*
|
||||||
|
* All backend adapters MUST convert their native output into this format.
|
||||||
|
* server.mjs only handles UnifiedChunks → OpenAI SSE conversion.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @typedef {Object} UnifiedChunk
|
||||||
|
* @property {"delta"|"done"|"error"} type
|
||||||
|
* @property {string} [content] — text content (for "delta")
|
||||||
|
* @property {string} model — canonical model id
|
||||||
|
* @property {string} backend — backend id (e.g. "claude-cli")
|
||||||
|
* @property {Object} [usage] — token usage (for "done")
|
||||||
|
* @property {number} [usage.promptTokens]
|
||||||
|
* @property {number} [usage.completionTokens]
|
||||||
|
* @property {Object} [cost] — cost info (for "done")
|
||||||
|
* @property {number} [cost.value]
|
||||||
|
* @property {"actual"|"estimated"|"free"} [cost.type]
|
||||||
|
* @property {string} [error] — error message (for "error")
|
||||||
|
*/
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Create a delta chunk (streaming content)
|
||||||
|
*/
|
||||||
|
export function delta(content, model, backend) {
|
||||||
|
return { type: "delta", content, model, backend };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Create a done chunk (stream finished)
|
||||||
|
*/
|
||||||
|
export function done(model, backend, usage = null, cost = null) {
|
||||||
|
return { type: "done", model, backend, ...(usage && { usage }), ...(cost && { cost }) };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Create an error chunk
|
||||||
|
*/
|
||||||
|
export function error(message, model, backend) {
|
||||||
|
return { type: "error", error: message, model, backend };
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user