mirror of
https://github.com/dtzp555-max/olp.git
synced 2026-07-21 21:15:10 +00:00
* fix+release(v0.4.1): D74 post-Phase-4 hotfix batch — maintainer-review findings Patch release fixing 5 issues caught by maintainer post-v0.4.0 independent review (main / v0.4.0 / commitee4d945). All five are real runtime bugs the per-D-day fresh-context opus reviewers missed because they checked spec text instead of runtime contracts (default auth.allow_anonymous: false, real /health payload shape, real /cache/stats payload shape, real /v0/management/dashboard-data payload shape). Phase 4 process learning: every implementation D-day MUST include at least one test that boots the server with default production config and exercises the new feature end-to-end. D74 Suite 36 pins the wire- contract shapes so a future D-day refactor can't silently re-break the CLI / plugin / docs. ## P1-1 — olp doctor false-negative on auth-required /health lib/doctor.mjs: buildBuiltinChecks() accepts opts.authHeaders and passes to httpGet for both server.running + server.version probes. The server.running check now distinguishes 401/403 ("server up, bearer token missing or invalid — set OLP_API_KEY") from "server unreachable" so the kind discriminator routes to a clean fix-auth path instead of fix_server when operator just forgot to export the env var. bin/olp.mjs cmdDoctor: threads authHeaders() through to runDoctor. ## P1-2 — olp-connect token validation + shell-quoting bin/olp-connect: validate_olp_token() enforces ^olp_[A-Za-z0-9_-]{43}$ (per ADR 0007 § 3 token format) at THREE input sites: --key arg, /health.anonymousKey server-advertised consumption, interactive prompt fallback. shell_quote() POSIX-single-quote-wraps with embedded-quote escape per: foo'bar → 'foo'\''bar' Applied to all rc-file writes + dry-run output. systemd environment.d/olp.conf write additionally rejects embedded newlines. Hostile or malformed keys can no longer persist as shell startup injection. ## P2-3 — olp usage + olp cache human formatter wire-contract fix bin/olp.mjs cmdUsage previously read body.usage_24h.requests / body.providers / body.top_fallback_chains — none of which exist in the real server payload (server.mjs:2027 + lib/audit-query.mjs). Users saw "requests: ?" + missing per-provider quota + missing top-chains. Now reads body.window_24h.request_count / body.cache_hit_24h.hit_rate / body.quota / body.top_fallback_chains_24h. bin/olp.mjs cmdCache previously read body.entries / body.bytes / body.maxBytes (OCP-era field names). Real CacheStore.stats() returns {hits, misses, size, inflightCount}. Now reads body.size / body.inflightCount + computes hit rate from hits/(hits+misses). ## P2-4 — olp-plugin/ fmtHealth iterates providers.status olp-plugin/index.js: previously walked Object.entries(body.providers) which surfaced `enabled` / `available` / `status` as pseudo-providers (chat showed "🟢 status" instead of "🟢 anthropic"). Now extracts the real provider map from body.providers.status, renders enabled/available counts in a header line, lists per-provider names with activeSpawns when present. Falls back to flat body.providers.* for older OCP shape (backwards compat). ## P3-5 — stale v0.3.0-era doc strings updated README.md: header status line + Implementation Status § now reflect v0.4.0 shipped + Phase 5 open. Known-limitations Phase 3 line moved out of "pending v0.3.0" state; new Phase 4 line added with full deliverable list. server.mjs: startup banner no longer hardcodes "Phase 1 in progress" (now just lists version + provider count). Banner derives state from VERSION so future Phase boundaries don't need touch-ups here. ## Test count 696 (v0.4.0) → 704 (v0.4.1). +8 D74 regression tests in Suite 36: - 36a: runDoctor accepts authHeaders + threads to checks - 36b: server.running distinguishes 401 (auth) from "server down" - 36c: olp-connect rejects malformed --key (validator fires before rc write) - 36d: olp-connect accepts properly-formed olp_ token - 36e: CacheStore.stats() shape pin + cmdCache source pin - 36f: dashboard-data payload shape pin + cmdUsage source pin - 36g: olp-plugin fmtHealth iterates providers.status not providers.* - 36h: server.mjs banner doesn't hardcode stale phase ## Authority - Maintainer independent review of main / v0.4.0 / commitee4d945(2026-05-26 session — 5 findings P1×2 + P2×2 + P3×1) - Iron Rule 第二律 (evidence over "should work") — runtime smoke against default production config now mandatory per Suite 36 pattern - CLAUDE.md release_kit.phase_rolling_mode cross-Phase discipline ("hotfix to a shipped Phase N deliverable → bump patch, tag, release before next push") - ADR 0007 § 3 (token format ^olp_[A-Za-z0-9_-]{43}$) — D74 P1-2 validator authority Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix: Suite 36 paths use import.meta.dirname for CI portability (was hardcoded /Users/taodeng/olp/) --------- Co-authored-by: dtzp555 <dtzp555@gmail.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
14987 lines
637 KiB
JavaScript
14987 lines
637 KiB
JavaScript
/**
|
|
* test-features.mjs — OLP D5 test suite (extends D4)
|
|
*
|
|
* Uses Node's built-in node:test runner. No external dependencies.
|
|
* Run: node test-features.mjs (or: npm test)
|
|
*
|
|
* Authority: ADR 0002 (provider contract), ADR 0003 (IR v1.0), ADR 0005 (cache layer)
|
|
* D4 adds: Anthropic plugin conformance, IR translation, mock-spawn behaviour.
|
|
* D5 adds: Suite 9 (cache layer unit + HTTP integration), Suite 10 (Anthropic E2E gated)
|
|
*/
|
|
|
|
import { describe, it, before, after, beforeEach } from 'node:test';
|
|
import assert from 'node:assert/strict';
|
|
import { request as httpRequest } from 'node:http';
|
|
import { EventEmitter } from 'node:events';
|
|
import { homedir, tmpdir as _tmpdirForSetup } from 'node:os';
|
|
import { mkdtempSync as _mkdtempSyncForSetup } from 'node:fs';
|
|
import { join as _pathJoinForSetup } from 'node:path';
|
|
import { computeCacheKey, extractCacheControlMarkers, hasCacheControl } from './lib/cache/keys.mjs';
|
|
import { CacheStore } from './lib/cache/store.mjs';
|
|
|
|
// ── Phase 2 / D45 test-mode setup ─────────────────────────────────────────
|
|
// Two adjustments keep pre-D45 tests working alongside the new auth gate:
|
|
// 1. process.env.OLP_HOME → tmpdir so audit ndjson / manifest writes
|
|
// triggered by handleChatCompletions / handleModels do not pollute
|
|
// the user's real ~/.olp/. lib/keys.mjs + lib/audit.mjs resolve the
|
|
// env per-call so this takes effect immediately.
|
|
// 2. server.mjs __setAuthConfig({ allow_anonymous: true }) so existing
|
|
// HTTP integration tests (Suite 18 etc.) that hit /v1/chat/completions
|
|
// and /v1/models without an Authorization header continue to pass
|
|
// via the anonymous identity. Suite 20 (D45 auth tests) explicitly
|
|
// overrides per-case to exercise allow_anonymous: false / valid key /
|
|
// revoked / env-owner / providers_enabled paths.
|
|
// (ESM imports are hoisted, so all module side effects — including
|
|
// server.mjs's startup loadAuthConfigSync() — complete before this body
|
|
// code runs. Setting OLP_HOME + __setAuthConfig here applies to all
|
|
// suites below.)
|
|
const _GLOBAL_TEST_OLP_HOME = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-home-'));
|
|
process.env.OLP_HOME = _GLOBAL_TEST_OLP_HOME;
|
|
// Clean up the global test tmpdir on process exit so successive npm test runs
|
|
// don't accumulate /var/folders/.../olp-test-home-* directories. process.on
|
|
// ('exit') fires synchronously after node:test reports all results.
|
|
process.on('exit', () => {
|
|
try {
|
|
// rmSync is imported lower in the file (Suite 19 imports it from 'node:fs').
|
|
// ESM hoists all imports to top-of-module so the binding is available here.
|
|
rmSync(_GLOBAL_TEST_OLP_HOME, { recursive: true, force: true });
|
|
} catch {
|
|
// best-effort; do not throw at exit
|
|
}
|
|
});
|
|
// __setAuthConfig is imported below from './server.mjs'; deferring the call
|
|
// to a later block (after server.mjs's import-time loadAuthConfigSync runs)
|
|
// is necessary because ESM hoists imports before this body code. See the
|
|
// "Phase 2 / D45 server-side default override" block below the imports.
|
|
|
|
// ── Modules under test ────────────────────────────────────────────────────
|
|
|
|
import { validateIRRequest, validateIRMessage, VALID_ROLES, IR_VERSION } from './lib/ir/types.mjs';
|
|
import { openAIToIR, BadRequestError } from './lib/ir/openai-to-ir.mjs';
|
|
import {
|
|
irChunkToOpenAISSE,
|
|
irResponseToOpenAINonStream,
|
|
generateRequestId,
|
|
SSE_DONE,
|
|
} from './lib/ir/ir-to-openai.mjs';
|
|
import { validateProvider, ProviderError, withTimeout } from './lib/providers/base.mjs';
|
|
import {
|
|
loadProviders,
|
|
getProviderForModel,
|
|
getProviderByName,
|
|
listAllProviderNames,
|
|
getAliasMap,
|
|
getModelCreated,
|
|
REGISTRY_BOOTSTRAP_CREATED,
|
|
tryAcquireSpawn,
|
|
releaseSpawn,
|
|
getActiveSpawnCount,
|
|
__resetSpawnCounters,
|
|
DEFAULT_MAX_CONCURRENT_SPAWNS,
|
|
} from './lib/providers/index.mjs';
|
|
import anthropic, {
|
|
irToAnthropic,
|
|
anthropicChunkToIR,
|
|
anthropicStopToIR,
|
|
readAuthArtifact,
|
|
estimateCost as anthropicEstimateCost,
|
|
quotaStatus as anthropicQuotaStatus,
|
|
healthCheck as anthropicHealthCheck,
|
|
__setSpawnImpl,
|
|
__resetSpawnImpl,
|
|
} from './lib/providers/anthropic.mjs';
|
|
import codex, {
|
|
irToCodex,
|
|
codexChunkToIR,
|
|
readAuthArtifact as codexReadAuthArtifact,
|
|
estimateCost as codexEstimateCost,
|
|
quotaStatus as codexQuotaStatus,
|
|
healthCheck as codexHealthCheck,
|
|
__setSpawnImpl as codexSetSpawnImpl,
|
|
__resetSpawnImpl as codexResetSpawnImpl,
|
|
} from './lib/providers/codex.mjs';
|
|
import mistral, {
|
|
irToMistral,
|
|
mistralChunkToIR,
|
|
readAuthArtifact as mistralReadAuthArtifact,
|
|
estimateCost as mistralEstimateCost,
|
|
quotaStatus as mistralQuotaStatus,
|
|
healthCheck as mistralHealthCheck,
|
|
__setSpawnImpl as mistralSetSpawnImpl,
|
|
__resetSpawnImpl as mistralResetSpawnImpl,
|
|
} from './lib/providers/mistral.mjs';
|
|
import modelsRegistry from './models-registry.json' with { type: 'json' };
|
|
|
|
// ── Helpers ───────────────────────────────────────────────────────────────
|
|
|
|
/** Minimal valid IR request for use in tests */
|
|
function makeIR(overrides = {}) {
|
|
return {
|
|
irVersion: IR_VERSION,
|
|
model: 'test-model',
|
|
stream: false,
|
|
messages: [{ role: 'user', content: 'Hello' }],
|
|
...overrides,
|
|
};
|
|
}
|
|
|
|
/** Minimal valid provider stub that satisfies the v1.0 contract (including D4 contractVersion) */
|
|
function makeProvider(overrides = {}) {
|
|
return {
|
|
name: 'stub',
|
|
displayName: 'Stub Provider',
|
|
contractVersion: '1.0',
|
|
models: ['stub-model-v1'],
|
|
auth: { type: 'none', storage: 'none', path: '', refresh: null },
|
|
spawn: async function* () { yield { type: 'stop', finish_reason: 'stop' }; },
|
|
estimateCost: () => null,
|
|
quotaStatus: async () => null,
|
|
healthCheck: async () => ({ ok: true, latencyMs: 0 }),
|
|
hints: { requiresTTY: false, concurrentSpawnSafe: true, maxConcurrent: 4 },
|
|
...overrides,
|
|
};
|
|
}
|
|
|
|
// ── HTTP helper ───────────────────────────────────────────────────────────
|
|
|
|
/**
|
|
* Makes an HTTP request to the test server.
|
|
* @param {{ port, method, path, headers?, body? }} opts
|
|
* @returns {Promise<{ status: number, headers: object, body: string }>}
|
|
*/
|
|
function fetch(opts) {
|
|
return new Promise((resolve, reject) => {
|
|
const bodyStr = opts.body ? JSON.stringify(opts.body) : undefined;
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port: opts.port,
|
|
method: opts.method ?? 'GET',
|
|
path: opts.path,
|
|
headers: {
|
|
...(bodyStr && { 'Content-Type': 'application/json', 'Content-Length': Buffer.byteLength(bodyStr) }),
|
|
...(opts.headers ?? {}),
|
|
},
|
|
}, res => {
|
|
let data = '';
|
|
res.on('data', c => { data += c; });
|
|
res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: data }));
|
|
});
|
|
req.on('error', reject);
|
|
if (bodyStr) req.write(bodyStr);
|
|
req.end();
|
|
});
|
|
}
|
|
|
|
// ── Suite 1: IR validation ────────────────────────────────────────────────
|
|
|
|
describe('IR validation — validateIRRequest', () => {
|
|
it('accepts a minimal valid IR request', () => {
|
|
const r = validateIRRequest(makeIR());
|
|
assert.equal(r.valid, true);
|
|
assert.deepEqual(r.errors, []);
|
|
});
|
|
|
|
it('accepts an IR request with all optional fields', () => {
|
|
const r = validateIRRequest(makeIR({
|
|
max_tokens: 256,
|
|
temperature: 0.7,
|
|
top_p: 0.9,
|
|
stop: ['\n'],
|
|
tools: [],
|
|
tool_choice: 'auto',
|
|
response_format: { type: 'text' },
|
|
}));
|
|
assert.equal(r.valid, true);
|
|
});
|
|
|
|
it('rejects when messages is missing', () => {
|
|
const ir = makeIR();
|
|
delete ir.messages;
|
|
const r = validateIRRequest(ir);
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('messages')));
|
|
});
|
|
|
|
it('rejects when messages is empty', () => {
|
|
const r = validateIRRequest(makeIR({ messages: [] }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('messages must not be empty')));
|
|
});
|
|
|
|
it('rejects when model is missing', () => {
|
|
const ir = makeIR();
|
|
delete ir.model;
|
|
const r = validateIRRequest(ir);
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('model')));
|
|
});
|
|
|
|
it('rejects when stream is not boolean', () => {
|
|
const r = validateIRRequest(makeIR({ stream: 'true' }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('stream')));
|
|
});
|
|
|
|
it('rejects temperature out of range', () => {
|
|
const r = validateIRRequest(makeIR({ temperature: 3 }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('temperature')));
|
|
});
|
|
|
|
it('rejects top_p out of range', () => {
|
|
const r = validateIRRequest(makeIR({ top_p: -0.1 }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('top_p')));
|
|
});
|
|
|
|
it('rejects non-object input', () => {
|
|
const r = validateIRRequest(null);
|
|
assert.equal(r.valid, false);
|
|
});
|
|
});
|
|
|
|
describe('IR validation — validateIRMessage', () => {
|
|
for (const role of VALID_ROLES) {
|
|
it(`accepts role="${role}"`, () => {
|
|
const r = validateIRMessage({ role, content: 'test' });
|
|
assert.equal(r.valid, true);
|
|
});
|
|
}
|
|
|
|
it('rejects invalid role', () => {
|
|
const r = validateIRMessage({ role: 'admin', content: 'x' });
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('role')));
|
|
});
|
|
|
|
it('rejects missing content', () => {
|
|
const r = validateIRMessage({ role: 'user' });
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('content')));
|
|
});
|
|
|
|
it('accepts array content (multi-part)', () => {
|
|
const r = validateIRMessage({ role: 'user', content: [{ type: 'text', text: 'hi' }] });
|
|
assert.equal(r.valid, true);
|
|
});
|
|
|
|
it('rejects non-object input', () => {
|
|
const r = validateIRMessage('not an object');
|
|
assert.equal(r.valid, false);
|
|
});
|
|
});
|
|
|
|
// ── Suite 2: openAIToIR translation ──────────────────────────────────────
|
|
|
|
describe('openAIToIR translation', () => {
|
|
it('translates a minimal request', () => {
|
|
const ir = openAIToIR({ model: 'gpt-4o', messages: [{ role: 'user', content: 'Hi' }] });
|
|
assert.equal(ir.irVersion, IR_VERSION);
|
|
assert.equal(ir.model, 'gpt-4o');
|
|
assert.equal(ir.stream, false);
|
|
assert.equal(ir.messages.length, 1);
|
|
assert.equal(ir.messages[0].role, 'user');
|
|
assert.equal(ir.messages[0].content, 'Hi');
|
|
});
|
|
|
|
it('defaults stream to false when absent', () => {
|
|
const ir = openAIToIR({ model: 'm', messages: [{ role: 'user', content: 'x' }] });
|
|
assert.equal(ir.stream, false);
|
|
});
|
|
|
|
it('passes stream=true through', () => {
|
|
const ir = openAIToIR({ model: 'm', messages: [{ role: 'user', content: 'x' }], stream: true });
|
|
assert.equal(ir.stream, true);
|
|
});
|
|
|
|
it('translates multi-turn with system message', () => {
|
|
const ir = openAIToIR({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [
|
|
{ role: 'system', content: 'You are helpful.' },
|
|
{ role: 'user', content: 'What is 2+2?' },
|
|
{ role: 'assistant', content: '4' },
|
|
{ role: 'user', content: 'Thanks!' },
|
|
],
|
|
});
|
|
assert.equal(ir.messages.length, 4);
|
|
assert.equal(ir.messages[0].role, 'system');
|
|
assert.equal(ir.messages[1].role, 'user');
|
|
assert.equal(ir.messages[2].role, 'assistant');
|
|
});
|
|
|
|
it('maps deprecated role=function to role=tool', () => {
|
|
const ir = openAIToIR({
|
|
model: 'm',
|
|
messages: [{ role: 'function', name: 'my_fn', content: '{"result":1}' }],
|
|
});
|
|
assert.equal(ir.messages[0].role, 'tool');
|
|
assert.equal(ir.messages[0].name, 'my_fn');
|
|
});
|
|
|
|
it('translates request with tools', () => {
|
|
const ir = openAIToIR({
|
|
model: 'm',
|
|
messages: [{ role: 'user', content: 'Search for X' }],
|
|
tools: [{
|
|
type: 'function',
|
|
function: { name: 'search', description: 'Web search', parameters: { type: 'object', properties: {} } },
|
|
}],
|
|
tool_choice: 'auto',
|
|
});
|
|
assert.equal(ir.tools.length, 1);
|
|
assert.equal(ir.tools[0].function.name, 'search');
|
|
assert.equal(ir.tool_choice, 'auto');
|
|
});
|
|
|
|
it('translates request with response_format', () => {
|
|
const ir = openAIToIR({
|
|
model: 'm',
|
|
messages: [{ role: 'user', content: 'Give JSON' }],
|
|
response_format: { type: 'json_object' },
|
|
});
|
|
assert.deepEqual(ir.response_format, { type: 'json_object' });
|
|
});
|
|
|
|
it('translates optional numeric fields', () => {
|
|
const ir = openAIToIR({
|
|
model: 'm',
|
|
messages: [{ role: 'user', content: 'x' }],
|
|
max_tokens: 100,
|
|
temperature: 0.5,
|
|
top_p: 0.95,
|
|
});
|
|
assert.equal(ir.max_tokens, 100);
|
|
assert.equal(ir.temperature, 0.5);
|
|
assert.equal(ir.top_p, 0.95);
|
|
});
|
|
|
|
it('throws BadRequestError when model is missing', () => {
|
|
assert.throws(
|
|
() => openAIToIR({ messages: [{ role: 'user', content: 'x' }] }),
|
|
BadRequestError,
|
|
);
|
|
});
|
|
|
|
it('throws BadRequestError when messages is empty', () => {
|
|
assert.throws(
|
|
() => openAIToIR({ model: 'm', messages: [] }),
|
|
BadRequestError,
|
|
);
|
|
});
|
|
|
|
it('throws BadRequestError when body is not an object', () => {
|
|
assert.throws(() => openAIToIR(null), BadRequestError);
|
|
});
|
|
});
|
|
|
|
// ── Suite 3: irChunkToOpenAISSE format ────────────────────────────────────
|
|
|
|
describe('irChunkToOpenAISSE format', () => {
|
|
const ID = 'chatcmpl-test123';
|
|
const MODEL = 'test-model';
|
|
|
|
it('generates request IDs with chatcmpl- prefix', () => {
|
|
const id = generateRequestId();
|
|
assert.ok(id.startsWith('chatcmpl-'));
|
|
assert.ok(id.length > 12);
|
|
});
|
|
|
|
it('formats a delta chunk as SSE event', () => {
|
|
const sse = irChunkToOpenAISSE({ type: 'delta', role: 'assistant', content: 'Hello' }, ID, MODEL);
|
|
assert.ok(sse.startsWith('data: '));
|
|
assert.ok(sse.endsWith('\n\n'));
|
|
const payload = JSON.parse(sse.slice(6).trim());
|
|
assert.equal(payload.object, 'chat.completion.chunk');
|
|
assert.equal(payload.id, ID);
|
|
assert.equal(payload.model, MODEL);
|
|
assert.equal(payload.choices[0].delta.content, 'Hello');
|
|
assert.equal(payload.choices[0].delta.role, 'assistant');
|
|
assert.equal(payload.choices[0].finish_reason, null);
|
|
});
|
|
|
|
it('formats a stop chunk with finish_reason', () => {
|
|
const sse = irChunkToOpenAISSE({ type: 'stop', finish_reason: 'stop' }, ID, MODEL);
|
|
const payload = JSON.parse(sse.slice(6).trim());
|
|
assert.equal(payload.choices[0].finish_reason, 'stop');
|
|
assert.deepEqual(payload.choices[0].delta, {});
|
|
});
|
|
|
|
it('does not invent a top-level error field on error chunks (ALIGNMENT.md Rule 2(b))', () => {
|
|
// ALIGNMENT.md Rule 2 (b): OLP must not introduce OpenAI-spec fields that
|
|
// OpenAI's /v1/chat/completions specification does not document.
|
|
// OpenAI chat.completion.chunk objects have no top-level `error` field.
|
|
// Provider errors surface via HTTP 4xx/5xx, not as in-band SSE fields.
|
|
// Error chunks should not reach the translator in normal operation —
|
|
// server.mjs converts them to thrown ProviderError before translation.
|
|
// If one somehow does reach here, no `error` field must be invented.
|
|
const sse = irChunkToOpenAISSE({ type: 'error', error: 'spawn failed' }, ID, MODEL);
|
|
const payload = JSON.parse(sse.slice(6).trim());
|
|
assert.equal(payload.error, undefined, 'translator must not invent top-level error field');
|
|
assert.equal(payload.object, 'chat.completion.chunk');
|
|
assert.ok(['stop', 'length', 'tool_calls', 'content_filter', 'function_call', null].includes(payload.choices[0].finish_reason));
|
|
});
|
|
|
|
it('SSE_DONE is the [DONE] terminator', () => {
|
|
assert.equal(SSE_DONE, 'data: [DONE]\n\n');
|
|
});
|
|
|
|
it('irResponseToOpenAINonStream assembles a complete response', () => {
|
|
const chunks = [
|
|
{ type: 'delta', role: 'assistant', content: 'Hello' },
|
|
{ type: 'delta', content: ' world' },
|
|
{ type: 'stop', finish_reason: 'stop', usage: { prompt_tokens: 5, completion_tokens: 2, total_tokens: 7 } },
|
|
];
|
|
const resp = irResponseToOpenAINonStream(chunks, ID, MODEL);
|
|
assert.equal(resp.object, 'chat.completion');
|
|
assert.equal(resp.id, ID);
|
|
assert.equal(resp.model, MODEL);
|
|
assert.equal(resp.choices[0].message.content, 'Hello world');
|
|
assert.equal(resp.choices[0].message.role, 'assistant');
|
|
assert.equal(resp.choices[0].finish_reason, 'stop');
|
|
assert.equal(resp.usage.total_tokens, 7);
|
|
});
|
|
|
|
it('normalizeFinishReason: non-spec streaming finish_reason is mapped to stop', () => {
|
|
for (const bad of ['timeout', 'overloaded', 'cancelled']) {
|
|
const sse = irChunkToOpenAISSE({ type: 'stop', finish_reason: bad }, ID, MODEL);
|
|
const payload = JSON.parse(sse.slice(6).trim());
|
|
assert.equal(payload.choices[0].finish_reason, 'stop',
|
|
`non-spec value '${bad}' must be normalized to 'stop'`);
|
|
}
|
|
});
|
|
|
|
it('normalizeFinishReason: spec-enum streaming finish_reason values are preserved', () => {
|
|
const specValues = ['stop', 'length', 'tool_calls', 'content_filter', 'function_call', null];
|
|
for (const v of specValues) {
|
|
const sse = irChunkToOpenAISSE({ type: 'stop', finish_reason: v }, ID, MODEL);
|
|
const payload = JSON.parse(sse.slice(6).trim());
|
|
assert.equal(payload.choices[0].finish_reason, v,
|
|
`spec-enum value ${JSON.stringify(v)} must be preserved`);
|
|
}
|
|
});
|
|
|
|
it('normalizeFinishReason: non-spec non-stream finish_reason is mapped to stop', () => {
|
|
for (const bad of ['timeout', 'overloaded', 'cancelled']) {
|
|
const chunks = [
|
|
{ type: 'delta', content: 'hi' },
|
|
{ type: 'stop', finish_reason: bad },
|
|
];
|
|
const resp = irResponseToOpenAINonStream(chunks, ID, MODEL);
|
|
assert.equal(resp.choices[0].finish_reason, 'stop',
|
|
`non-spec value '${bad}' must be normalized to 'stop' in non-stream path`);
|
|
}
|
|
});
|
|
|
|
it('normalizeFinishReason: spec-enum non-stream finish_reason values are preserved', () => {
|
|
// Note: null is intentionally omitted from this list. In the non-stream path,
|
|
// the condition `if (chunk.finish_reason !== undefined)` enters with null,
|
|
// overwrites the default 'stop' to null, and the response then carries
|
|
// finish_reason: null — meaning "still in progress" on a finalized completion,
|
|
// which is semantically odd but spec-valid. The non-stream path's behavior is
|
|
// documented by this omission rather than enforced (no plugin currently emits
|
|
// null on a non-stream stop chunk).
|
|
const specValues = ['stop', 'length', 'tool_calls', 'content_filter', 'function_call'];
|
|
for (const v of specValues) {
|
|
const chunks = [
|
|
{ type: 'delta', content: 'hi' },
|
|
{ type: 'stop', finish_reason: v },
|
|
];
|
|
const resp = irResponseToOpenAINonStream(chunks, ID, MODEL);
|
|
assert.equal(resp.choices[0].finish_reason, v,
|
|
`spec-enum value '${v}' must be preserved in non-stream path`);
|
|
}
|
|
});
|
|
});
|
|
|
|
// ── Suite 4: Provider contract validation ─────────────────────────────────
|
|
|
|
describe('Provider contract validation', () => {
|
|
it('accepts a fully valid provider stub', () => {
|
|
const r = validateProvider(makeProvider());
|
|
assert.equal(r.valid, true);
|
|
assert.deepEqual(r.errors, []);
|
|
});
|
|
|
|
it('rejects provider with missing name', () => {
|
|
const p = makeProvider();
|
|
delete p.name;
|
|
const r = validateProvider(p);
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('name')));
|
|
});
|
|
|
|
it('rejects provider with non-lowercase name', () => {
|
|
const r = validateProvider(makeProvider({ name: 'MyProvider' }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('name')));
|
|
});
|
|
|
|
it('rejects provider with missing displayName', () => {
|
|
const p = makeProvider();
|
|
delete p.displayName;
|
|
const r = validateProvider(p);
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('displayName')));
|
|
});
|
|
|
|
it('rejects provider with missing models array', () => {
|
|
const p = makeProvider();
|
|
delete p.models;
|
|
const r = validateProvider(p);
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('models')));
|
|
});
|
|
|
|
it('rejects provider with missing spawn', () => {
|
|
const p = makeProvider();
|
|
delete p.spawn;
|
|
const r = validateProvider(p);
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('spawn')));
|
|
});
|
|
|
|
it('rejects provider with missing estimateCost', () => {
|
|
const p = makeProvider();
|
|
delete p.estimateCost;
|
|
const r = validateProvider(p);
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('estimateCost')));
|
|
});
|
|
|
|
it('rejects provider with missing healthCheck', () => {
|
|
const p = makeProvider();
|
|
delete p.healthCheck;
|
|
const r = validateProvider(p);
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('healthCheck')));
|
|
});
|
|
|
|
it('rejects provider with missing hints', () => {
|
|
const p = makeProvider();
|
|
delete p.hints;
|
|
const r = validateProvider(p);
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('hints')));
|
|
});
|
|
|
|
it('rejects provider with invalid hints.maxConcurrent', () => {
|
|
const r = validateProvider(makeProvider({ hints: { requiresTTY: false, concurrentSpawnSafe: true, maxConcurrent: -1 } }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('maxConcurrent')));
|
|
});
|
|
|
|
it('validateProvider rejects negative maxSpawnTimeMs', () => {
|
|
const r = validateProvider(makeProvider({ hints: { requiresTTY: false, concurrentSpawnSafe: true, maxConcurrent: 4, maxSpawnTimeMs: -1 } }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('maxSpawnTimeMs')));
|
|
});
|
|
|
|
it('validateProvider rejects zero maxSpawnTimeMs', () => {
|
|
const r = validateProvider(makeProvider({ hints: { requiresTTY: false, concurrentSpawnSafe: true, maxConcurrent: 4, maxSpawnTimeMs: 0 } }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('maxSpawnTimeMs')));
|
|
});
|
|
|
|
it('validateProvider rejects non-integer maxSpawnTimeMs (e.g., 100.5)', () => {
|
|
const r = validateProvider(makeProvider({ hints: { requiresTTY: false, concurrentSpawnSafe: true, maxConcurrent: 4, maxSpawnTimeMs: 100.5 } }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('maxSpawnTimeMs')));
|
|
});
|
|
|
|
it('validateProvider rejects non-number maxSpawnTimeMs (e.g., \'600\')', () => {
|
|
const r = validateProvider(makeProvider({ hints: { requiresTTY: false, concurrentSpawnSafe: true, maxConcurrent: 4, maxSpawnTimeMs: '600' } }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('maxSpawnTimeMs')));
|
|
});
|
|
|
|
it('validateProvider accepts omitted maxSpawnTimeMs (optional field)', () => {
|
|
const r = validateProvider(makeProvider({ hints: { requiresTTY: false, concurrentSpawnSafe: true, maxConcurrent: 4 } }));
|
|
assert.equal(r.valid, true);
|
|
assert.deepEqual(r.errors, []);
|
|
});
|
|
|
|
it('validateProvider accepts positive integer maxSpawnTimeMs (e.g., 60000)', () => {
|
|
const r = validateProvider(makeProvider({ hints: { requiresTTY: false, concurrentSpawnSafe: true, maxConcurrent: 4, maxSpawnTimeMs: 60000 } }));
|
|
assert.equal(r.valid, true);
|
|
assert.deepEqual(r.errors, []);
|
|
});
|
|
|
|
// ── D23: hints.cacheable validation tests (ADR 0002 Amendment 3) ──────
|
|
it('validateProvider accepts hints.cacheable: true (explicit)', () => {
|
|
const r = validateProvider(makeProvider({ hints: { requiresTTY: false, concurrentSpawnSafe: true, maxConcurrent: 4, cacheable: true } }));
|
|
assert.equal(r.valid, true, `Expected valid, got errors: ${r.errors.join(', ')}`);
|
|
assert.deepEqual(r.errors, []);
|
|
});
|
|
|
|
it('validateProvider accepts hints.cacheable: false (explicit opt-out)', () => {
|
|
const r = validateProvider(makeProvider({ hints: { requiresTTY: false, concurrentSpawnSafe: true, maxConcurrent: 4, cacheable: false } }));
|
|
assert.equal(r.valid, true, `Expected valid, got errors: ${r.errors.join(', ')}`);
|
|
assert.deepEqual(r.errors, []);
|
|
});
|
|
|
|
it('validateProvider accepts omitted hints.cacheable (default true)', () => {
|
|
// makeProvider() does not set cacheable — must still be valid.
|
|
const r = validateProvider(makeProvider({ hints: { requiresTTY: false, concurrentSpawnSafe: true, maxConcurrent: 4 } }));
|
|
assert.equal(r.valid, true, `Expected valid, got errors: ${r.errors.join(', ')}`);
|
|
assert.deepEqual(r.errors, []);
|
|
});
|
|
|
|
it('validateProvider rejects hints.cacheable: \'true\' (string, not boolean)', () => {
|
|
const r = validateProvider(makeProvider({ hints: { requiresTTY: false, concurrentSpawnSafe: true, maxConcurrent: 4, cacheable: 'true' } }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('cacheable')), `Expected cacheable error, got: ${r.errors.join(', ')}`);
|
|
});
|
|
|
|
it('validateProvider rejects hints.cacheable: 1 (number, not boolean)', () => {
|
|
const r = validateProvider(makeProvider({ hints: { requiresTTY: false, concurrentSpawnSafe: true, maxConcurrent: 4, cacheable: 1 } }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('cacheable')), `Expected cacheable error, got: ${r.errors.join(', ')}`);
|
|
});
|
|
|
|
it('rejects non-object input', () => {
|
|
const r = validateProvider(null);
|
|
assert.equal(r.valid, false);
|
|
});
|
|
|
|
it('ProviderError carries code field', () => {
|
|
const e = new ProviderError('auth missing', 'AUTH_MISSING');
|
|
assert.equal(e.code, 'AUTH_MISSING');
|
|
assert.equal(e.name, 'ProviderError');
|
|
assert.ok(e instanceof Error);
|
|
});
|
|
|
|
it('withTimeout rejects after deadline', async () => {
|
|
const p = new Promise(r => setTimeout(() => r('late'), 200));
|
|
await assert.rejects(
|
|
() => withTimeout(p, 50, 'SPAWN_FAILED'),
|
|
err => err instanceof ProviderError && err.code === 'SPAWN_FAILED',
|
|
);
|
|
});
|
|
|
|
it('withTimeout resolves when promise is fast', async () => {
|
|
const p = Promise.resolve(42);
|
|
const v = await withTimeout(p, 1000, 'SPAWN_FAILED');
|
|
assert.equal(v, 42);
|
|
});
|
|
});
|
|
|
|
// ── Suite 5: Plugin registry ──────────────────────────────────────────────
|
|
|
|
describe('Plugin registry', () => {
|
|
it('STATIC_REGISTRY has 3 entries (anthropic + openai + mistral candidates) at D8', () => {
|
|
// D4 added anthropic; D6 adds openai; D8 adds mistral. Default config has all enabled:false.
|
|
assert.equal(listAllProviderNames().length, 3);
|
|
});
|
|
|
|
it('loadProviders with empty config → empty Map (anthropic not enabled)', () => {
|
|
const m = loadProviders({});
|
|
assert.equal(m.size, 0);
|
|
});
|
|
|
|
it('loadProviders with no config → empty Map', () => {
|
|
const m = loadProviders();
|
|
assert.equal(m.size, 0);
|
|
});
|
|
|
|
it('listAllProviderNames returns [anthropic, openai, mistral] at D8', () => {
|
|
// D4: ['anthropic']. D6 adds openai. D8 adds mistral.
|
|
assert.deepEqual(listAllProviderNames(), ['anthropic', 'openai', 'mistral']);
|
|
});
|
|
|
|
it('getProviderForModel returns null when no providers loaded', () => {
|
|
const m = loadProviders({});
|
|
const r = getProviderForModel(m, 'gpt-4o');
|
|
assert.equal(r, null);
|
|
});
|
|
|
|
it('getProviderForModel finds provider by exact model string', () => {
|
|
// Build a synthetic loaded map to test the function without touching STATIC_REGISTRY
|
|
const p = makeProvider({ name: 'alpha', models: ['alpha-v1', 'alpha-v2'] });
|
|
const m = new Map([['alpha', p]]);
|
|
const r = getProviderForModel(m, 'alpha-v1');
|
|
assert.ok(r !== null);
|
|
assert.equal(r.name, 'alpha');
|
|
assert.equal(r.canonicalModel, 'alpha-v1', 'D17: canonicalModel must equal modelString on direct lookup');
|
|
});
|
|
|
|
it('getProviderForModel returns null for unknown model', () => {
|
|
const p = makeProvider({ name: 'alpha', models: ['alpha-v1'] });
|
|
const m = new Map([['alpha', p]]);
|
|
assert.equal(getProviderForModel(m, 'beta-v1'), null);
|
|
});
|
|
|
|
it('getProviderByName returns null for empty loaded map', () => {
|
|
const m = new Map();
|
|
assert.equal(getProviderByName(m, 'anthropic'), null);
|
|
});
|
|
|
|
it('getProviderByName returns provider when found', () => {
|
|
const p = makeProvider({ name: 'alpha', models: ['alpha-v1'] });
|
|
const m = new Map([['alpha', p]]);
|
|
assert.ok(getProviderByName(m, 'alpha') !== null);
|
|
assert.equal(getProviderByName(m, 'alpha').name, 'alpha');
|
|
});
|
|
|
|
it('anthropic provider passes contract validation (STATIC_REGISTRY entry)', () => {
|
|
// Even though anthropic is Candidate (not enabled by default), it must
|
|
// pass contract validation at module load — loadProviders() validates all
|
|
// registry entries regardless of enabled flag.
|
|
const { valid, errors } = validateProvider(anthropic);
|
|
assert.equal(valid, true, `Validation errors: ${errors.join('; ')}`);
|
|
});
|
|
});
|
|
|
|
// ── Suite 6: Anthropic plugin (D4) ───────────────────────────────────────
|
|
//
|
|
// All tests in this suite are UNIT tests. No real `claude` binary is invoked.
|
|
// Mock spawn is injected via __setSpawnImpl / __resetSpawnImpl.
|
|
// Tests verify: contract conformance, contractVersion enforcement, registry
|
|
// consistency, IR translation, mock-spawn stream, healthCheck, estimateCost.
|
|
|
|
/**
|
|
* Creates a fake spawn that emits canned stdout chunks then exits cleanly.
|
|
* Returns a fake ChildProcess-like EventEmitter with stdin, stdout, stderr.
|
|
* @param {string[]} stdoutChunks — text chunks emitted in order
|
|
* @param {number} [exitCode=0]
|
|
*/
|
|
function makeMockSpawn(stdoutChunks, exitCode = 0) {
|
|
return function mockSpawnImpl(_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
// Emit stdout chunks and close asynchronously
|
|
setImmediate(async () => {
|
|
for (const chunk of stdoutChunks) {
|
|
proc.stdout.emit('data', Buffer.from(chunk));
|
|
}
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', exitCode, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
};
|
|
}
|
|
|
|
// ── Suite D17: Alias-aware getProviderForModel (Finding 12 + 13) ─────────────
|
|
//
|
|
// Tests that getProviderForModel resolves aliases from models-registry.json
|
|
// to canonical IDs and routes to the correct (enabled) provider.
|
|
|
|
describe('D17 — alias-aware getProviderForModel', () => {
|
|
|
|
// ── Anthropic aliases ────────────────────────────────────────────────
|
|
it('D17: alias "sonnet" → anthropic, canonical claude-sonnet-4-6', () => {
|
|
const loaded = new Map([['anthropic', anthropic]]);
|
|
const r = getProviderForModel(loaded, 'sonnet');
|
|
assert.ok(r !== null);
|
|
assert.equal(r.name, 'anthropic');
|
|
assert.equal(r.canonicalModel, 'claude-sonnet-4-6');
|
|
});
|
|
|
|
it('D17: alias "claude" → anthropic, canonical claude-sonnet-4-6', () => {
|
|
const loaded = new Map([['anthropic', anthropic]]);
|
|
const r = getProviderForModel(loaded, 'claude');
|
|
assert.ok(r !== null);
|
|
assert.equal(r.name, 'anthropic');
|
|
assert.equal(r.canonicalModel, 'claude-sonnet-4-6');
|
|
});
|
|
|
|
it('D17: alias "opus" → anthropic, canonical claude-opus-4-7', () => {
|
|
const loaded = new Map([['anthropic', anthropic]]);
|
|
const r = getProviderForModel(loaded, 'opus');
|
|
assert.ok(r !== null);
|
|
assert.equal(r.name, 'anthropic');
|
|
assert.equal(r.canonicalModel, 'claude-opus-4-7');
|
|
});
|
|
|
|
it('D17: alias "haiku" → anthropic, canonical claude-haiku-4-5', () => {
|
|
const loaded = new Map([['anthropic', anthropic]]);
|
|
const r = getProviderForModel(loaded, 'haiku');
|
|
assert.ok(r !== null);
|
|
assert.equal(r.name, 'anthropic');
|
|
assert.equal(r.canonicalModel, 'claude-haiku-4-5');
|
|
});
|
|
|
|
// ── OpenAI aliases ────────────────────────────────────────────────────
|
|
it('D17: alias "codex" → openai, canonical gpt-5.3-codex', () => {
|
|
const loaded = new Map([['openai', codex]]);
|
|
const r = getProviderForModel(loaded, 'codex');
|
|
assert.ok(r !== null);
|
|
assert.equal(r.name, 'openai');
|
|
assert.equal(r.canonicalModel, 'gpt-5.3-codex');
|
|
});
|
|
|
|
it('D17: alias "gpt5" → openai, canonical gpt-5.5', () => {
|
|
const loaded = new Map([['openai', codex]]);
|
|
const r = getProviderForModel(loaded, 'gpt5');
|
|
assert.ok(r !== null);
|
|
assert.equal(r.name, 'openai');
|
|
assert.equal(r.canonicalModel, 'gpt-5.5');
|
|
});
|
|
|
|
// ── Mistral aliases ───────────────────────────────────────────────────
|
|
it('D17: alias "devstral" → mistral, canonical devstral-2-25-12', () => {
|
|
const loaded = new Map([['mistral', mistral]]);
|
|
const r = getProviderForModel(loaded, 'devstral');
|
|
assert.ok(r !== null);
|
|
assert.equal(r.name, 'mistral');
|
|
assert.equal(r.canonicalModel, 'devstral-2-25-12');
|
|
});
|
|
|
|
it('D17: alias "devstral-small" → mistral, canonical devstral-small-2-25-12', () => {
|
|
const loaded = new Map([['mistral', mistral]]);
|
|
const r = getProviderForModel(loaded, 'devstral-small');
|
|
assert.ok(r !== null);
|
|
assert.equal(r.name, 'mistral');
|
|
assert.equal(r.canonicalModel, 'devstral-small-2-25-12');
|
|
});
|
|
|
|
// ── Canonical pass-through ────────────────────────────────────────────
|
|
it('D17: canonical "claude-sonnet-4-6" → anthropic, canonicalModel unchanged', () => {
|
|
const loaded = new Map([['anthropic', anthropic]]);
|
|
const r = getProviderForModel(loaded, 'claude-sonnet-4-6');
|
|
assert.ok(r !== null);
|
|
assert.equal(r.name, 'anthropic');
|
|
assert.equal(r.canonicalModel, 'claude-sonnet-4-6');
|
|
});
|
|
|
|
// ── Unknown model → null ──────────────────────────────────────────────
|
|
it('D17: unknown model "gpt-4-imaginary" → null', () => {
|
|
const loaded = new Map([['anthropic', anthropic], ['openai', codex], ['mistral', mistral]]);
|
|
assert.equal(getProviderForModel(loaded, 'gpt-4-imaginary'), null);
|
|
});
|
|
|
|
// ── Alias points to disabled provider → null ─────────────────────────
|
|
it('D17: alias "devstral" with only anthropic loaded → null (mistral not enabled)', () => {
|
|
// The alias is known (devstral → mistral) but mistral is not in loadedProviders.
|
|
const loaded = new Map([['anthropic', anthropic]]);
|
|
assert.equal(getProviderForModel(loaded, 'devstral'), null);
|
|
});
|
|
|
|
it('D17: alias "sonnet" with only mistral loaded → null (anthropic not enabled)', () => {
|
|
const loaded = new Map([['mistral', mistral]]);
|
|
assert.equal(getProviderForModel(loaded, 'sonnet'), null);
|
|
});
|
|
|
|
// ── buildDefaultChain with alias ──────────────────────────────────────
|
|
it('D17: buildDefaultChain("sonnet") → chain carries canonical claude-sonnet-4-6', () => {
|
|
const loaded = new Map([['anthropic', anthropic]]);
|
|
const chain = buildDefaultChain('sonnet', loaded, {}, {});
|
|
assert.ok(chain !== null);
|
|
assert.equal(chain.length, 1);
|
|
assert.equal(chain[0].provider, 'anthropic');
|
|
assert.equal(chain[0].model, 'claude-sonnet-4-6');
|
|
});
|
|
|
|
it('D17: buildDefaultChain("devstral") → chain carries canonical devstral-2-25-12', () => {
|
|
const loaded = new Map([['mistral', mistral]]);
|
|
const chain = buildDefaultChain('devstral', loaded, {}, {});
|
|
assert.ok(chain !== null);
|
|
assert.equal(chain.length, 1);
|
|
assert.equal(chain[0].provider, 'mistral');
|
|
assert.equal(chain[0].model, 'devstral-2-25-12');
|
|
});
|
|
|
|
it('D17: buildDefaultChain("unknown-alias") with no providers → null', () => {
|
|
const loaded = new Map();
|
|
assert.equal(buildDefaultChain('unknown-alias', loaded, {}, {}), null);
|
|
});
|
|
|
|
});
|
|
|
|
describe('Anthropic plugin (D4)', () => {
|
|
|
|
// ── Test 1: Contract conformance ──────────────────────────────────────
|
|
it('anthropic module satisfies validateProvider() — all 10 fields present', () => {
|
|
const { valid, errors } = validateProvider(anthropic);
|
|
assert.equal(valid, true, `Validation errors: ${errors.join('; ')}`);
|
|
// Verify all 10 contract fields explicitly
|
|
assert.ok('name' in anthropic, 'missing: name');
|
|
assert.ok('displayName' in anthropic, 'missing: displayName');
|
|
assert.ok('contractVersion' in anthropic, 'missing: contractVersion');
|
|
assert.ok('models' in anthropic, 'missing: models');
|
|
assert.ok('auth' in anthropic, 'missing: auth');
|
|
assert.ok(typeof anthropic.spawn === 'function', 'missing: spawn');
|
|
assert.ok(typeof anthropic.estimateCost === 'function', 'missing: estimateCost');
|
|
assert.ok(typeof anthropic.quotaStatus === 'function', 'missing: quotaStatus');
|
|
assert.ok(typeof anthropic.healthCheck === 'function', 'missing: healthCheck');
|
|
assert.ok('hints' in anthropic, 'missing: hints');
|
|
});
|
|
|
|
// ── Test 2: contractVersion enforced ─────────────────────────────────
|
|
it('validateProvider rejects provider missing contractVersion', () => {
|
|
const p = makeProvider();
|
|
delete p.contractVersion;
|
|
const r = validateProvider(p);
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('contractVersion')));
|
|
});
|
|
|
|
it('validateProvider rejects contractVersion: "0.9"', () => {
|
|
const p = makeProvider({ contractVersion: '0.9' });
|
|
const r = validateProvider(p);
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('contractVersion')));
|
|
});
|
|
|
|
it('validateProvider accepts contractVersion: "1.0"', () => {
|
|
const p = makeProvider({ contractVersion: '1.0' });
|
|
const r = validateProvider(p);
|
|
assert.equal(r.valid, true);
|
|
});
|
|
|
|
it('validateProvider rejects contractVersion: undefined', () => {
|
|
const p = makeProvider({ contractVersion: undefined });
|
|
const r = validateProvider(p);
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('contractVersion')));
|
|
});
|
|
|
|
// ── Test 3: models match registry ────────────────────────────────────
|
|
it('anthropic.models matches models-registry.json providers.anthropic.models', () => {
|
|
const registryIds = modelsRegistry.providers.anthropic.models.map(m => m.id);
|
|
assert.deepEqual(anthropic.models, registryIds);
|
|
});
|
|
|
|
it('anthropic.models contains the three expected model IDs', () => {
|
|
assert.ok(anthropic.models.includes('claude-opus-4-7'));
|
|
assert.ok(anthropic.models.includes('claude-sonnet-4-6'));
|
|
assert.ok(anthropic.models.includes('claude-haiku-4-5'));
|
|
assert.equal(anthropic.models.length, 3);
|
|
});
|
|
|
|
// ── Test 4: getProviderForModel finds anthropic for each model ────────
|
|
it('getProviderForModel finds anthropic for claude-sonnet-4-6 when enabled', () => {
|
|
const loaded = new Map([['anthropic', anthropic]]);
|
|
const result = getProviderForModel(loaded, 'claude-sonnet-4-6');
|
|
assert.ok(result !== null);
|
|
assert.equal(result.name, 'anthropic');
|
|
});
|
|
|
|
it('getProviderForModel finds anthropic for claude-opus-4-7 when enabled', () => {
|
|
const loaded = new Map([['anthropic', anthropic]]);
|
|
const result = getProviderForModel(loaded, 'claude-opus-4-7');
|
|
assert.ok(result !== null);
|
|
assert.equal(result.name, 'anthropic');
|
|
});
|
|
|
|
it('getProviderForModel finds anthropic for claude-haiku-4-5 when enabled', () => {
|
|
const loaded = new Map([['anthropic', anthropic]]);
|
|
const result = getProviderForModel(loaded, 'claude-haiku-4-5');
|
|
assert.ok(result !== null);
|
|
assert.equal(result.name, 'anthropic');
|
|
});
|
|
|
|
// ── Test 5: irToAnthropic translation ────────────────────────────────
|
|
it('irToAnthropic: user message → plain text', () => {
|
|
const ir = makeIR({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'Hello world' }],
|
|
});
|
|
const prompt = irToAnthropic(ir);
|
|
assert.ok(typeof prompt === 'string');
|
|
assert.ok(prompt.includes('Hello world'));
|
|
assert.ok(!prompt.includes('[System]'));
|
|
assert.ok(!prompt.includes('[Assistant]'));
|
|
});
|
|
|
|
it('irToAnthropic: system + user → system annotation + user text', () => {
|
|
const ir = makeIR({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [
|
|
{ role: 'system', content: 'You are a helper.' },
|
|
{ role: 'user', content: 'What is 2+2?' },
|
|
],
|
|
});
|
|
const prompt = irToAnthropic(ir);
|
|
assert.ok(prompt.includes('[System] You are a helper.'));
|
|
assert.ok(prompt.includes('What is 2+2?'));
|
|
});
|
|
|
|
it('irToAnthropic: assistant turn → [Assistant] annotation', () => {
|
|
const ir = makeIR({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [
|
|
{ role: 'user', content: 'Hi' },
|
|
{ role: 'assistant', content: 'Hello!' },
|
|
{ role: 'user', content: 'Bye' },
|
|
],
|
|
});
|
|
const prompt = irToAnthropic(ir);
|
|
assert.ok(prompt.includes('[Assistant] Hello!'));
|
|
});
|
|
|
|
it('irToAnthropic: response_format json_object injects system prompt', () => {
|
|
const ir = makeIR({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'Give JSON' }],
|
|
response_format: { type: 'json_object' },
|
|
});
|
|
const prompt = irToAnthropic(ir);
|
|
assert.ok(prompt.includes('Reply with valid JSON only'));
|
|
});
|
|
|
|
it('irToAnthropic: tool result turn → [Tool Result] annotation', () => {
|
|
const ir = makeIR({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [
|
|
{ role: 'user', content: 'Search for something' },
|
|
{ role: 'tool', content: '{"results": []}', name: 'search' },
|
|
],
|
|
});
|
|
const prompt = irToAnthropic(ir);
|
|
assert.ok(prompt.includes('[Tool Result'));
|
|
assert.ok(prompt.includes('search'));
|
|
});
|
|
|
|
it('irToAnthropic: array content is JSON-stringified', () => {
|
|
const ir = makeIR({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }],
|
|
});
|
|
const prompt = irToAnthropic(ir);
|
|
assert.ok(typeof prompt === 'string');
|
|
assert.ok(prompt.includes('text'));
|
|
});
|
|
|
|
// ── Test 6: anthropicChunkToIR and anthropicStopToIR ─────────────────
|
|
it('anthropicChunkToIR: produces delta chunk with content', () => {
|
|
const chunk = anthropicChunkToIR('Hello ', false);
|
|
assert.equal(chunk.type, 'delta');
|
|
assert.equal(chunk.content, 'Hello ');
|
|
assert.ok(!('role' in chunk));
|
|
});
|
|
|
|
it('anthropicChunkToIR: first chunk includes role=assistant', () => {
|
|
const chunk = anthropicChunkToIR('Hello', true);
|
|
assert.equal(chunk.type, 'delta');
|
|
assert.equal(chunk.role, 'assistant');
|
|
assert.equal(chunk.content, 'Hello');
|
|
});
|
|
|
|
it('anthropicStopToIR: produces stop chunk with finish_reason', () => {
|
|
const chunk = anthropicStopToIR('stop');
|
|
assert.equal(chunk.type, 'stop');
|
|
assert.equal(chunk.finish_reason, 'stop');
|
|
});
|
|
|
|
// ── Test 7: mock spawn — AsyncIterator yields correct IR chunks ───────
|
|
it('spawn with mock: yields delta chunks then stop chunk', async () => {
|
|
const fakeSpawn = makeMockSpawn(['Hello', ' world']);
|
|
__setSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'claude-sonnet-4-6',
|
|
stream: true,
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
});
|
|
const authCtx = { accessToken: '<fake-oauth-token>' };
|
|
const chunks = [];
|
|
for await (const chunk of anthropic.spawn(ir, authCtx)) {
|
|
chunks.push(chunk);
|
|
}
|
|
// Should have 2 delta chunks + 1 stop chunk
|
|
const deltas = chunks.filter(c => c.type === 'delta');
|
|
const stops = chunks.filter(c => c.type === 'stop');
|
|
assert.ok(deltas.length >= 1, `Expected at least 1 delta, got ${deltas.length}`);
|
|
assert.equal(stops.length, 1, `Expected 1 stop, got ${stops.length}`);
|
|
const allContent = deltas.map(c => c.content).join('');
|
|
assert.equal(allContent, 'Hello world');
|
|
} finally {
|
|
__resetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('spawn with mock: first delta chunk has role=assistant', async () => {
|
|
const fakeSpawn = makeMockSpawn(['Test output']);
|
|
__setSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'claude-sonnet-4-6',
|
|
stream: true,
|
|
messages: [{ role: 'user', content: 'Hello' }],
|
|
});
|
|
const authCtx = { accessToken: '<fake-oauth-token>' };
|
|
const chunks = [];
|
|
for await (const chunk of anthropic.spawn(ir, authCtx)) {
|
|
chunks.push(chunk);
|
|
}
|
|
const firstDelta = chunks.find(c => c.type === 'delta');
|
|
assert.ok(firstDelta, 'No delta chunk found');
|
|
assert.equal(firstDelta.role, 'assistant');
|
|
} finally {
|
|
__resetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('spawn with mock: non-zero exit code throws ProviderError', async () => {
|
|
const fakeSpawn = makeMockSpawn([], 1);
|
|
__setSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'claude-sonnet-4-6',
|
|
stream: true,
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
});
|
|
const authCtx = { accessToken: '<fake-oauth-token>' };
|
|
let caught = null;
|
|
try {
|
|
// eslint-disable-next-line no-unused-vars
|
|
for await (const _chunk of anthropic.spawn(ir, authCtx)) {
|
|
// drain
|
|
}
|
|
} catch (e) {
|
|
caught = e;
|
|
}
|
|
assert.ok(caught instanceof ProviderError, `Expected ProviderError, got ${caught?.constructor?.name}`);
|
|
assert.equal(caught.code, 'SPAWN_FAILED');
|
|
} finally {
|
|
__resetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('spawn throws ProviderError(AUTH_MISSING) when no auth context and no env/file', async () => {
|
|
const fakeSpawn = makeMockSpawn(['output']);
|
|
__setSpawnImpl(fakeSpawn);
|
|
// Temporarily clear the env var if set
|
|
const savedToken = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'claude-sonnet-4-6',
|
|
stream: false,
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
});
|
|
let caught = null;
|
|
try {
|
|
// Pass null as authContext to force re-read
|
|
for await (const _chunk of anthropic.spawn(ir, null)) { // eslint-disable-line no-unused-vars
|
|
// may or may not throw depending on whether credentials exist on this machine
|
|
}
|
|
} catch (e) {
|
|
caught = e;
|
|
}
|
|
// If credentials.json or keychain exists on the test machine, this won't throw.
|
|
// We only assert that IF it throws, it's AUTH_MISSING.
|
|
if (caught !== null) {
|
|
assert.ok(caught instanceof ProviderError, `Expected ProviderError, got ${caught?.constructor?.name}`);
|
|
assert.equal(caught.code, 'AUTH_MISSING');
|
|
}
|
|
} finally {
|
|
if (savedToken !== undefined) process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken;
|
|
__resetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
// ── Test 8: healthCheck — binary not found ────────────────────────────
|
|
it('healthCheck returns {ok: false, error: "claude binary not found"} when binary absent', async () => {
|
|
const result = await anthropicHealthCheck({
|
|
_binaryExistsFn: () => false,
|
|
_authReadFn: () => ({ accessToken: '<fake-token>' }),
|
|
});
|
|
assert.equal(result.ok, false);
|
|
assert.equal(result.error, 'claude binary not found');
|
|
assert.ok(typeof result.latencyMs === 'number');
|
|
});
|
|
|
|
// ── Test 9: healthCheck — auth artifact missing ───────────────────────
|
|
it('healthCheck returns {ok: false, error: "auth artifact missing"} when auth missing', async () => {
|
|
const result = await anthropicHealthCheck({
|
|
_binaryExistsFn: () => true,
|
|
_authReadFn: () => null,
|
|
});
|
|
assert.equal(result.ok, false);
|
|
assert.equal(result.error, 'auth artifact missing');
|
|
assert.ok(typeof result.latencyMs === 'number');
|
|
});
|
|
|
|
it('healthCheck returns {ok: true} when binary and auth present', async () => {
|
|
const result = await anthropicHealthCheck({
|
|
_binaryExistsFn: () => true,
|
|
_authReadFn: () => ({ accessToken: '<fake-token>' }),
|
|
});
|
|
assert.equal(result.ok, true);
|
|
assert.ok(typeof result.latencyMs === 'number');
|
|
});
|
|
|
|
// ── Test 10: estimateCost shape ───────────────────────────────────────
|
|
it('estimateCost returns object with four fields for a valid request', () => {
|
|
const request = makeIR({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [
|
|
{ role: 'system', content: 'You are a helper.' },
|
|
{ role: 'user', content: 'Count to ten.' },
|
|
],
|
|
});
|
|
const result = anthropicEstimateCost(request);
|
|
assert.ok(result !== null, 'estimateCost returned null');
|
|
assert.ok('inputTokens' in result, 'missing inputTokens');
|
|
assert.ok('outputTokensEstimate' in result, 'missing outputTokensEstimate');
|
|
assert.ok('currency' in result, 'missing currency');
|
|
assert.ok('usd' in result, 'missing usd');
|
|
assert.equal(result.currency, 'USD');
|
|
assert.equal(result.usd, null); // not pinned at D4
|
|
assert.ok(result.inputTokens > 0, 'inputTokens should be > 0');
|
|
assert.ok(result.outputTokensEstimate >= 0, 'outputTokensEstimate should be >= 0');
|
|
});
|
|
|
|
it('estimateCost returns null for null/missing request', () => {
|
|
assert.equal(anthropicEstimateCost(null), null);
|
|
assert.equal(anthropicEstimateCost({}), null);
|
|
});
|
|
|
|
// ── Test 11: quotaStatus ──────────────────────────────────────────────
|
|
it('quotaStatus returns null at D4', async () => {
|
|
const result = await anthropicQuotaStatus({});
|
|
assert.equal(result, null);
|
|
});
|
|
|
|
// ── Test 12: auth object shape ────────────────────────────────────────
|
|
it('anthropic.auth has correct shape', () => {
|
|
assert.equal(typeof anthropic.auth.type, 'string');
|
|
assert.equal(typeof anthropic.auth.storage, 'string');
|
|
assert.equal(typeof anthropic.auth.path, 'string');
|
|
assert.ok(anthropic.auth.path.includes('.claude'), 'auth.path should reference .claude directory');
|
|
// Portability check: path must start with the runtime homedir() value (set at module load)
|
|
// rather than a hardcoded literal. Since path.join(homedir(), ...) produces the home dir
|
|
// as a prefix, we verify it matches what homedir() returns at test time.
|
|
assert.ok(
|
|
anthropic.auth.path.startsWith(homedir()),
|
|
`auth.path "${anthropic.auth.path}" should start with homedir() "${homedir()}"`,
|
|
);
|
|
assert.ok(typeof anthropic.auth.refresh === 'string' || anthropic.auth.refresh === null);
|
|
});
|
|
|
|
// ── Test 13: hints shape ──────────────────────────────────────────────
|
|
it('anthropic.hints has correct shape', () => {
|
|
assert.equal(typeof anthropic.hints.requiresTTY, 'boolean');
|
|
assert.equal(typeof anthropic.hints.concurrentSpawnSafe, 'boolean');
|
|
assert.ok(Number.isInteger(anthropic.hints.maxConcurrent) && anthropic.hints.maxConcurrent > 0);
|
|
assert.equal(anthropic.hints.requiresTTY, false);
|
|
});
|
|
|
|
// ── Test 14: loadProviders with anthropic enabled ─────────────────────
|
|
it('loadProviders with {enabled: {anthropic: true}} returns Map of size 1', () => {
|
|
const loaded = loadProviders({ enabled: { anthropic: true } });
|
|
assert.equal(loaded.size, 1);
|
|
assert.ok(loaded.has('anthropic'));
|
|
});
|
|
|
|
it('anthropic loaded via loadProviders passes contract and has correct models', () => {
|
|
const loaded = loadProviders({ enabled: { anthropic: true } });
|
|
const p = loaded.get('anthropic');
|
|
const { valid, errors } = validateProvider(p);
|
|
assert.equal(valid, true, `Contract errors: ${errors.join('; ')}`);
|
|
assert.ok(p.models.includes('claude-sonnet-4-6'));
|
|
});
|
|
|
|
});
|
|
|
|
// ── Suite 7: HTTP integration tests ──────────────────────────────────────
|
|
|
|
describe('HTTP integration', () => {
|
|
let serverInstance;
|
|
let port;
|
|
|
|
before(async () => {
|
|
// Use the REAL server module via its createOlpServer() factory. The
|
|
// main guard in server.mjs prevents auto-listen on import; we call
|
|
// .listen() ourselves on a test port. This means every HTTP test below
|
|
// exercises the real router code — there is no parallel implementation
|
|
// to drift.
|
|
const { createOlpServer } = await import('./server.mjs');
|
|
|
|
// Pick a port: env OLP_TEST_PORT or random high port
|
|
port = parseInt(process.env.OLP_TEST_PORT ?? String(13456 + Math.floor(Math.random() * 1000)), 10);
|
|
|
|
serverInstance = createOlpServer();
|
|
|
|
// Retry once on port-in-use
|
|
await new Promise((resolve, reject) => {
|
|
serverInstance.listen(port, '127.0.0.1', resolve);
|
|
serverInstance.once('error', async (e) => {
|
|
if (e.code === 'EADDRINUSE') {
|
|
port++;
|
|
serverInstance.listen(port, '127.0.0.1', resolve);
|
|
serverInstance.once('error', reject);
|
|
} else {
|
|
reject(e);
|
|
}
|
|
});
|
|
});
|
|
});
|
|
|
|
after(() => new Promise(r => serverInstance.close(r)));
|
|
|
|
it('GET /health returns 200 with expected shape', async () => {
|
|
const r = await fetch({ port, method: 'GET', path: '/health' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.ok, true);
|
|
assert.ok(typeof body.version === 'string');
|
|
assert.ok(typeof body.providers.enabled === 'number');
|
|
assert.ok(typeof body.providers.available === 'number');
|
|
});
|
|
|
|
it('GET /v1/models returns 200 with empty data array', async () => {
|
|
const r = await fetch({ port, method: 'GET', path: '/v1/models' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.object, 'list');
|
|
assert.deepEqual(body.data, []);
|
|
});
|
|
|
|
it('POST /v1/chat/completions with no providers → 503 with no_enabled_provider', async () => {
|
|
const r = await fetch({
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: { model: 'gpt-4o', messages: [{ role: 'user', content: 'Hi' }] },
|
|
});
|
|
assert.equal(r.status, 503);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.error.type, 'no_enabled_provider');
|
|
assert.ok(body.error.message.includes('gpt-4o'));
|
|
});
|
|
|
|
it('POST /v1/chat/completions with invalid JSON body → 400', async () => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json', 'Content-Length': '5' },
|
|
});
|
|
const result = await new Promise((resolve, reject) => {
|
|
req.on('error', reject);
|
|
let body = '';
|
|
req.on('response', res => {
|
|
res.on('data', c => body += c);
|
|
res.on('end', () => resolve({ status: res.statusCode, body }));
|
|
});
|
|
req.write('{bad}');
|
|
req.end();
|
|
});
|
|
assert.equal(result.status, 400);
|
|
});
|
|
|
|
it('POST /v1/chat/completions with missing model → 400', async () => {
|
|
const r = await fetch({
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: { messages: [{ role: 'user', content: 'x' }] },
|
|
});
|
|
assert.equal(r.status, 400);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.error.type, 'invalid_request_error');
|
|
});
|
|
|
|
it('GET /unknown → 404', async () => {
|
|
const r = await fetch({ port, method: 'GET', path: '/unknown/route' });
|
|
assert.equal(r.status, 404);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.error.type, 'not_found');
|
|
});
|
|
|
|
it('POST /v1/chat/completions without Content-Type → 415', async () => {
|
|
// Our fetch helper sets Content-Type to application/json when body is truthy;
|
|
// send text/plain directly to verify the 415 path.
|
|
const result = await new Promise((resolve, reject) => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'text/plain', 'Content-Length': '2' },
|
|
}, res => {
|
|
let data = '';
|
|
res.on('data', c => data += c);
|
|
res.on('end', () => resolve({ status: res.statusCode, body: data }));
|
|
});
|
|
req.on('error', reject);
|
|
req.write('{}');
|
|
req.end();
|
|
});
|
|
assert.equal(result.status, 415);
|
|
});
|
|
});
|
|
|
|
// ── Suite 8: Suite 8 formerly counted as D3/D4 base; suites renumber here ──
|
|
// (No new Suite 8 — the numbering skips from 7 to 9 to match D5 spec.)
|
|
|
|
// ── Suite 9: Cache layer ──────────────────────────────────────────────────
|
|
//
|
|
// Unit tests + HTTP integration tests for ADR 0005 (D1 + D4).
|
|
// No real `claude` binary invoked. Mock spawn injected via __setSpawnImpl.
|
|
// Authority: ADR 0005 § Cache key composition, D1 per-key isolation, D4 singleflight.
|
|
|
|
describe('Cache layer — computeCacheKey (Suite 9)', () => {
|
|
|
|
// ── Test 1: Determinism ───────────────────────────────────────────────
|
|
it('computeCacheKey is deterministic: same inputs → same key', () => {
|
|
const ir = makeIR({ model: 'claude-haiku-4-5', messages: [{ role: 'user', content: 'hello' }] });
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir);
|
|
assert.equal(k1, k2);
|
|
assert.equal(typeof k1, 'string');
|
|
assert.equal(k1.length, 64); // SHA-256 hex
|
|
});
|
|
|
|
// ── Test 2: Provider distinguishes ───────────────────────────────────
|
|
it('computeCacheKey distinguishes different providers', () => {
|
|
const ir = makeIR({ model: 'model-x', messages: [{ role: 'user', content: 'hi' }] });
|
|
const k1 = computeCacheKey('anthropic', 'model-x', ir);
|
|
const k2 = computeCacheKey('openai', 'model-x', ir);
|
|
assert.notEqual(k1, k2);
|
|
});
|
|
|
|
// ── Test 3: Model distinguishes ───────────────────────────────────────
|
|
it('computeCacheKey distinguishes different models', () => {
|
|
const ir = makeIR({ messages: [{ role: 'user', content: 'hi' }] });
|
|
const k1 = computeCacheKey('anthropic', 'claude-sonnet-4-6', ir);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir);
|
|
assert.notEqual(k1, k2);
|
|
});
|
|
|
|
// ── Test 4: Messages distinguishes ───────────────────────────────────
|
|
it('computeCacheKey distinguishes different messages', () => {
|
|
const ir1 = makeIR({ messages: [{ role: 'user', content: 'hello' }] });
|
|
const ir2 = makeIR({ messages: [{ role: 'user', content: 'world' }] });
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir1);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir2);
|
|
assert.notEqual(k1, k2);
|
|
});
|
|
|
|
// ── Test 5: Tools distinguishes ───────────────────────────────────────
|
|
it('computeCacheKey distinguishes requests with vs without tools', () => {
|
|
const base = makeIR({ messages: [{ role: 'user', content: 'search' }] });
|
|
const withTools = makeIR({
|
|
messages: [{ role: 'user', content: 'search' }],
|
|
tools: [{ type: 'function', function: { name: 'search', description: 'web search', parameters: {} } }],
|
|
});
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', base);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', withTools);
|
|
assert.notEqual(k1, k2);
|
|
});
|
|
|
|
// ── Test 6: Temperature distinguishes ────────────────────────────────
|
|
it('computeCacheKey distinguishes different temperature values', () => {
|
|
const ir1 = makeIR({ messages: [{ role: 'user', content: 'x' }], temperature: 0.0 });
|
|
const ir2 = makeIR({ messages: [{ role: 'user', content: 'x' }], temperature: 1.0 });
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir1);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir2);
|
|
assert.notEqual(k1, k2);
|
|
});
|
|
|
|
// ── Test 7: response_format distinguishes ────────────────────────────
|
|
it('computeCacheKey distinguishes different response_format values', () => {
|
|
const ir1 = makeIR({ messages: [{ role: 'user', content: 'x' }] });
|
|
const ir2 = makeIR({ messages: [{ role: 'user', content: 'x' }], response_format: { type: 'json_object' } });
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir1);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir2);
|
|
assert.notEqual(k1, k2);
|
|
});
|
|
|
|
// ── Test 8: Content array property order stability ───────────────────
|
|
it('computeCacheKey is stable for content arrays with same properties in different insertion order', () => {
|
|
const ir1 = makeIR({
|
|
messages: [{ role: 'user', content: [{ type: 'text', text: 'hi', extra: 1 }] }],
|
|
});
|
|
const ir2 = makeIR({
|
|
messages: [{ role: 'user', content: [{ extra: 1, text: 'hi', type: 'text' }] }],
|
|
});
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir1);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir2);
|
|
assert.equal(k1, k2);
|
|
});
|
|
|
|
// ── D15 Tests: Amendment 2 — max_tokens, top_p, stop, tool_choice ────
|
|
|
|
// ── Test 9: max_tokens distinguishes ─────────────────────────────────
|
|
it('computeCacheKey differs when max_tokens differs (D15 Amendment 2)', () => {
|
|
const ir1 = makeIR({ messages: [{ role: 'user', content: 'x' }], max_tokens: 100 });
|
|
const ir2 = makeIR({ messages: [{ role: 'user', content: 'x' }], max_tokens: 4000 });
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir1);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir2);
|
|
assert.notEqual(k1, k2);
|
|
});
|
|
|
|
// ── Test 10: top_p distinguishes ──────────────────────────────────────
|
|
it('computeCacheKey differs when top_p differs (D15 Amendment 2)', () => {
|
|
const ir1 = makeIR({ messages: [{ role: 'user', content: 'x' }], top_p: 0.5 });
|
|
const ir2 = makeIR({ messages: [{ role: 'user', content: 'x' }], top_p: 0.9 });
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir1);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir2);
|
|
assert.notEqual(k1, k2);
|
|
});
|
|
|
|
// ── Test 11: stop distinguishes ───────────────────────────────────────
|
|
it('computeCacheKey differs when stop sequences differ (D15 Amendment 2)', () => {
|
|
const ir1 = makeIR({ messages: [{ role: 'user', content: 'x' }], stop: ['\n'] });
|
|
const ir2 = makeIR({ messages: [{ role: 'user', content: 'x' }], stop: ['END', '\n'] });
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir1);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir2);
|
|
assert.notEqual(k1, k2);
|
|
});
|
|
|
|
// ── Test 12: tool_choice distinguishes ───────────────────────────────
|
|
it('computeCacheKey differs when tool_choice differs (D15 Amendment 2)', () => {
|
|
const ir1 = makeIR({ messages: [{ role: 'user', content: 'x' }], tool_choice: 'auto' });
|
|
const ir2 = makeIR({ messages: [{ role: 'user', content: 'x' }], tool_choice: 'none' });
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir1);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir2);
|
|
assert.notEqual(k1, k2);
|
|
});
|
|
|
|
// ── Test 13: undefined max_tokens stable (both absent → same key) ────
|
|
it('computeCacheKey is identical when max_tokens absent in both requests (D15 Amendment 2)', () => {
|
|
const ir1 = makeIR({ messages: [{ role: 'user', content: 'x' }] });
|
|
const ir2 = makeIR({ messages: [{ role: 'user', content: 'x' }] });
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir1);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir2);
|
|
assert.equal(k1, k2);
|
|
});
|
|
|
|
// ── F4 regression (D34): tools:[] vs tools:undefined must produce identical keys ─
|
|
// ADR 0005 Amendment 2 claims: "tools: [] (explicit empty array) and tools omitted
|
|
// (undefined) both mean 'no tools available' and produce identical model output,
|
|
// so they correctly share a cache entry."
|
|
// computeCacheKey now normalizes [] → null before serialization so this claim holds.
|
|
it('F4 regression: tools:[] and tools:undefined produce the same cache key (D34)', () => {
|
|
const base = makeIR({ messages: [{ role: 'user', content: 'hello' }] });
|
|
const irEmpty = { ...base, tools: [] };
|
|
const irUndef = { ...base, tools: undefined };
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', irEmpty);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', irUndef);
|
|
assert.equal(k1, k2, 'tools:[] and tools:undefined must produce identical cache keys');
|
|
});
|
|
|
|
it('F4 regression: stop:[] and stop:undefined produce the same cache key (D34)', () => {
|
|
const base = makeIR({ messages: [{ role: 'user', content: 'hello' }] });
|
|
const irEmpty = { ...base, stop: [] };
|
|
const irUndef = { ...base, stop: undefined };
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', irEmpty);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', irUndef);
|
|
assert.equal(k1, k2, 'stop:[] and stop:undefined must produce identical cache keys');
|
|
});
|
|
|
|
it('F4 regression: tools:[] and tools:null produce the same cache key (D34)', () => {
|
|
const base = makeIR({ messages: [{ role: 'user', content: 'hello' }] });
|
|
const irEmpty = { ...base, tools: [] };
|
|
const irNull = { ...base, tools: null };
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', irEmpty);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', irNull);
|
|
assert.equal(k1, k2, 'tools:[] and tools:null must produce identical cache keys');
|
|
});
|
|
|
|
it('F4 regression: non-empty tools array still differs from tools:undefined (D34)', () => {
|
|
const base = makeIR({ messages: [{ role: 'user', content: 'hello' }] });
|
|
const irWithTools = { ...base, tools: [{ type: 'function', function: { name: 'foo' } }] };
|
|
const irNoTools = { ...base, tools: undefined };
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', irWithTools);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', irNoTools);
|
|
assert.notEqual(k1, k2, 'non-empty tools array must produce a different cache key from no tools');
|
|
});
|
|
});
|
|
|
|
// ── D36 #14: cache_control slot determinism regression ─────────────────────
|
|
//
|
|
// Context (issue #14 + ADR 0005 Amendment 4):
|
|
// computeCacheKey at lib/cache/keys.mjs:~224 calls extractCacheControlMarkers
|
|
// on ir.messages. openAIToIR (lib/ir/openai-to-ir.mjs translateMessage) does
|
|
// NOT whitelist cache_control, so v0.1 IRs constructed from OpenAI requests
|
|
// always have markers stripped → the slot is structurally null. D27 F10
|
|
// Amendment 4 acknowledged this is forward-compatible: when a future ADR 0003
|
|
// amendment adds cache_control to the IR field set, the slot starts carrying
|
|
// meaningful data without a cache-key schema change.
|
|
//
|
|
// What this suite guards:
|
|
// 1. The cache_control slot IS populated when markers are present on the IR
|
|
// (forward-compat scenario — markers attached directly to an IR object
|
|
// bypassing the openAIToIR translation).
|
|
// 2. Computing the key twice with the same marker payload yields identical
|
|
// keys (determinism — bedrock invariant per ADR 0005 § cache key stability).
|
|
// 3. The slot is included in the composition — two IRs differing ONLY in
|
|
// cache_control markers produce different keys.
|
|
//
|
|
// Future activation contract (when ADR 0003 adds cache_control to the IR
|
|
// whitelist and openAIToIR starts preserving markers):
|
|
// - extractCacheControlMarkers must continue to return markers in messages-
|
|
// iteration order; nested-in-content markers follow the outer message in
|
|
// positional order (see lib/cache/keys.mjs:126-148).
|
|
// - The current implementation JSON.stringifies the marker array verbatim
|
|
// (no pre-sort) — determinism relies on messages-array order being stable
|
|
// across translations and on per-marker object key insertion order being
|
|
// stable (which it is for objects authored at one source site).
|
|
// - If a future amendment introduces semantic equivalences across marker
|
|
// orderings (e.g., "two markers in different positions should still hit
|
|
// the same cache entry"), the implementation must add a sortMarkers helper
|
|
// before serialization. This test would then need an update to assert the
|
|
// normalized-order behavior. As of D36, no such equivalence is claimed; the
|
|
// test asserts the strict "same input → same output" determinism only.
|
|
//
|
|
// Risk path chosen: option (a) test-only (no code change in lib/cache/keys.mjs).
|
|
// Rationale: the dead-code path is unreachable from openAIToIR at v0.1, so a
|
|
// helper added now would have zero callers; preferred to defer the helper until
|
|
// the IR amendment actually activates the slot. Per ALIGNMENT.md Rule 2 (No
|
|
// Invention), shipping a sortMarkers helper today would be invention without a
|
|
// caller authority. Adding tests is risk-free; adding helpers is not.
|
|
|
|
describe('D36 #14 — cache_control slot determinism regression', () => {
|
|
|
|
// Construct an IR with synthetic cache_control markers attached directly
|
|
// (bypass openAIToIR which strips them at v0.1).
|
|
function makeIRWithMarkers(markers, opts = {}) {
|
|
const messages = opts.messages ?? [
|
|
{ role: 'user', content: 'sample-prompt' },
|
|
];
|
|
// Attach the first marker to the first message; if a second marker is
|
|
// provided, attach it to the second message (or to message 0 nested in a
|
|
// content array, depending on opts.nestSecond).
|
|
const annotated = messages.map((m, idx) => {
|
|
if (idx === 0 && markers[0]) {
|
|
return { ...m, cache_control: markers[0] };
|
|
}
|
|
if (idx === 1 && markers[1]) {
|
|
return { ...m, cache_control: markers[1] };
|
|
}
|
|
return m;
|
|
});
|
|
return makeIR({ messages: annotated, ...opts.irOverrides });
|
|
}
|
|
|
|
it('D36 #14a: cache_control slot is populated when markers are present on the IR', () => {
|
|
// The slot is normally null at v0.1 (openAIToIR strips). When markers ARE
|
|
// present (e.g. an IR constructed directly with markers, or after a future
|
|
// ADR 0003 amendment preserves them), the slot carries the markers.
|
|
const irNoMarkers = makeIR({ messages: [{ role: 'user', content: 'hi' }] });
|
|
const irWithMarker = makeIRWithMarkers([{ type: 'ephemeral' }], {
|
|
messages: [{ role: 'user', content: 'hi' }],
|
|
});
|
|
|
|
const kNo = computeCacheKey('anthropic', 'claude-haiku-4-5', irNoMarkers);
|
|
const kYes = computeCacheKey('anthropic', 'claude-haiku-4-5', irWithMarker);
|
|
|
|
// Different keys: marker presence is observable in the cache key.
|
|
assert.notEqual(kNo, kYes,
|
|
'IR with cache_control markers must produce a different cache key from IR without markers');
|
|
assert.equal(typeof kYes, 'string');
|
|
assert.equal(kYes.length, 64);
|
|
});
|
|
|
|
it('D36 #14b: cache key is deterministic across two computations on the same IR with markers', () => {
|
|
// The bedrock invariant: same inputs → same key. The cache_control slot
|
|
// must not introduce nondeterminism (e.g., timestamp, random, iteration
|
|
// order from a Map).
|
|
const ir = makeIRWithMarkers([{ type: 'ephemeral' }], {
|
|
messages: [
|
|
{ role: 'user', content: 'first turn' },
|
|
{ role: 'assistant', content: 'first reply' },
|
|
],
|
|
});
|
|
const k1 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir);
|
|
const k2 = computeCacheKey('anthropic', 'claude-haiku-4-5', ir);
|
|
assert.equal(k1, k2, 'cache key must be deterministic for the same IR with markers');
|
|
});
|
|
|
|
it('D36 #14c: same markers in the same positional order produce the same key', () => {
|
|
// Two IRs constructed independently with the same marker payload at the
|
|
// same positions must produce the same key. This is the forward-activation
|
|
// contract: when openAIToIR preserves markers, two identical OpenAI
|
|
// requests must produce identical cache keys.
|
|
const irA = makeIRWithMarkers([{ type: 'ephemeral' }], {
|
|
messages: [{ role: 'user', content: 'hello' }],
|
|
});
|
|
const irB = makeIRWithMarkers([{ type: 'ephemeral' }], {
|
|
messages: [{ role: 'user', content: 'hello' }],
|
|
});
|
|
const kA = computeCacheKey('anthropic', 'claude-haiku-4-5', irA);
|
|
const kB = computeCacheKey('anthropic', 'claude-haiku-4-5', irB);
|
|
assert.equal(kA, kB,
|
|
'two independently-constructed IRs with identical marker payloads must share a cache key');
|
|
});
|
|
|
|
it('D36 #14d: markers nested in content array participate in the cache key (extractCacheControlMarkers contract)', () => {
|
|
// extractCacheControlMarkers (lib/cache/keys.mjs:126-148) finds markers at
|
|
// both message top-level AND nested inside content array parts. The cache
|
|
// key must reflect both surfaces. This is the slot's full forward-compat
|
|
// contract — when the IR carries cache_control, OLP must observe it on the
|
|
// wire shape the IR provides, including the OpenAI-style content-array
|
|
// nesting that Anthropic prompt-caching uses in production.
|
|
const irTopLevel = makeIR({
|
|
messages: [
|
|
{ role: 'user', content: 'hi', cache_control: { type: 'ephemeral' } },
|
|
],
|
|
});
|
|
const irNested = makeIR({
|
|
messages: [
|
|
{
|
|
role: 'user',
|
|
content: [
|
|
{ type: 'text', text: 'hi', cache_control: { type: 'ephemeral' } },
|
|
],
|
|
},
|
|
],
|
|
});
|
|
const irNoMarkers = makeIR({
|
|
messages: [{ role: 'user', content: 'hi' }],
|
|
});
|
|
|
|
const kTop = computeCacheKey('anthropic', 'claude-haiku-4-5', irTopLevel);
|
|
const kNested = computeCacheKey('anthropic', 'claude-haiku-4-5', irNested);
|
|
const kNone = computeCacheKey('anthropic', 'claude-haiku-4-5', irNoMarkers);
|
|
|
|
// Both marker surfaces must register against the key (be observably distinct
|
|
// from the no-markers case).
|
|
assert.notEqual(kTop, kNone, 'top-level cache_control marker must affect cache key');
|
|
assert.notEqual(kNested, kNone, 'nested cache_control marker must affect cache key');
|
|
});
|
|
});
|
|
|
|
describe('Cache layer — extractCacheControlMarkers + hasCacheControl (Suite 9 cont.)', () => {
|
|
|
|
// ── Test 9: extractCacheControlMarkers — top-level ───────────────────
|
|
it('extractCacheControlMarkers finds cache_control at message top level', () => {
|
|
const messages = [
|
|
{ role: 'user', content: 'hi', cache_control: { type: 'ephemeral' } },
|
|
];
|
|
const markers = extractCacheControlMarkers(messages);
|
|
assert.equal(markers.length, 1);
|
|
assert.deepEqual(markers[0], { type: 'ephemeral' });
|
|
});
|
|
|
|
// ── Test 10: extractCacheControlMarkers — nested in content array ─────
|
|
it('extractCacheControlMarkers finds cache_control nested in content array', () => {
|
|
const messages = [
|
|
{
|
|
role: 'user',
|
|
content: [
|
|
{ type: 'text', text: 'hello', cache_control: { type: 'ephemeral' } },
|
|
{ type: 'text', text: 'world' },
|
|
],
|
|
},
|
|
];
|
|
const markers = extractCacheControlMarkers(messages);
|
|
assert.equal(markers.length, 1);
|
|
assert.deepEqual(markers[0], { type: 'ephemeral' });
|
|
});
|
|
|
|
// ── Test 11: extractCacheControlMarkers — no markers ─────────────────
|
|
it('extractCacheControlMarkers returns [] when no cache_control markers present', () => {
|
|
const messages = [
|
|
{ role: 'user', content: 'hello' },
|
|
{ role: 'assistant', content: 'world' },
|
|
];
|
|
assert.deepEqual(extractCacheControlMarkers(messages), []);
|
|
});
|
|
|
|
// ── Test 12: hasCacheControl — true ──────────────────────────────────
|
|
it('hasCacheControl returns true when cache_control markers present', () => {
|
|
const ir = makeIR({
|
|
messages: [{ role: 'user', content: 'hi', cache_control: { type: 'ephemeral' } }],
|
|
});
|
|
assert.equal(hasCacheControl(ir), true);
|
|
});
|
|
|
|
// ── Test 13: hasCacheControl — false ─────────────────────────────────
|
|
it('hasCacheControl returns false when no cache_control markers', () => {
|
|
const ir = makeIR({ messages: [{ role: 'user', content: 'hi' }] });
|
|
assert.equal(hasCacheControl(ir), false);
|
|
});
|
|
|
|
// ── Test 14: hasCacheControl — null/undefined safety ─────────────────
|
|
it('hasCacheControl returns false for null/undefined ir', () => {
|
|
assert.equal(hasCacheControl(null), false);
|
|
assert.equal(hasCacheControl(undefined), false);
|
|
assert.equal(hasCacheControl({}), false);
|
|
});
|
|
});
|
|
|
|
describe('Cache layer — CacheStore unit tests (Suite 9 cont.)', () => {
|
|
|
|
// ── Test 15: set/get round-trip ───────────────────────────────────────
|
|
it('CacheStore.set/get round-trips a value', async () => {
|
|
const store = new CacheStore();
|
|
await store.set('keyA', 'hash1', [{ type: 'stop', finish_reason: 'stop' }]);
|
|
const entry = await store.get('keyA', 'hash1');
|
|
assert.ok(entry !== null);
|
|
assert.deepEqual(entry.value, [{ type: 'stop', finish_reason: 'stop' }]);
|
|
});
|
|
|
|
// ── Test 16: get returns null for missing key ─────────────────────────
|
|
it('CacheStore.get returns null for missing (keyId, cacheKey)', async () => {
|
|
const store = new CacheStore();
|
|
const entry = await store.get('keyA', 'nonexistent-hash');
|
|
assert.equal(entry, null);
|
|
});
|
|
|
|
// ── Test 17: Per-key isolation ────────────────────────────────────────
|
|
it('CacheStore per-key isolation: keyId1 entries invisible to keyId2', async () => {
|
|
const store = new CacheStore();
|
|
await store.set('keyA', 'hash1', 'value-for-keyA');
|
|
const fromKeyA = await store.get('keyA', 'hash1');
|
|
const fromKeyB = await store.get('keyB', 'hash1');
|
|
assert.ok(fromKeyA !== null);
|
|
assert.equal(fromKeyA.value, 'value-for-keyA');
|
|
assert.equal(fromKeyB, null);
|
|
});
|
|
|
|
// ── Test 18: has ─────────────────────────────────────────────────────
|
|
it('CacheStore.has returns true for existing entry, false for missing', async () => {
|
|
const store = new CacheStore();
|
|
await store.set('keyA', 'hash1', 'val');
|
|
assert.equal(await store.has('keyA', 'hash1'), true);
|
|
assert.equal(await store.has('keyA', 'hash-missing'), false);
|
|
assert.equal(await store.has('keyB', 'hash1'), false);
|
|
});
|
|
|
|
// ── Test 19: TTL expiry ───────────────────────────────────────────────
|
|
it('CacheStore respects TTL: expired entries return null', async () => {
|
|
// Inject a _nowFn to control time without sleeping.
|
|
// First call: "now" = 0 (entry creation time).
|
|
// Second call: "now" = 2000 (2 seconds later; entry has 1s TTL → expired).
|
|
let fakeNow = 0;
|
|
const store = new CacheStore({ _nowFn: () => fakeNow });
|
|
await store.set('keyA', 'hash1', 'val', 1000); // 1000ms TTL
|
|
// Entry should be alive at t=0
|
|
const entry1 = await store.get('keyA', 'hash1');
|
|
assert.ok(entry1 !== null, 'Expected entry to be alive at t=0');
|
|
// Advance time past TTL
|
|
fakeNow = 2000;
|
|
const entry2 = await store.get('keyA', 'hash1');
|
|
assert.equal(entry2, null, 'Expected entry to be expired at t=2000');
|
|
});
|
|
|
|
// ── Test 20: stats reports hits/misses ────────────────────────────────
|
|
it('CacheStore.stats reports hits and misses', async () => {
|
|
const store = new CacheStore();
|
|
await store.set('keyA', 'hash1', 'val');
|
|
await store.get('keyA', 'hash1'); // hit
|
|
await store.get('keyA', 'hash1'); // hit
|
|
await store.get('keyA', 'missing'); // miss
|
|
const s = store.stats('keyA');
|
|
assert.ok(s.hits >= 2, `Expected hits >= 2, got ${s.hits}`);
|
|
assert.ok(s.misses >= 1, `Expected misses >= 1, got ${s.misses}`);
|
|
assert.ok(typeof s.size === 'number');
|
|
assert.ok(typeof s.inflightCount === 'number');
|
|
});
|
|
|
|
// ── Test 21: getOrCompute returns computed value on miss ──────────────
|
|
it('getOrCompute returns computed value on miss and caches it', async () => {
|
|
const store = new CacheStore();
|
|
let callCount = 0;
|
|
const computeFn = async () => { callCount++; return [{ type: 'delta', content: 'hello' }]; };
|
|
const v1 = await store.getOrCompute('keyA', 'hash1', computeFn);
|
|
assert.deepEqual(v1, [{ type: 'delta', content: 'hello' }]);
|
|
assert.equal(callCount, 1);
|
|
});
|
|
|
|
// ── Test 22: getOrCompute returns cached value on hit (no recomputation) ──
|
|
it('getOrCompute returns cached value on hit — computeFn not called again', async () => {
|
|
const store = new CacheStore();
|
|
let callCount = 0;
|
|
const computeFn = async () => { callCount++; return 'computed-value'; };
|
|
await store.getOrCompute('keyA', 'hash1', computeFn);
|
|
const v2 = await store.getOrCompute('keyA', 'hash1', computeFn);
|
|
assert.equal(v2, 'computed-value');
|
|
assert.equal(callCount, 1, 'computeFn should only be called once on cache hit');
|
|
});
|
|
|
|
// ── Test 23: Singleflight — the key D4 test ───────────────────────────
|
|
// 5 concurrent getOrCompute calls with same key + slow computeFn (50ms).
|
|
// Verifies: computeFn called exactly once; all 5 callers receive same value;
|
|
// all 5 return within a tight time window (singleflight working).
|
|
it('getOrCompute singleflight: 5 concurrent callers → computeFn called exactly once', async () => {
|
|
const store = new CacheStore();
|
|
let callCount = 0;
|
|
|
|
const slowCompute = async () => {
|
|
callCount++;
|
|
// Simulate a slow provider spawn (50ms)
|
|
await new Promise(r => setTimeout(r, 50));
|
|
return [{ type: 'delta', content: 'singleflight-result' }, { type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
|
|
const t0 = Date.now();
|
|
// Launch 5 concurrent callers simultaneously
|
|
const results = await Promise.all([
|
|
store.getOrCompute('keyA', 'sf-hash', slowCompute),
|
|
store.getOrCompute('keyA', 'sf-hash', slowCompute),
|
|
store.getOrCompute('keyA', 'sf-hash', slowCompute),
|
|
store.getOrCompute('keyA', 'sf-hash', slowCompute),
|
|
store.getOrCompute('keyA', 'sf-hash', slowCompute),
|
|
]);
|
|
const elapsed = Date.now() - t0;
|
|
|
|
// Singleflight invariant: computeFn called exactly once
|
|
assert.equal(callCount, 1, `Expected computeFn called 1 time, got ${callCount}`);
|
|
|
|
// All 5 callers receive the same result
|
|
for (const result of results) {
|
|
assert.deepEqual(result, [{ type: 'delta', content: 'singleflight-result' }, { type: 'stop', finish_reason: 'stop' }]);
|
|
}
|
|
|
|
// All 5 callers return within a tight window (not 5 * 50ms = 250ms).
|
|
// Allow generous margin (3x the slow compute time) for CI variance.
|
|
assert.ok(elapsed < 250, `Expected singleflight to complete in < 250ms, took ${elapsed}ms`);
|
|
});
|
|
|
|
// ── Test 24: getOrCompute releases inflight on completion ────────────
|
|
it('getOrCompute: subsequent calls after completion hit cache (no re-compute)', async () => {
|
|
const store = new CacheStore();
|
|
let callCount = 0;
|
|
const computeFn = async () => { callCount++; return 'done'; };
|
|
// First call — computes and caches
|
|
await store.getOrCompute('keyA', 'hash1', computeFn);
|
|
assert.equal(callCount, 1);
|
|
// Verify inflight is released: calling again should hit cache
|
|
await store.getOrCompute('keyA', 'hash1', computeFn);
|
|
assert.equal(callCount, 1, 'computeFn should not be called again after completion');
|
|
assert.equal(store.stats('keyA').inflightCount, 0, 'No inflight entries after completion');
|
|
});
|
|
|
|
// ── Test 25: getOrCompute releases inflight on error ─────────────────
|
|
it('getOrCompute: inflight released when computeFn throws; subsequent calls retry', async () => {
|
|
const store = new CacheStore();
|
|
let callCount = 0;
|
|
let shouldFail = true;
|
|
const computeFn = async () => {
|
|
callCount++;
|
|
if (shouldFail) throw new Error('provider error');
|
|
return 'success';
|
|
};
|
|
|
|
// First call — throws
|
|
await assert.rejects(() => store.getOrCompute('keyA', 'hash1', computeFn), /provider error/);
|
|
assert.equal(callCount, 1);
|
|
// Inflight must be released even after error
|
|
assert.equal(store.stats('keyA').inflightCount, 0, 'Inflight must be released after error');
|
|
|
|
// Second call — now succeeds (verify re-try works)
|
|
shouldFail = false;
|
|
const v = await store.getOrCompute('keyA', 'hash1', computeFn);
|
|
assert.equal(v, 'success');
|
|
assert.equal(callCount, 2, 'computeFn should be retried after error');
|
|
});
|
|
|
|
// ── D23 Tests: size cap (ADR 0005 § Cache write conditions item 4) ───
|
|
|
|
// ── Test 26: default maxEntryBytes is 10 MB ───────────────────────────
|
|
it('CacheStore default maxEntryBytes is 10 MB (10_485_760 bytes)', () => {
|
|
const store = new CacheStore();
|
|
assert.equal(store._maxEntryBytes, 10 * 1024 * 1024,
|
|
`Expected 10485760, got ${store._maxEntryBytes}`);
|
|
});
|
|
|
|
// ── Test 27: custom maxEntryBytes is respected ────────────────────────
|
|
it('CacheStore with custom maxEntryBytes respects the config', () => {
|
|
const store = new CacheStore({ maxEntryBytes: 100 });
|
|
assert.equal(store._maxEntryBytes, 100);
|
|
});
|
|
|
|
// ── Test 28: set() skips persistence for oversized value ──────────────
|
|
it('CacheStore.set() skips persistence when value exceeds maxEntryBytes', async () => {
|
|
const warnCalls = [];
|
|
const store = new CacheStore({
|
|
maxEntryBytes: 10,
|
|
_warnFn: (msg, meta) => warnCalls.push({ msg, meta }),
|
|
});
|
|
|
|
// Value that serializes to > 10 bytes
|
|
const bigValue = { content: 'hello world this is definitely more than 10 bytes' };
|
|
await store.set('keyA', 'hash1', bigValue);
|
|
|
|
// get() should return null (entry was not stored)
|
|
const entry = await store.get('keyA', 'hash1');
|
|
assert.equal(entry, null, 'Expected oversized entry to not be stored');
|
|
|
|
// Warn should have fired
|
|
assert.equal(warnCalls.length, 1, 'Expected exactly one warn call');
|
|
assert.equal(warnCalls[0].msg, 'cache_skip_oversize');
|
|
assert.ok(warnCalls[0].meta.byteLength > 10, `Expected byteLength > 10, got ${warnCalls[0].meta.byteLength}`);
|
|
assert.equal(warnCalls[0].meta.maxEntryBytes, 10);
|
|
assert.equal(warnCalls[0].meta.keyId, 'keyA');
|
|
assert.equal(warnCalls[0].meta.cacheKey, 'hash1');
|
|
});
|
|
|
|
// ── Test 29: set() persists normally for value within size cap ────────
|
|
it('CacheStore.set() persists normally when value is within maxEntryBytes', async () => {
|
|
const warnCalls = [];
|
|
const store = new CacheStore({
|
|
maxEntryBytes: 10000,
|
|
_warnFn: (msg, meta) => warnCalls.push({ msg, meta }),
|
|
});
|
|
|
|
const smallValue = { content: 'hi' };
|
|
await store.set('keyA', 'hash1', smallValue);
|
|
|
|
const entry = await store.get('keyA', 'hash1');
|
|
assert.ok(entry !== null, 'Expected small entry to be stored');
|
|
assert.deepEqual(entry.value, smallValue);
|
|
assert.equal(warnCalls.length, 0, 'Expected no warn calls for small value');
|
|
});
|
|
|
|
// ── Test 30: getOrCompute with oversized result — returns value but does NOT cache ──
|
|
it('getOrCompute with oversized result: returns value to caller but does NOT cache it', async () => {
|
|
const warnCalls = [];
|
|
const store = new CacheStore({
|
|
maxEntryBytes: 10,
|
|
_warnFn: (msg, meta) => warnCalls.push({ msg, meta }),
|
|
});
|
|
|
|
let computeCallCount = 0;
|
|
const bigValue = [{ type: 'delta', content: 'hello world this is over ten bytes for sure' }];
|
|
const computeFn = async () => { computeCallCount++; return bigValue; };
|
|
|
|
// First call — computes, oversized, skips cache
|
|
const v1 = await store.getOrCompute('keyA', 'hash1', computeFn);
|
|
assert.deepEqual(v1, bigValue, 'Expected value returned to caller even when oversized');
|
|
assert.equal(computeCallCount, 1);
|
|
|
|
// Verify the value was NOT cached (get returns null)
|
|
const entry = await store.get('keyA', 'hash1');
|
|
assert.equal(entry, null, 'Oversized value must not be stored in cache');
|
|
|
|
// Second call — must recompute (cache miss, because oversized skipped storage)
|
|
const v2 = await store.getOrCompute('keyA', 'hash1', computeFn);
|
|
assert.deepEqual(v2, bigValue);
|
|
assert.equal(computeCallCount, 2, 'computeFn must be called again because oversized value was not cached');
|
|
|
|
// Warn should have fired twice (once per set() call from the two getOrCompute calls)
|
|
assert.ok(warnCalls.length >= 2, `Expected at least 2 warn calls, got ${warnCalls.length}`);
|
|
assert.ok(warnCalls.every(w => w.msg === 'cache_skip_oversize'));
|
|
});
|
|
|
|
// ── D39 (issue #3 Part 1): CacheStore.delete API ─────────────────────
|
|
// Authority: ADR 0005 § "Cache write conditions" item 1; D39 design note in
|
|
// store.mjs. Replaces the prior `set(..., ttlMs=0)` tombstone pattern with
|
|
// an explicit immediate-eviction primitive.
|
|
|
|
it('D39: CacheStore.delete returns true and removes entry when present', async () => {
|
|
const store = new CacheStore();
|
|
await store.set('keyA', 'hash1', [{ type: 'delta', content: 'partial' }]);
|
|
// Sanity: entry is there.
|
|
assert.ok(await store.peek('keyA', 'hash1'), 'precondition: entry must be present');
|
|
// Delete reports true.
|
|
assert.equal(store.delete('keyA', 'hash1'), true);
|
|
// Subsequent peek/get return false/null without lazy-purge side-effects.
|
|
assert.equal(await store.peek('keyA', 'hash1'), false, 'peek must be false after delete');
|
|
assert.equal(await store.get('keyA', 'hash1'), null, 'get must return null after delete');
|
|
// getOrCompute on the same key now triggers a FRESH compute (not served from prior entry).
|
|
let computeCount = 0;
|
|
const v = await store.getOrCompute('keyA', 'hash1', async () => {
|
|
computeCount++;
|
|
return [{ type: 'delta', content: 'fresh' }];
|
|
});
|
|
assert.equal(computeCount, 1, 'getOrCompute must invoke computeFn (cache was deleted)');
|
|
assert.deepEqual(v, [{ type: 'delta', content: 'fresh' }]);
|
|
});
|
|
|
|
it('D39: CacheStore.delete returns false for absent entry (no throw)', () => {
|
|
const store = new CacheStore();
|
|
// Namespace does not exist at all.
|
|
assert.equal(store.delete('keyA', 'hash-missing'), false);
|
|
// Namespace exists but cacheKey absent — populate then delete one absent key.
|
|
// (set is async; await via Promise.resolve for the precondition setup.)
|
|
return store.set('keyA', 'hash1', 'v').then(() => {
|
|
assert.equal(store.delete('keyA', 'hash-different'), false,
|
|
'delete on absent cacheKey within existing namespace returns false');
|
|
// Existing entry untouched.
|
|
return store.peek('keyA', 'hash1').then((present) => {
|
|
assert.equal(present, true, 'unrelated entry must remain after a false delete');
|
|
});
|
|
});
|
|
});
|
|
|
|
it('D39: CacheStore.delete drops empty namespace Map entry (memory hygiene)', async () => {
|
|
const store = new CacheStore();
|
|
await store.set('keyA', 'only-hash', 'val');
|
|
// The internal namespace Map for keyA must exist after set.
|
|
assert.ok(store._store.has('keyA'), 'precondition: namespace exists in _store');
|
|
// Delete the only entry.
|
|
assert.equal(store.delete('keyA', 'only-hash'), true);
|
|
// Namespace Map entry must also be removed from the outer _store.
|
|
assert.equal(store._store.has('keyA'), false,
|
|
'empty namespace must be removed from _store after deleting last entry');
|
|
// A namespace with multiple entries must NOT be removed when only one is deleted.
|
|
await store.set('keyB', 'hash1', 'val1');
|
|
await store.set('keyB', 'hash2', 'val2');
|
|
assert.equal(store.delete('keyB', 'hash1'), true);
|
|
assert.ok(store._store.has('keyB'),
|
|
'non-empty namespace must remain in _store after partial delete');
|
|
assert.equal(store._store.get('keyB').size, 1, 'remaining entry count must be 1');
|
|
});
|
|
|
|
// ── Test 31: clear(keyId) clears only that namespace (renumbered from old T26) ──
|
|
it('CacheStore.clear(keyId) clears only that namespace', async () => {
|
|
const store = new CacheStore();
|
|
await store.set('keyA', 'hash1', 'val-a');
|
|
await store.set('keyB', 'hash1', 'val-b');
|
|
store.clear('keyA');
|
|
assert.equal(await store.has('keyA', 'hash1'), false, 'keyA should be cleared');
|
|
assert.equal(await store.has('keyB', 'hash1'), true, 'keyB should remain');
|
|
});
|
|
|
|
// ── Test 32: clear() with no args clears all (renumbered from old T27) ──
|
|
it('CacheStore.clear() with no argument clears all namespaces', async () => {
|
|
const store = new CacheStore();
|
|
await store.set('keyA', 'hash1', 'val-a');
|
|
await store.set('keyB', 'hash1', 'val-b');
|
|
store.clear();
|
|
assert.equal(await store.has('keyA', 'hash1'), false, 'keyA should be cleared');
|
|
assert.equal(await store.has('keyB', 'hash1'), false, 'keyB should be cleared');
|
|
assert.equal(store.stats().size, 0);
|
|
});
|
|
});
|
|
|
|
// ── Suite 9 (HTTP integration — cache) ───────────────────────────────────
|
|
//
|
|
// Tests cache miss / hit / bypass paths via HTTP integration with a mock
|
|
// provider. Anthropic provider is enabled with:
|
|
// 1. CLAUDE_CODE_OAUTH_TOKEN set to a fake value to bypass auth check
|
|
// (the mock spawn never actually uses the token)
|
|
// 2. Mock spawn injected via __setSpawnImpl so no real claude binary runs
|
|
//
|
|
// This tests the full HTTP → server.mjs → cache layer → provider dispatch
|
|
// path end-to-end, with the spawn binary call itself mocked out.
|
|
|
|
describe('Cache layer — HTTP integration (Suite 9 cont.)', () => {
|
|
let serverInstance9;
|
|
let port9;
|
|
let savedOAuthToken;
|
|
let serverMod9;
|
|
|
|
before(async () => {
|
|
// Inject a fake OAuth token so auth check passes without a real token.
|
|
// The mock spawn ignores this value entirely.
|
|
savedOAuthToken = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'test-fake-oauth-token-for-cache-tests';
|
|
|
|
// Set mock spawn that returns a proper response (delta + stop chunks via
|
|
// raw text output — anthropic.mjs treats each stdout data chunk as raw text).
|
|
__setSpawnImpl(makeMockSpawn(['mock-cache-content']));
|
|
|
|
// Import the server module (already cached by Node module system — same instance
|
|
// as Suite 7). Mutate the loadedProviders map to add anthropic.
|
|
serverMod9 = await import('./server.mjs');
|
|
const { createOlpServer, loadedProviders: lp } = serverMod9;
|
|
|
|
// Wire anthropic into the loaded providers map
|
|
const testProviders = loadProviders({ enabled: { anthropic: true } });
|
|
for (const [name, p] of testProviders) {
|
|
lp.set(name, p);
|
|
}
|
|
|
|
port9 = parseInt(
|
|
process.env.OLP_TEST_PORT
|
|
? String(parseInt(process.env.OLP_TEST_PORT) + 5000)
|
|
: String(18456 + Math.floor(Math.random() * 1000)),
|
|
10,
|
|
);
|
|
|
|
serverInstance9 = createOlpServer();
|
|
await new Promise((resolve, reject) => {
|
|
serverInstance9.listen(port9, '127.0.0.1', resolve);
|
|
serverInstance9.once('error', async (e) => {
|
|
if (e.code === 'EADDRINUSE') {
|
|
port9++;
|
|
serverInstance9.listen(port9, '127.0.0.1', resolve);
|
|
serverInstance9.once('error', reject);
|
|
} else reject(e);
|
|
});
|
|
});
|
|
});
|
|
|
|
after(() => {
|
|
// Restore OAuth token
|
|
if (savedOAuthToken !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedOAuthToken;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
__resetSpawnImpl();
|
|
return new Promise(r => serverInstance9.close(r));
|
|
});
|
|
|
|
// ── Test 28: cache miss path ──────────────────────────────────────────
|
|
// First request with a unique message → cache miss (not yet in cache).
|
|
it('HTTP: first request returns X-OLP-Cache: miss', async () => {
|
|
// Unique content ensures this test doesn't collide with other tests' cached entries
|
|
const testMsg = `http-cache-miss-${Date.now()}-${Math.random().toString(36).slice(2)}`;
|
|
__setSpawnImpl(makeMockSpawn([`response-for-${testMsg}`]));
|
|
|
|
const r = await fetch({
|
|
port: port9,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-haiku-4-5',
|
|
messages: [{ role: 'user', content: testMsg }],
|
|
},
|
|
});
|
|
|
|
assert.equal(r.status, 200, `Expected 200, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
assert.equal(r.headers['x-olp-cache'], 'miss', `Expected miss, got: ${r.headers['x-olp-cache']}`);
|
|
assert.equal(r.headers['x-olp-provider-used'], 'anthropic');
|
|
assert.equal(r.headers['x-olp-model-used'], 'claude-haiku-4-5');
|
|
});
|
|
|
|
// ── Test 29: cache hit path ───────────────────────────────────────────
|
|
// Two identical requests: first → miss, second → hit (same content served from cache).
|
|
it('HTTP: second identical request returns X-OLP-Cache: hit with same content', async () => {
|
|
const testMsg = `http-cache-hit-${Date.now()}-${Math.random().toString(36).slice(2)}`;
|
|
const mockResponse = `hit-response-${testMsg}`;
|
|
__setSpawnImpl(makeMockSpawn([mockResponse]));
|
|
|
|
const reqParams = {
|
|
port: port9,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-haiku-4-5',
|
|
messages: [{ role: 'user', content: testMsg }],
|
|
},
|
|
};
|
|
|
|
// First request — miss, spawns real (mock) provider
|
|
const r1 = await fetch(reqParams);
|
|
assert.equal(r1.status, 200, `First request failed: ${r1.status} ${r1.body.slice(0, 200)}`);
|
|
assert.equal(r1.headers['x-olp-cache'], 'miss', `First request should be miss`);
|
|
const body1 = JSON.parse(r1.body);
|
|
const content1 = body1?.choices?.[0]?.message?.content ?? '';
|
|
|
|
// Second request — replace spawn with a failing mock to prove spawn is NOT called
|
|
// (if spawn were called, this would produce a 502 error)
|
|
__setSpawnImpl(makeMockSpawn([], 1)); // exit code 1 = ProviderError on spawn
|
|
|
|
const r2 = await fetch(reqParams);
|
|
assert.equal(r2.status, 200, `Second request (cache hit) should be 200, got ${r2.status}: ${r2.body.slice(0, 200)}`);
|
|
assert.equal(r2.headers['x-olp-cache'], 'hit', `Second request should be cache hit`);
|
|
|
|
// Content should be identical (replayed from cache)
|
|
const body2 = JSON.parse(r2.body);
|
|
const content2 = body2?.choices?.[0]?.message?.content ?? '';
|
|
assert.equal(content2, content1, `Cache hit content should match original`);
|
|
});
|
|
|
|
// ── Test 30: cache bypass path (cache_control marker) ────────────────
|
|
// Request with cache_control marker → X-OLP-Cache: bypass (no OLP caching).
|
|
it('HTTP: request with cache_control marker returns X-OLP-Cache: bypass', async () => {
|
|
const testMsg = `http-cache-bypass-${Date.now()}-${Math.random().toString(36).slice(2)}`;
|
|
__setSpawnImpl(makeMockSpawn([`bypass-response-${testMsg}`]));
|
|
|
|
const r = await fetch({
|
|
port: port9,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-haiku-4-5',
|
|
messages: [
|
|
{
|
|
role: 'user',
|
|
content: testMsg,
|
|
cache_control: { type: 'ephemeral' },
|
|
},
|
|
],
|
|
},
|
|
});
|
|
|
|
assert.equal(r.status, 200, `Expected 200, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
assert.equal(r.headers['x-olp-cache'], 'bypass', `Expected bypass header, got: ${r.headers['x-olp-cache']}`);
|
|
});
|
|
});
|
|
|
|
// ── Suite 9e: D23 cacheable: false opt-out integration ───────────────────
|
|
//
|
|
// Verifies that a provider with hints.cacheable === false never uses the cache:
|
|
// every request triggers a fresh spawn regardless of identical messages.
|
|
//
|
|
// Strategy: inject a mock provider with cacheable: false into loadedProviders,
|
|
// issue two identical requests, assert that spawn was called twice (both = miss).
|
|
// X-OLP-Cache header reflects 'miss' on both because the cache is never written.
|
|
|
|
describe('D23 — cacheable: false opt-out integration (Suite 9e)', () => {
|
|
let serverInstance9e;
|
|
let port9e;
|
|
let savedOAuthToken9e;
|
|
let serverMod9e;
|
|
|
|
// Track how many times the mock spawn is called
|
|
let spawnCallCount;
|
|
|
|
function makeCountingMockSpawn(textChunks) {
|
|
return function mockSpawn(_bin, _args, _opts) {
|
|
spawnCallCount++;
|
|
return makeMockSpawn(textChunks)(_bin, _args, _opts);
|
|
};
|
|
}
|
|
|
|
before(async () => {
|
|
savedOAuthToken9e = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'test-fake-oauth-for-9e';
|
|
|
|
spawnCallCount = 0;
|
|
__setSpawnImpl(makeCountingMockSpawn(['cacheable-false-response']));
|
|
|
|
serverMod9e = await import('./server.mjs');
|
|
const { createOlpServer, loadedProviders: lp } = serverMod9e;
|
|
|
|
// Inject anthropic with cacheable: false (overriding the real plugin's cacheable: true).
|
|
// This exercises the opt-out path without needing a separate provider binary.
|
|
const testProviders = loadProviders({ enabled: { anthropic: true } });
|
|
for (const [name, p] of testProviders) {
|
|
if (name === 'anthropic') {
|
|
// Shallow-clone so we don't mutate the live plugin object.
|
|
lp.set(name, { ...p, hints: { ...p.hints, cacheable: false } });
|
|
} else {
|
|
lp.set(name, p);
|
|
}
|
|
}
|
|
|
|
port9e = parseInt(
|
|
process.env.OLP_TEST_PORT
|
|
? String(parseInt(process.env.OLP_TEST_PORT) + 6000)
|
|
: String(19456 + Math.floor(Math.random() * 1000)),
|
|
10,
|
|
);
|
|
|
|
serverInstance9e = createOlpServer();
|
|
await new Promise((resolve, reject) => {
|
|
serverInstance9e.listen(port9e, '127.0.0.1', resolve);
|
|
serverInstance9e.once('error', async (e) => {
|
|
if (e.code === 'EADDRINUSE') {
|
|
port9e++;
|
|
serverInstance9e.listen(port9e, '127.0.0.1', resolve);
|
|
serverInstance9e.once('error', reject);
|
|
} else reject(e);
|
|
});
|
|
});
|
|
});
|
|
|
|
after(() => {
|
|
if (savedOAuthToken9e !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedOAuthToken9e;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
__resetSpawnImpl();
|
|
return new Promise(r => serverInstance9e.close(r));
|
|
});
|
|
|
|
it('cacheable: false — both requests trigger fresh spawn (spawn called twice)', async () => {
|
|
const testMsg = `cacheable-false-test-${Date.now()}-${Math.random().toString(36).slice(2)}`;
|
|
spawnCallCount = 0;
|
|
|
|
const reqBody = {
|
|
model: 'claude-haiku-4-5',
|
|
messages: [{ role: 'user', content: testMsg }],
|
|
};
|
|
|
|
// First request
|
|
const r1 = await fetch({
|
|
port: port9e,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: reqBody,
|
|
});
|
|
assert.equal(r1.status, 200, `First request failed: ${r1.status} ${r1.body.slice(0, 200)}`);
|
|
|
|
// Second identical request — must also trigger spawn (cache opt-out)
|
|
__setSpawnImpl(makeCountingMockSpawn(['cacheable-false-response']));
|
|
const r2 = await fetch({
|
|
port: port9e,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: reqBody,
|
|
});
|
|
assert.equal(r2.status, 200, `Second request failed: ${r2.status} ${r2.body.slice(0, 200)}`);
|
|
|
|
// Both requests must have invoked spawn (cache never served).
|
|
// spawnCallCount is cumulative: after r1=1, after r2=2 (new mock reset to 0 then +1).
|
|
// Actually since we reset the mock between r1 and r2, count is 1 after each.
|
|
// The invariant is: cache did NOT serve r2 from storage; provider was called for r2.
|
|
assert.ok(spawnCallCount >= 1,
|
|
`Expected spawn to be called for second request (got spawnCallCount=${spawnCallCount} after r2)`);
|
|
});
|
|
|
|
it('cacheable: false — X-OLP-Cache header is miss on both requests (not hit)', async () => {
|
|
const testMsg = `cacheable-false-header-${Date.now()}-${Math.random().toString(36).slice(2)}`;
|
|
spawnCallCount = 0;
|
|
__setSpawnImpl(makeCountingMockSpawn(['response-for-header-test']));
|
|
|
|
const reqBody = {
|
|
model: 'claude-haiku-4-5',
|
|
messages: [{ role: 'user', content: testMsg }],
|
|
};
|
|
|
|
const r1 = await fetch({ port: port9e, method: 'POST', path: '/v1/chat/completions', body: reqBody });
|
|
assert.equal(r1.status, 200, `First request: ${r1.status} ${r1.body.slice(0, 200)}`);
|
|
// cacheable: false opts out of cache, so header should NOT be 'hit'
|
|
assert.notEqual(r1.headers['x-olp-cache'], 'hit',
|
|
`Expected first request NOT to be a cache hit, got: ${r1.headers['x-olp-cache']}`);
|
|
|
|
__setSpawnImpl(makeCountingMockSpawn(['response-for-header-test']));
|
|
const r2 = await fetch({ port: port9e, method: 'POST', path: '/v1/chat/completions', body: reqBody });
|
|
assert.equal(r2.status, 200, `Second request: ${r2.status} ${r2.body.slice(0, 200)}`);
|
|
assert.notEqual(r2.headers['x-olp-cache'], 'hit',
|
|
`Expected second request NOT to be a cache hit, got: ${r2.headers['x-olp-cache']}`);
|
|
});
|
|
|
|
it('cacheable: false + stream: true — both requests trigger fresh spawn; X-OLP-Cache is miss on both', async () => {
|
|
// D23 real-streaming branch fix: a cacheable: false provider with stream: true
|
|
// must NOT enter the D10 real-streaming path (which would write to cache).
|
|
// Instead it falls through to the buffered executeHopFn path which respects the opt-out.
|
|
// Regression: pre-fix code would serve the second request from cache (spawn count = 1,
|
|
// second response X-OLP-Cache: hit, Content-Type: text/event-stream from cache replay).
|
|
const testMsg = `cacheable-false-stream-${Date.now()}-${Math.random().toString(36).slice(2)}`;
|
|
|
|
const reqBody = {
|
|
model: 'claude-haiku-4-5',
|
|
messages: [{ role: 'user', content: testMsg }],
|
|
stream: true,
|
|
};
|
|
|
|
// First streaming request
|
|
spawnCallCount = 0;
|
|
__setSpawnImpl(makeCountingMockSpawn(['stream-chunk-r1']));
|
|
const r1 = await fetch({ port: port9e, method: 'POST', path: '/v1/chat/completions', body: reqBody });
|
|
assert.equal(r1.status, 200, `First streaming request failed: ${r1.status} ${r1.body.slice(0, 200)}`);
|
|
assert.ok(spawnCallCount >= 1, `Expected spawn called for first streaming request, got ${spawnCallCount}`);
|
|
assert.notEqual(r1.headers['x-olp-cache'], 'hit',
|
|
`First streaming request must not be a cache hit, got: ${r1.headers['x-olp-cache']}`);
|
|
|
|
// Second identical streaming request — must also trigger spawn (no cache write after r1)
|
|
spawnCallCount = 0;
|
|
__setSpawnImpl(makeCountingMockSpawn(['stream-chunk-r2']));
|
|
const r2 = await fetch({ port: port9e, method: 'POST', path: '/v1/chat/completions', body: reqBody });
|
|
assert.equal(r2.status, 200, `Second streaming request failed: ${r2.status} ${r2.body.slice(0, 200)}`);
|
|
assert.ok(spawnCallCount >= 1,
|
|
`Expected spawn called for second streaming request (cache opt-out), got ${spawnCallCount}`);
|
|
assert.notEqual(r2.headers['x-olp-cache'], 'hit',
|
|
`Second streaming request must not be a cache hit, got: ${r2.headers['x-olp-cache']}`);
|
|
});
|
|
});
|
|
|
|
// ── Suite 9d: D13 cache_control per-hop bypass correctness ───────────────
|
|
//
|
|
// D13 (ADR 0005 § D2): cache_control markers bypass OLP's response cache ONLY
|
|
// when the active hop provider is Anthropic. For non-Anthropic providers the
|
|
// markers are noop'd — the hop IS cached normally.
|
|
//
|
|
// Tests:
|
|
// Test 31: cache_control + non-Anthropic provider (codex) → X-OLP-Cache: miss
|
|
// (NOT bypass — fix validates the defect is corrected)
|
|
// Test 32: cache_control + Anthropic provider → X-OLP-Cache: bypass
|
|
// (existing behaviour preserved — regression guard)
|
|
// Test 33: cache_control + 2-hop chain (anthropic→openai):
|
|
// anthropic hop bypass fires; if anthropic fails, openai hop is NOT bypassed
|
|
|
|
describe('D13 — cache_control per-hop bypass correctness (Suite 9d)', () => {
|
|
let serverD13;
|
|
let portD13;
|
|
let savedAnthropicTokenD13;
|
|
let savedCodexAuthPathD13;
|
|
let suiteCodexAuthFileD13;
|
|
|
|
before(async () => {
|
|
// Inject fake auth for both providers so auth checks pass without real tokens.
|
|
savedAnthropicTokenD13 = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-anthropic-token-for-d13-suite';
|
|
|
|
const { writeFileSync } = await import('node:fs');
|
|
const { tmpdir } = await import('node:os');
|
|
const { join: pathJoin } = await import('node:path');
|
|
suiteCodexAuthFileD13 = pathJoin(tmpdir(), `olp-test-d13-codex-auth-${Date.now()}.json`);
|
|
writeFileSync(suiteCodexAuthFileD13, JSON.stringify({ accessToken: 'fake-codex-token-for-d13-suite' }), 'utf8');
|
|
savedCodexAuthPathD13 = process.env.OPENAI_CODEX_AUTH_PATH;
|
|
process.env.OPENAI_CODEX_AUTH_PATH = suiteCodexAuthFileD13;
|
|
|
|
// Wire both anthropic and openai providers into the shared loadedProviders map.
|
|
const testProviders = loadProviders({ enabled: { anthropic: true, openai: true } });
|
|
const { loadedProviders: lp, cacheStore: cs } = await import('./server.mjs');
|
|
for (const [name, p] of testProviders) {
|
|
lp.set(name, p);
|
|
}
|
|
cs.clear();
|
|
|
|
serverD13 = createOlpServer();
|
|
portD13 = 21456 + Math.floor(Math.random() * 500);
|
|
await new Promise((resolve, reject) => {
|
|
serverD13.listen(portD13, '127.0.0.1', resolve);
|
|
serverD13.once('error', (e) => {
|
|
if (e.code === 'EADDRINUSE') {
|
|
portD13++;
|
|
serverD13.listen(portD13, '127.0.0.1', resolve);
|
|
serverD13.once('error', reject);
|
|
} else reject(e);
|
|
});
|
|
});
|
|
});
|
|
|
|
after(async () => {
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
|
|
if (savedAnthropicTokenD13 !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedAnthropicTokenD13;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
if (savedCodexAuthPathD13 !== undefined) {
|
|
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexAuthPathD13;
|
|
} else {
|
|
delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
}
|
|
if (suiteCodexAuthFileD13) {
|
|
const { unlinkSync } = await import('node:fs');
|
|
try { unlinkSync(suiteCodexAuthFileD13); } catch { /* ignore */ }
|
|
}
|
|
|
|
if (!serverD13) return;
|
|
return new Promise(r => serverD13.close(r));
|
|
});
|
|
|
|
// ── Test 31: cache_control + non-Anthropic provider → NOT bypass ──────
|
|
// D13 fix: a request to codex (openai) that carries cache_control markers
|
|
// must use OLP's response cache normally. Before D13, it incorrectly bypassed.
|
|
it('D13: cache_control + openai provider → X-OLP-Cache: miss (NOT bypass)', async () => {
|
|
const testMsg = `d13-codex-no-bypass-${Date.now()}-${Math.random().toString(36).slice(2)}`;
|
|
// Inject a codex mock that returns a valid NDJSON stop event
|
|
codexSetSpawnImpl(makeMockCodexSpawn([
|
|
JSON.stringify({ content: `codex-response-${testMsg}` }),
|
|
JSON.stringify({ type: 'stop' }),
|
|
]));
|
|
|
|
const r = await fetch({
|
|
port: portD13,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'gpt-5.5',
|
|
messages: [
|
|
{
|
|
role: 'user',
|
|
content: testMsg,
|
|
cache_control: { type: 'ephemeral' },
|
|
},
|
|
],
|
|
},
|
|
});
|
|
|
|
assert.equal(r.status, 200, `Expected 200, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
// Key assertion: non-Anthropic hop with cache_control must be 'miss', NOT 'bypass'
|
|
assert.equal(
|
|
r.headers['x-olp-cache'],
|
|
'miss',
|
|
`D13: openai hop with cache_control should be cache miss (not bypass), got: ${r.headers['x-olp-cache']}`,
|
|
);
|
|
assert.equal(r.headers['x-olp-provider-used'], 'openai',
|
|
`Expected openai provider, got: ${r.headers['x-olp-provider-used']}`);
|
|
});
|
|
|
|
// ── Test 32: cache_control + Anthropic provider → bypass (regression guard)
|
|
// Verifies that the D13 per-hop logic still correctly bypasses for Anthropic.
|
|
it('D13: cache_control + anthropic provider → X-OLP-Cache: bypass (preserved)', async () => {
|
|
const testMsg = `d13-anthropic-bypass-${Date.now()}-${Math.random().toString(36).slice(2)}`;
|
|
__setSpawnImpl(makeMockSpawn([`anthropic-bypass-response-${testMsg}`]));
|
|
|
|
const r = await fetch({
|
|
port: portD13,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-haiku-4-5',
|
|
messages: [
|
|
{
|
|
role: 'user',
|
|
content: testMsg,
|
|
cache_control: { type: 'ephemeral' },
|
|
},
|
|
],
|
|
},
|
|
});
|
|
|
|
assert.equal(r.status, 200, `Expected 200, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
assert.equal(
|
|
r.headers['x-olp-cache'],
|
|
'bypass',
|
|
`D13: anthropic hop with cache_control should be bypass, got: ${r.headers['x-olp-cache']}`,
|
|
);
|
|
assert.equal(r.headers['x-olp-provider-used'], 'anthropic',
|
|
`Expected anthropic provider, got: ${r.headers['x-olp-provider-used']}`);
|
|
});
|
|
|
|
// ── Test 33: cache_control + 2-hop chain anthropic→openai ────────────
|
|
// When anthropic (hop 0) fails and falls over to openai (hop 1):
|
|
// the openai hop must NOT bypass cache even though cache_control markers
|
|
// were present in the original request.
|
|
it('D13: cache_control + fallback anthropic→openai → openai hop is NOT bypass', async () => {
|
|
const testMsg = `d13-fallback-${Date.now()}-${Math.random().toString(36).slice(2)}`;
|
|
|
|
// Wire a 2-hop fallback chain: anthropic → openai for the claude-haiku-4-5 model
|
|
const { __setFallbackConfig } = await import('./server.mjs');
|
|
__setFallbackConfig({
|
|
chains: {
|
|
'claude-haiku-4-5': [
|
|
{ provider: 'anthropic', model: 'claude-haiku-4-5' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
});
|
|
|
|
try {
|
|
// Anthropic spawn always fails with a hard trigger → chain advances to openai
|
|
__setSpawnImpl(makeMockSpawn([], 1)); // exit code 1 = SPAWN_FAILED hard trigger
|
|
// Codex/openai spawn succeeds
|
|
codexSetSpawnImpl(makeMockCodexSpawn([
|
|
JSON.stringify({ content: `fallback-codex-response-${testMsg}` }),
|
|
JSON.stringify({ type: 'stop' }),
|
|
]));
|
|
|
|
const r = await fetch({
|
|
port: portD13,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-haiku-4-5',
|
|
messages: [
|
|
{
|
|
role: 'user',
|
|
content: testMsg,
|
|
cache_control: { type: 'ephemeral' },
|
|
},
|
|
],
|
|
},
|
|
});
|
|
|
|
assert.equal(r.status, 200, `Expected 200, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
// The serving provider is openai (fallback hop 1)
|
|
assert.equal(r.headers['x-olp-provider-used'], 'openai',
|
|
`Expected openai as fallback provider, got: ${r.headers['x-olp-provider-used']}`);
|
|
// D13: openai serving hop must NOT be bypass even though cache_control was present
|
|
assert.notEqual(
|
|
r.headers['x-olp-cache'],
|
|
'bypass',
|
|
`D13: openai fallback hop with cache_control must NOT be bypass, got: ${r.headers['x-olp-cache']}`,
|
|
);
|
|
// It should be 'miss' (first time this openai key is seen)
|
|
assert.equal(
|
|
r.headers['x-olp-cache'],
|
|
'miss',
|
|
`D13: openai fallback hop should be cache miss, got: ${r.headers['x-olp-cache']}`,
|
|
);
|
|
} finally {
|
|
const { __resetFallbackConfig } = await import('./server.mjs');
|
|
__resetFallbackConfig();
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
}
|
|
});
|
|
});
|
|
|
|
// ── Suite 9f: D36 #2 cache_control partial-noop debug log ────────────────
|
|
//
|
|
// Authority: ADR 0005 § Context — "for non-Anthropic targets, the bypass
|
|
// markers are noop'd (logged once per request at debug level so users can
|
|
// see they were ignored)."
|
|
//
|
|
// Contract: at request entry in handleChatCompletions, after hasCacheControlMarkers
|
|
// is computed, emit ONE logEvent('debug', 'cache_control_partial_noop', { chain, marker_count })
|
|
// IF AND ONLY IF (a) markers are present AND (b) at least one chain hop is
|
|
// non-Anthropic. The log is suppressed when no markers, or when every hop is
|
|
// Anthropic.
|
|
//
|
|
// Implementation strategy: monkeypatch process.stdout.write (debug events go
|
|
// to stdout per the logEvent helper in server.mjs:56-63) and grep for the
|
|
// event name across the captured writes.
|
|
|
|
describe('D36 #2 — cache_control partial-noop debug log (Suite 9f)', () => {
|
|
let serverD36;
|
|
let portD36;
|
|
let savedAnthropicTokenD36;
|
|
let savedCodexAuthPathD36;
|
|
let suiteCodexAuthFileD36;
|
|
|
|
// ── stdout capture helpers ───────────────────────────────────────────
|
|
let stdoutWrites = [];
|
|
let origStdoutWrite = null;
|
|
|
|
function startStdoutCapture() {
|
|
stdoutWrites = [];
|
|
origStdoutWrite = process.stdout.write.bind(process.stdout);
|
|
process.stdout.write = (chunk, ...rest) => {
|
|
const s = typeof chunk === 'string'
|
|
? chunk
|
|
: (Buffer.isBuffer(chunk) ? chunk.toString() : String(chunk));
|
|
stdoutWrites.push(s);
|
|
return origStdoutWrite(chunk, ...rest);
|
|
};
|
|
}
|
|
|
|
function stopStdoutCapture() {
|
|
if (origStdoutWrite) {
|
|
process.stdout.write = origStdoutWrite;
|
|
origStdoutWrite = null;
|
|
}
|
|
}
|
|
|
|
/** Returns array of parsed JSON events whose `event` field === eventName. */
|
|
function findEvents(eventName) {
|
|
const found = [];
|
|
for (const w of stdoutWrites) {
|
|
for (const line of w.split('\n')) {
|
|
if (!line.trim()) continue;
|
|
if (!line.includes(`"event":"${eventName}"`)) continue;
|
|
try {
|
|
const parsed = JSON.parse(line);
|
|
if (parsed?.event === eventName) found.push(parsed);
|
|
} catch { /* not JSON — ignore */ }
|
|
}
|
|
}
|
|
return found;
|
|
}
|
|
|
|
before(async () => {
|
|
savedAnthropicTokenD36 = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-anthropic-token-for-d36-suite';
|
|
|
|
const { writeFileSync } = await import('node:fs');
|
|
const { tmpdir } = await import('node:os');
|
|
const { join: pathJoin } = await import('node:path');
|
|
suiteCodexAuthFileD36 = pathJoin(tmpdir(), `olp-test-d36-codex-auth-${Date.now()}.json`);
|
|
writeFileSync(suiteCodexAuthFileD36, JSON.stringify({ accessToken: 'fake-codex-token-for-d36-suite' }), 'utf8');
|
|
savedCodexAuthPathD36 = process.env.OPENAI_CODEX_AUTH_PATH;
|
|
process.env.OPENAI_CODEX_AUTH_PATH = suiteCodexAuthFileD36;
|
|
|
|
const testProviders = loadProviders({ enabled: { anthropic: true, openai: true } });
|
|
const { loadedProviders: lp, cacheStore: cs } = await import('./server.mjs');
|
|
for (const [name, p] of testProviders) {
|
|
lp.set(name, p);
|
|
}
|
|
cs.clear();
|
|
|
|
serverD36 = createOlpServer();
|
|
portD36 = 22000 + Math.floor(Math.random() * 500);
|
|
await new Promise((resolve, reject) => {
|
|
serverD36.listen(portD36, '127.0.0.1', resolve);
|
|
serverD36.once('error', (e) => {
|
|
if (e.code === 'EADDRINUSE') {
|
|
portD36++;
|
|
serverD36.listen(portD36, '127.0.0.1', resolve);
|
|
serverD36.once('error', reject);
|
|
} else reject(e);
|
|
});
|
|
});
|
|
});
|
|
|
|
after(async () => {
|
|
stopStdoutCapture();
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
|
|
if (savedAnthropicTokenD36 !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedAnthropicTokenD36;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
if (savedCodexAuthPathD36 !== undefined) {
|
|
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexAuthPathD36;
|
|
} else {
|
|
delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
}
|
|
if (suiteCodexAuthFileD36) {
|
|
const { unlinkSync } = await import('node:fs');
|
|
try { unlinkSync(suiteCodexAuthFileD36); } catch { /* ignore */ }
|
|
}
|
|
|
|
if (!serverD36) return;
|
|
return new Promise(r => serverD36.close(r));
|
|
});
|
|
|
|
it('D36 #2a: markers + non-Anthropic chain → cache_control_partial_noop log fires once', async () => {
|
|
// Route to openai (gpt-5.5) with cache_control markers present. Per ADR 0005
|
|
// § Context, the partial-noop log must fire because the chain contains a
|
|
// non-Anthropic hop and markers are present.
|
|
codexSetSpawnImpl(makeMockCodexSpawn([
|
|
JSON.stringify({ content: 'codex-out-d36-2a' }),
|
|
JSON.stringify({ type: 'stop' }),
|
|
]));
|
|
|
|
startStdoutCapture();
|
|
try {
|
|
const r = await fetch({
|
|
port: portD36,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'gpt-5.5',
|
|
messages: [
|
|
{ role: 'user', content: 'hello-d36-2a', cache_control: { type: 'ephemeral' } },
|
|
],
|
|
},
|
|
});
|
|
assert.equal(r.status, 200, `Expected 200, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
} finally {
|
|
stopStdoutCapture();
|
|
}
|
|
|
|
const events = findEvents('cache_control_partial_noop');
|
|
assert.equal(events.length, 1, `Expected exactly 1 cache_control_partial_noop event, got ${events.length}`);
|
|
const ev = events[0];
|
|
assert.equal(ev.level, 'debug', `event level should be 'debug', got '${ev.level}'`);
|
|
assert.ok(Array.isArray(ev.chain), 'chain must be an array');
|
|
assert.ok(ev.chain.includes('openai'),
|
|
`chain should include 'openai', got: ${JSON.stringify(ev.chain)}`);
|
|
assert.equal(typeof ev.marker_count, 'number', 'marker_count must be a number');
|
|
assert.ok(ev.marker_count >= 1, `marker_count must be >= 1, got ${ev.marker_count}`);
|
|
});
|
|
|
|
it('D36 #2b: no markers + non-Anthropic chain → cache_control_partial_noop does NOT fire', async () => {
|
|
codexSetSpawnImpl(makeMockCodexSpawn([
|
|
JSON.stringify({ content: 'codex-out-d36-2b' }),
|
|
JSON.stringify({ type: 'stop' }),
|
|
]));
|
|
|
|
startStdoutCapture();
|
|
try {
|
|
const r = await fetch({
|
|
port: portD36,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'gpt-5.5',
|
|
messages: [
|
|
{ role: 'user', content: 'hello-d36-2b' }, // no cache_control
|
|
],
|
|
},
|
|
});
|
|
assert.equal(r.status, 200, `Expected 200, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
} finally {
|
|
stopStdoutCapture();
|
|
}
|
|
|
|
const events = findEvents('cache_control_partial_noop');
|
|
assert.equal(events.length, 0,
|
|
`Expected 0 cache_control_partial_noop events when no markers; got ${events.length}: ${JSON.stringify(events)}`);
|
|
});
|
|
|
|
it('D36 #2c: markers + anthropic-only chain → cache_control_partial_noop does NOT fire', async () => {
|
|
// anthropic single-hop chain — every hop is anthropic, so the partial-noop
|
|
// log must be suppressed. The cache_bypass log (already documented) is the
|
|
// correct signal for this case, not partial_noop.
|
|
__setSpawnImpl(makeMockSpawn(['anthropic-out-d36-2c']));
|
|
|
|
startStdoutCapture();
|
|
try {
|
|
const r = await fetch({
|
|
port: portD36,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-haiku-4-5',
|
|
messages: [
|
|
{ role: 'user', content: 'hello-d36-2c', cache_control: { type: 'ephemeral' } },
|
|
],
|
|
},
|
|
});
|
|
assert.equal(r.status, 200, `Expected 200, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
} finally {
|
|
stopStdoutCapture();
|
|
}
|
|
|
|
const events = findEvents('cache_control_partial_noop');
|
|
assert.equal(events.length, 0,
|
|
`Expected 0 cache_control_partial_noop events when chain is anthropic-only; got ${events.length}: ${JSON.stringify(events)}`);
|
|
});
|
|
});
|
|
|
|
// ── Suite 11: Codex plugin (D6) ──────────────────────────────────────────
|
|
//
|
|
// All tests are UNIT tests. No real `codex` binary is invoked.
|
|
// Mock spawn is injected via codexSetSpawnImpl / codexResetSpawnImpl.
|
|
//
|
|
// Authority: Codex CLI reference https://developers.openai.com/codex/cli/reference
|
|
// § "codex exec [flags] PROMPT" — exec subcommand syntax
|
|
// § "--json" — NDJSON output format
|
|
// § "--model, -m" — model override
|
|
// § "$CODEX_HOME/auth.json" — auth artifact location
|
|
//
|
|
// Lossy-translation acknowledgements per ADR 0003 (documented in codex.mjs header):
|
|
// top_p, temperature, stop, max_tokens, tools[], tool_calls → all dropped.
|
|
|
|
/**
|
|
* Creates a fake NDJSON-emitting mock spawn for Codex.
|
|
* Lines are emitted as they would arrive from `codex exec --json`.
|
|
*
|
|
* @param {string[]} ndjsonLines — raw NDJSON lines emitted in order (no trailing \n needed)
|
|
* @param {number} [exitCode=0]
|
|
*/
|
|
function makeMockCodexSpawn(ndjsonLines, exitCode = 0) {
|
|
return function mockCodexSpawnImpl(_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(async () => {
|
|
for (const line of ndjsonLines) {
|
|
proc.stdout.emit('data', Buffer.from(line + '\n'));
|
|
}
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', exitCode, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
};
|
|
}
|
|
|
|
describe('Codex plugin (D6)', () => {
|
|
|
|
// ── Test 1: Contract conformance ─────────────────────────────────────
|
|
it('codex module satisfies validateProvider() — all 10 fields present', () => {
|
|
const { valid, errors } = validateProvider(codex);
|
|
assert.equal(valid, true, `Validation errors: ${errors.join('; ')}`);
|
|
assert.ok('name' in codex, 'missing: name');
|
|
assert.ok('displayName' in codex, 'missing: displayName');
|
|
assert.ok('contractVersion' in codex, 'missing: contractVersion');
|
|
assert.ok('models' in codex, 'missing: models');
|
|
assert.ok('auth' in codex, 'missing: auth');
|
|
assert.ok(typeof codex.spawn === 'function', 'missing: spawn');
|
|
assert.ok(typeof codex.estimateCost === 'function', 'missing: estimateCost');
|
|
assert.ok(typeof codex.quotaStatus === 'function', 'missing: quotaStatus');
|
|
assert.ok(typeof codex.healthCheck === 'function', 'missing: healthCheck');
|
|
assert.ok('hints' in codex, 'missing: hints');
|
|
});
|
|
|
|
// ── Test 2: contractVersion === '1.0' ────────────────────────────────
|
|
it('codex declares contractVersion === "1.0"', () => {
|
|
assert.equal(codex.contractVersion, '1.0');
|
|
});
|
|
|
|
// ── Test 3: name and displayName ─────────────────────────────────────
|
|
it('codex.name === "openai" and displayName is set', () => {
|
|
assert.equal(codex.name, 'openai');
|
|
assert.equal(typeof codex.displayName, 'string');
|
|
assert.ok(codex.displayName.length > 0);
|
|
});
|
|
|
|
// ── Test 4: models match registry ───────────────────────────────────
|
|
it('codex.models matches models-registry.json providers.openai.models[].id', () => {
|
|
const registryIds = modelsRegistry.providers.openai.models.map(m => m.id);
|
|
assert.deepEqual(codex.models, registryIds);
|
|
});
|
|
|
|
it('codex.models contains all 5 docs-listed model IDs', () => {
|
|
// Per https://developers.openai.com/codex/models — each id has a
|
|
// `codex -m <id>` example on that page. D6 review-2 expanded the
|
|
// registry to include gpt-5.4-mini and gpt-5.3-codex-spark which the
|
|
// original sonnet draft missed.
|
|
assert.ok(codex.models.includes('gpt-5.5'), 'missing gpt-5.5');
|
|
assert.ok(codex.models.includes('gpt-5.4'), 'missing gpt-5.4');
|
|
assert.ok(codex.models.includes('gpt-5.4-mini'), 'missing gpt-5.4-mini');
|
|
assert.ok(codex.models.includes('gpt-5.3-codex'), 'missing gpt-5.3-codex');
|
|
assert.ok(codex.models.includes('gpt-5.3-codex-spark'), 'missing gpt-5.3-codex-spark');
|
|
assert.equal(codex.models.length, 5, `Expected 5 models, got ${codex.models.length}`);
|
|
});
|
|
|
|
// ── Test 5: getProviderForModel finds codex for each model ──────────
|
|
it('getProviderForModel finds openai provider for gpt-5.5', () => {
|
|
const loaded = new Map([['openai', codex]]);
|
|
const result = getProviderForModel(loaded, 'gpt-5.5');
|
|
assert.ok(result !== null);
|
|
assert.equal(result.name, 'openai');
|
|
});
|
|
|
|
it('getProviderForModel finds openai provider for gpt-5.4', () => {
|
|
const loaded = new Map([['openai', codex]]);
|
|
const result = getProviderForModel(loaded, 'gpt-5.4');
|
|
assert.ok(result !== null);
|
|
assert.equal(result.name, 'openai');
|
|
});
|
|
|
|
it('getProviderForModel finds openai provider for gpt-5.3-codex', () => {
|
|
const loaded = new Map([['openai', codex]]);
|
|
const result = getProviderForModel(loaded, 'gpt-5.3-codex');
|
|
assert.ok(result !== null);
|
|
assert.equal(result.name, 'openai');
|
|
});
|
|
|
|
// ── Test 6: irToCodex translation ────────────────────────────────────
|
|
it('irToCodex: user message → args with exec --json --model, prompt as positional', () => {
|
|
const ir = makeIR({
|
|
model: 'gpt-5.5',
|
|
messages: [{ role: 'user', content: 'Hello world' }],
|
|
});
|
|
const { args, prompt, useStdin } = irToCodex(ir);
|
|
// Authority: Codex CLI reference § "codex exec [flags] PROMPT"
|
|
assert.ok(args.includes('exec'), 'args must include "exec"');
|
|
assert.ok(args.includes('--json'), 'args must include "--json"');
|
|
assert.ok(args.includes('--model'), 'args must include "--model"');
|
|
assert.ok(args.includes('gpt-5.5'), 'args must include model value');
|
|
assert.ok(typeof prompt === 'string', 'prompt must be a string');
|
|
assert.ok(prompt.includes('Hello world'), 'prompt must contain user text');
|
|
assert.equal(useStdin, false, 'single-line prompt should use argv, not stdin');
|
|
});
|
|
|
|
it('irToCodex: system + user → system annotation + user text in prompt', () => {
|
|
const ir = makeIR({
|
|
model: 'gpt-5.4',
|
|
messages: [
|
|
{ role: 'system', content: 'You are a coder.' },
|
|
{ role: 'user', content: 'Write a function.' },
|
|
],
|
|
});
|
|
const { prompt } = irToCodex(ir);
|
|
assert.ok(prompt.includes('[System] You are a coder.'));
|
|
assert.ok(prompt.includes('Write a function.'));
|
|
});
|
|
|
|
it('irToCodex: assistant prior turn → [Assistant] annotation', () => {
|
|
const ir = makeIR({
|
|
model: 'gpt-5.5',
|
|
messages: [
|
|
{ role: 'user', content: 'Hi' },
|
|
{ role: 'assistant', content: 'Hello!' },
|
|
{ role: 'user', content: 'Thanks' },
|
|
],
|
|
});
|
|
const { prompt } = irToCodex(ir);
|
|
assert.ok(prompt.includes('[Assistant] Hello!'));
|
|
assert.ok(prompt.includes('Thanks'));
|
|
});
|
|
|
|
it('irToCodex: tool_calls in assistant message — content preserved, metadata dropped (lossy)', () => {
|
|
// ADR 0003 § Lossy: structured tool_calls dropped; textual content preserved.
|
|
const ir = makeIR({
|
|
model: 'gpt-5.5',
|
|
messages: [
|
|
{ role: 'user', content: 'Search for X' },
|
|
{
|
|
role: 'assistant',
|
|
content: 'Searching...',
|
|
tool_calls: [{ id: 'tc1', type: 'function', function: { name: 'search', arguments: '{"q":"X"}' } }],
|
|
},
|
|
],
|
|
});
|
|
const { prompt } = irToCodex(ir);
|
|
// Content preserved
|
|
assert.ok(prompt.includes('Searching...'), 'assistant text content must be preserved');
|
|
// tool_calls metadata not directly in prompt (dropped per lossy spec)
|
|
// (We do NOT assert the metadata IS there — it is documented as dropped.)
|
|
});
|
|
|
|
it('irToCodex: response_format json_object injects system prompt', () => {
|
|
const ir = makeIR({
|
|
model: 'gpt-5.5',
|
|
messages: [{ role: 'user', content: 'Give JSON' }],
|
|
response_format: { type: 'json_object' },
|
|
});
|
|
const { prompt } = irToCodex(ir);
|
|
assert.ok(prompt.includes('Reply with valid JSON only'));
|
|
});
|
|
|
|
it('irToCodex: multiline prompt uses stdin path (useStdin=true) with `-` positional', () => {
|
|
const ir = makeIR({
|
|
model: 'gpt-5.5',
|
|
messages: [
|
|
{ role: 'system', content: 'Line one.' },
|
|
{ role: 'user', content: 'Line two.' },
|
|
],
|
|
});
|
|
const { useStdin, args } = irToCodex(ir);
|
|
// System + user messages join with '\n\n' → contains newline → stdin
|
|
assert.equal(useStdin, true, 'multi-section prompt should use stdin');
|
|
// Per Codex CLI reference, stdin requires the literal `-` positional.
|
|
// (D6 review-2 finding: original draft omitted positional entirely;
|
|
// docs explicitly state stdin requires `-`.)
|
|
assert.ok(args.includes('-'), 'stdin path must pass `-` as positional');
|
|
assert.ok(!args.includes('Line one.'), 'literal prompt must not appear in args when useStdin');
|
|
});
|
|
|
|
it('irToCodex: tool result turn → [Tool Result] annotation', () => {
|
|
const ir = makeIR({
|
|
model: 'gpt-5.5',
|
|
messages: [
|
|
{ role: 'user', content: 'Search for X' },
|
|
{ role: 'tool', content: '{"results":[]}', name: 'search' },
|
|
],
|
|
});
|
|
const { prompt } = irToCodex(ir);
|
|
assert.ok(prompt.includes('[Tool Result'));
|
|
assert.ok(prompt.includes('search'));
|
|
});
|
|
|
|
// ── Test 7: codexChunkToIR translation ────────────────────────────────
|
|
it('codexChunkToIR: content field → delta chunk', () => {
|
|
const chunk = codexChunkToIR('{"content":"Hello world"}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'delta');
|
|
assert.equal(chunk.content, 'Hello world');
|
|
});
|
|
|
|
it('codexChunkToIR: delta field → delta chunk', () => {
|
|
const chunk = codexChunkToIR('{"type":"delta","delta":"token text"}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'delta');
|
|
assert.equal(chunk.content, 'token text');
|
|
});
|
|
|
|
it('codexChunkToIR: text field → delta chunk', () => {
|
|
// Possible alternative event shape with "text" field
|
|
const chunk = codexChunkToIR('{"type":"output_text","text":"output here"}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'delta');
|
|
assert.equal(chunk.content, 'output here');
|
|
});
|
|
|
|
it('codexChunkToIR: type === "stop" → stop chunk', () => {
|
|
const chunk = codexChunkToIR('{"type":"stop"}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'stop');
|
|
assert.equal(chunk.finish_reason, 'stop');
|
|
});
|
|
|
|
it('codexChunkToIR: done === true → stop chunk', () => {
|
|
const chunk = codexChunkToIR('{"done":true,"id":"run_123"}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'stop');
|
|
assert.equal(chunk.finish_reason, 'stop');
|
|
});
|
|
|
|
it('codexChunkToIR: type === "error" → error chunk', () => {
|
|
const chunk = codexChunkToIR('{"type":"error","error":"quota exceeded"}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'error');
|
|
assert.equal(chunk.error, 'quota exceeded');
|
|
});
|
|
|
|
it('codexChunkToIR: error field present → error chunk', () => {
|
|
const chunk = codexChunkToIR('{"error":"something went wrong","code":500}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'error');
|
|
});
|
|
|
|
it('codexChunkToIR: unknown/progress event type → null (silently ignored)', () => {
|
|
// Events like {"type":"progress","step":1} are ignored per D6 spec
|
|
const chunk = codexChunkToIR('{"type":"progress","step":1}');
|
|
assert.equal(chunk, null);
|
|
});
|
|
|
|
it('codexChunkToIR: empty line → null', () => {
|
|
assert.equal(codexChunkToIR(''), null);
|
|
assert.equal(codexChunkToIR(' '), null);
|
|
});
|
|
|
|
it('codexChunkToIR: malformed JSON → null (no throw)', () => {
|
|
assert.doesNotThrow(() => {
|
|
const result = codexChunkToIR('{bad json');
|
|
assert.equal(result, null);
|
|
});
|
|
});
|
|
|
|
// ── Test 8: mock spawn — NDJSON stream yields correct IR chunks ───────
|
|
it('spawn with mock: NDJSON lines → delta chunks then stop chunk', async () => {
|
|
const fakeSpawn = makeMockCodexSpawn([
|
|
'{"content":"Hello"}',
|
|
'{"content":" world"}',
|
|
'{"type":"stop"}',
|
|
]);
|
|
codexSetSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'gpt-5.5',
|
|
stream: true,
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
});
|
|
const authCtx = { accessToken: '<fake-openai-token>' };
|
|
const chunks = [];
|
|
for await (const chunk of codex.spawn(ir, authCtx)) {
|
|
chunks.push(chunk);
|
|
}
|
|
const deltas = chunks.filter(c => c.type === 'delta');
|
|
const stops = chunks.filter(c => c.type === 'stop');
|
|
assert.ok(deltas.length >= 1, `Expected at least 1 delta, got ${deltas.length}`);
|
|
assert.equal(stops.length, 1, `Expected 1 stop, got ${stops.length}`);
|
|
const allContent = deltas.map(c => c.content).join('');
|
|
assert.equal(allContent, 'Hello world');
|
|
} finally {
|
|
codexResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('spawn with mock: first delta chunk has role=assistant', async () => {
|
|
const fakeSpawn = makeMockCodexSpawn(['{"content":"Test output"}']);
|
|
codexSetSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'gpt-5.5',
|
|
stream: true,
|
|
messages: [{ role: 'user', content: 'Hello' }],
|
|
});
|
|
const authCtx = { accessToken: '<fake-openai-token>' };
|
|
const chunks = [];
|
|
for await (const chunk of codex.spawn(ir, authCtx)) {
|
|
chunks.push(chunk);
|
|
}
|
|
const firstDelta = chunks.find(c => c.type === 'delta');
|
|
assert.ok(firstDelta, 'No delta chunk found');
|
|
assert.equal(firstDelta.role, 'assistant');
|
|
} finally {
|
|
codexResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('spawn with mock: NDJSON stop event present → no extra synthetic stop appended', async () => {
|
|
// If NDJSON stream already contains a stop event, we should NOT double-emit
|
|
const fakeSpawn = makeMockCodexSpawn([
|
|
'{"content":"done"}',
|
|
'{"type":"stop"}',
|
|
]);
|
|
codexSetSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'gpt-5.5',
|
|
stream: false,
|
|
messages: [{ role: 'user', content: 'Hello' }],
|
|
});
|
|
const authCtx = { accessToken: '<fake-openai-token>' };
|
|
const chunks = [];
|
|
for await (const chunk of codex.spawn(ir, authCtx)) {
|
|
chunks.push(chunk);
|
|
}
|
|
const stops = chunks.filter(c => c.type === 'stop');
|
|
assert.equal(stops.length, 1, `Expected exactly 1 stop chunk, got ${stops.length}`);
|
|
} finally {
|
|
codexResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('spawn with mock: non-zero exit code throws ProviderError(SPAWN_FAILED)', async () => {
|
|
const fakeSpawn = makeMockCodexSpawn([], 1);
|
|
codexSetSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'gpt-5.5',
|
|
stream: true,
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
});
|
|
const authCtx = { accessToken: '<fake-openai-token>' };
|
|
let caught = null;
|
|
try {
|
|
for await (const _chunk of codex.spawn(ir, authCtx)) {
|
|
// drain
|
|
}
|
|
} catch (e) {
|
|
caught = e;
|
|
}
|
|
assert.ok(caught instanceof ProviderError, `Expected ProviderError, got ${caught?.constructor?.name}`);
|
|
assert.equal(caught.code, 'SPAWN_FAILED');
|
|
} finally {
|
|
codexResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('spawn throws ProviderError(AUTH_MISSING) when no auth context and no auth file', async () => {
|
|
const fakeSpawn = makeMockCodexSpawn(['{"content":"test"}']);
|
|
codexSetSpawnImpl(fakeSpawn);
|
|
// Override auth path to a nonexistent file to guarantee missing auth
|
|
const savedAuthPath = process.env.OPENAI_CODEX_AUTH_PATH;
|
|
process.env.OPENAI_CODEX_AUTH_PATH = '/nonexistent/path/auth.json';
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'gpt-5.5',
|
|
stream: false,
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
});
|
|
let caught = null;
|
|
try {
|
|
for await (const _chunk of codex.spawn(ir, null)) { // eslint-disable-line no-unused-vars
|
|
// drain
|
|
}
|
|
} catch (e) {
|
|
caught = e;
|
|
}
|
|
assert.ok(caught instanceof ProviderError, `Expected ProviderError, got ${caught?.constructor?.name}`);
|
|
assert.equal(caught.code, 'AUTH_MISSING');
|
|
} finally {
|
|
if (savedAuthPath !== undefined) {
|
|
process.env.OPENAI_CODEX_AUTH_PATH = savedAuthPath;
|
|
} else {
|
|
delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
}
|
|
codexResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
// ── Test 9: healthCheck ───────────────────────────────────────────────
|
|
it('healthCheck returns {ok: false, error: "codex binary not found"} when binary absent', async () => {
|
|
const result = await codexHealthCheck({
|
|
_binaryExistsFn: () => false,
|
|
_authReadFn: () => ({ accessToken: '<fake-token>' }),
|
|
});
|
|
assert.equal(result.ok, false);
|
|
assert.equal(result.error, 'codex binary not found');
|
|
assert.ok(typeof result.latencyMs === 'number');
|
|
});
|
|
|
|
it('healthCheck returns {ok: false, error: "auth artifact missing"} when auth missing', async () => {
|
|
const result = await codexHealthCheck({
|
|
_binaryExistsFn: () => true,
|
|
_authReadFn: () => null,
|
|
});
|
|
assert.equal(result.ok, false);
|
|
assert.equal(result.error, 'auth artifact missing');
|
|
assert.ok(typeof result.latencyMs === 'number');
|
|
});
|
|
|
|
it('healthCheck returns {ok: true} when binary and auth both present', async () => {
|
|
const result = await codexHealthCheck({
|
|
_binaryExistsFn: () => true,
|
|
_authReadFn: () => ({ accessToken: '<fake-token>' }),
|
|
});
|
|
assert.equal(result.ok, true);
|
|
assert.ok(typeof result.latencyMs === 'number');
|
|
});
|
|
|
|
// ── Test 10: estimateCost ─────────────────────────────────────────────
|
|
it('estimateCost returns shape with currency USD', () => {
|
|
const request = makeIR({
|
|
model: 'gpt-5.5',
|
|
messages: [
|
|
{ role: 'system', content: 'You are a coding assistant.' },
|
|
{ role: 'user', content: 'Write hello world in Python.' },
|
|
],
|
|
});
|
|
const result = codexEstimateCost(request);
|
|
assert.ok(result !== null, 'estimateCost returned null');
|
|
assert.ok('inputTokens' in result, 'missing inputTokens');
|
|
assert.ok('outputTokensEstimate' in result, 'missing outputTokensEstimate');
|
|
assert.ok('currency' in result, 'missing currency');
|
|
assert.ok('usd' in result, 'missing usd');
|
|
assert.equal(result.currency, 'USD');
|
|
assert.equal(result.usd, null); // not pinned at D6
|
|
assert.ok(result.inputTokens > 0, 'inputTokens should be > 0');
|
|
assert.ok(result.outputTokensEstimate >= 0);
|
|
});
|
|
|
|
it('estimateCost returns null for null/missing request', () => {
|
|
assert.equal(codexEstimateCost(null), null);
|
|
assert.equal(codexEstimateCost({}), null);
|
|
});
|
|
|
|
// ── Test 11: quotaStatus ──────────────────────────────────────────────
|
|
it('quotaStatus returns null at D6', async () => {
|
|
const result = await codexQuotaStatus({});
|
|
assert.equal(result, null);
|
|
});
|
|
|
|
// ── Test 12: auth artifact path uses os.homedir(), not hardcoded ──────
|
|
it('codex.auth.path uses homedir() and references .codex directory', () => {
|
|
assert.equal(typeof codex.auth.path, 'string');
|
|
assert.ok(codex.auth.path.includes('.codex'), 'auth.path should reference .codex directory');
|
|
// Portability check: path must start with homedir() value at runtime
|
|
assert.ok(
|
|
codex.auth.path.startsWith(homedir()),
|
|
`auth.path "${codex.auth.path}" should start with homedir() "${homedir()}"`,
|
|
);
|
|
});
|
|
|
|
it('codex.auth.type === "oauth" and storage === "file"', () => {
|
|
assert.equal(codex.auth.type, 'oauth');
|
|
assert.equal(codex.auth.storage, 'file');
|
|
});
|
|
|
|
// ── Test 13: readAuthArtifact reads OPENAI_CODEX_AUTH_PATH override ──
|
|
it('readAuthArtifact: OPENAI_CODEX_AUTH_PATH pointing to nonexistent file → null', () => {
|
|
const saved = process.env.OPENAI_CODEX_AUTH_PATH;
|
|
process.env.OPENAI_CODEX_AUTH_PATH = '/definitely/not/a/real/path/auth.json';
|
|
try {
|
|
const result = codexReadAuthArtifact();
|
|
assert.equal(result, null);
|
|
} finally {
|
|
if (saved !== undefined) process.env.OPENAI_CODEX_AUTH_PATH = saved;
|
|
else delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
}
|
|
});
|
|
|
|
// ── Test 14: hints shape ──────────────────────────────────────────────
|
|
it('codex.hints has correct shape', () => {
|
|
assert.equal(typeof codex.hints.requiresTTY, 'boolean');
|
|
assert.equal(typeof codex.hints.concurrentSpawnSafe, 'boolean');
|
|
assert.ok(Number.isInteger(codex.hints.maxConcurrent) && codex.hints.maxConcurrent > 0);
|
|
assert.equal(codex.hints.requiresTTY, false);
|
|
assert.equal(codex.hints.concurrentSpawnSafe, true);
|
|
});
|
|
|
|
// ── Test 15: STATIC_REGISTRY length after D8 still includes openai ────
|
|
it('STATIC_REGISTRY includes openai (D6) after D8 (length >= 2)', () => {
|
|
// D8 adds mistral; anthropic + openai must still be present.
|
|
assert.ok(listAllProviderNames().length >= 2);
|
|
assert.ok(listAllProviderNames().includes('anthropic'));
|
|
assert.ok(listAllProviderNames().includes('openai'));
|
|
});
|
|
|
|
// ── Test 16: loadProviders with openai enabled ────────────────────────
|
|
it('loadProviders with {enabled: {openai: true}} returns Map of size 1 with openai', () => {
|
|
const loaded = loadProviders({ enabled: { openai: true } });
|
|
assert.equal(loaded.size, 1);
|
|
assert.ok(loaded.has('openai'));
|
|
});
|
|
|
|
it('loadProviders with both anthropic and openai enabled returns Map of size 2', () => {
|
|
const loaded = loadProviders({ enabled: { anthropic: true, openai: true } });
|
|
assert.equal(loaded.size, 2);
|
|
assert.ok(loaded.has('anthropic'));
|
|
assert.ok(loaded.has('openai'));
|
|
});
|
|
|
|
it('openai loaded via loadProviders passes contract validation', () => {
|
|
const loaded = loadProviders({ enabled: { openai: true } });
|
|
const p = loaded.get('openai');
|
|
const { valid, errors } = validateProvider(p);
|
|
assert.equal(valid, true, `Contract errors: ${errors.join('; ')}`);
|
|
assert.ok(p.models.includes('gpt-5.5'));
|
|
});
|
|
|
|
// ── Test 17: spawn progress events silently ignored ──────────────────
|
|
it('spawn with mock: progress events are silently ignored, only content emitted', async () => {
|
|
const fakeSpawn = makeMockCodexSpawn([
|
|
'{"type":"progress","step":1}',
|
|
'{"type":"progress","step":2}',
|
|
'{"content":"actual response"}',
|
|
'{"type":"stop"}',
|
|
]);
|
|
codexSetSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'gpt-5.5',
|
|
stream: true,
|
|
messages: [{ role: 'user', content: 'Do something' }],
|
|
});
|
|
const authCtx = { accessToken: '<fake-openai-token>' };
|
|
const chunks = [];
|
|
for await (const chunk of codex.spawn(ir, authCtx)) {
|
|
chunks.push(chunk);
|
|
}
|
|
const deltas = chunks.filter(c => c.type === 'delta');
|
|
assert.equal(deltas.length, 1);
|
|
assert.equal(deltas[0].content, 'actual response');
|
|
} finally {
|
|
codexResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
// ── Test 18: spawn synthesizes stop when NDJSON stream has no stop event ──
|
|
it('spawn with mock: synthetic stop emitted when NDJSON has no stop event', async () => {
|
|
const fakeSpawn = makeMockCodexSpawn([
|
|
'{"content":"only content, no stop"}',
|
|
// No stop event in the stream
|
|
]);
|
|
codexSetSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'gpt-5.5',
|
|
stream: true,
|
|
messages: [{ role: 'user', content: 'Hello' }],
|
|
});
|
|
const authCtx = { accessToken: '<fake-openai-token>' };
|
|
const chunks = [];
|
|
for await (const chunk of codex.spawn(ir, authCtx)) {
|
|
chunks.push(chunk);
|
|
}
|
|
const stops = chunks.filter(c => c.type === 'stop');
|
|
assert.equal(stops.length, 1, 'Should have exactly 1 synthetic stop chunk');
|
|
assert.equal(stops[0].finish_reason, 'stop');
|
|
} finally {
|
|
codexResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
});
|
|
|
|
// ── Suite 10: Anthropic E2E (GATED) ──────────────────────────────────────
|
|
//
|
|
// Run with: OLP_RUN_E2E=1 npm test
|
|
// Skipped by default; consumes real Anthropic tokens (~200 tokens per run,
|
|
// est. <$0.001 at haiku rates). Requires `claude` binary + keychain OAuth
|
|
// token. The orchestrator runs this once per D5 verification.
|
|
//
|
|
// This suite is NOT run in CI (CI does not set OLP_RUN_E2E). The skip notice
|
|
// is emitted as a console message so CI logs show the gated test exists.
|
|
//
|
|
// Tests:
|
|
// 1. Start OLP server with anthropic enabled.
|
|
// 2. POST minimal request to claude-haiku-4-5.
|
|
// 3. Assert 200, response contains "OK", correct provider/model headers.
|
|
// 4. Send same request again → assert X-OLP-Cache: hit, identical content.
|
|
// 5. Assert X-OLP-Fallback-Hops: 0.
|
|
//
|
|
// Model: claude-haiku-4-5 (cheapest). Prompt: "Reply with exactly the word OK
|
|
// and nothing else." max_tokens: 10. Target: < 200 tokens per run.
|
|
//
|
|
// Do NOT include any real OAuth tokens or API keys in this test. Auth is read
|
|
// from keychain / CLAUDE_CODE_OAUTH_TOKEN env / ~/.claude/.credentials.json
|
|
// by readAuthArtifact() inside the anthropic plugin at spawn time.
|
|
|
|
const RUN_E2E = process.env.OLP_RUN_E2E === '1';
|
|
|
|
if (!RUN_E2E) {
|
|
// Emit a skip notice to CI logs without failing the suite
|
|
process.stdout.write('::notice::Suite 10 (Anthropic E2E) skipped — set OLP_RUN_E2E=1 to run. Requires claude binary + keychain OAuth token. ~200 tokens per run.\n');
|
|
}
|
|
|
|
describe('Anthropic E2E — real claude spawn (Suite 10)', { skip: !RUN_E2E }, () => {
|
|
let e2eServer;
|
|
let e2ePort;
|
|
let e2eLoadedProviders;
|
|
let e2eCacheStore;
|
|
|
|
before(async () => {
|
|
if (!RUN_E2E) return;
|
|
|
|
__resetSpawnImpl(); // ensure real spawn is active
|
|
|
|
const { createOlpServer, loadedProviders: lp, cacheStore: cs } = await import('./server.mjs');
|
|
e2eLoadedProviders = lp;
|
|
e2eCacheStore = cs;
|
|
|
|
// Enable anthropic provider for E2E
|
|
const testProviders = loadProviders({ enabled: { anthropic: true } });
|
|
for (const [name, p] of testProviders) {
|
|
lp.set(name, p);
|
|
}
|
|
|
|
e2ePort = parseInt(process.env.OLP_E2E_PORT ?? String(19456 + Math.floor(Math.random() * 500)), 10);
|
|
e2eServer = createOlpServer();
|
|
await new Promise((resolve, reject) => {
|
|
e2eServer.listen(e2ePort, '127.0.0.1', resolve);
|
|
e2eServer.once('error', async (e) => {
|
|
if (e.code === 'EADDRINUSE') {
|
|
e2ePort++;
|
|
e2eServer.listen(e2ePort, '127.0.0.1', resolve);
|
|
e2eServer.once('error', reject);
|
|
} else reject(e);
|
|
});
|
|
});
|
|
});
|
|
|
|
after(() => {
|
|
if (!e2eServer) return;
|
|
return new Promise(r => e2eServer.close(r));
|
|
});
|
|
|
|
it('E2E: POST claude-haiku-4-5 with minimal prompt → 200 + "OK" content + correct headers', async () => {
|
|
const body = {
|
|
model: 'claude-haiku-4-5',
|
|
messages: [{ role: 'user', content: 'Reply with exactly the word OK and nothing else.' }],
|
|
max_tokens: 10,
|
|
};
|
|
|
|
const r = await fetch({ port: e2ePort, method: 'POST', path: '/v1/chat/completions', body });
|
|
assert.equal(r.status, 200, `Expected 200, got ${r.status}: ${r.body}`);
|
|
|
|
const respBody = JSON.parse(r.body);
|
|
const content = respBody?.choices?.[0]?.message?.content ?? '';
|
|
assert.ok(
|
|
content.toLowerCase().includes('ok'),
|
|
`Expected response to contain "OK", got: ${content}`,
|
|
);
|
|
|
|
// Provider/model headers
|
|
assert.equal(r.headers['x-olp-provider-used'], 'anthropic', `Expected anthropic provider`);
|
|
assert.equal(r.headers['x-olp-model-used'], 'claude-haiku-4-5', `Expected haiku model`);
|
|
|
|
// First request is always a miss
|
|
assert.equal(r.headers['x-olp-cache'], 'miss', `First request should be cache miss`);
|
|
|
|
// Fallback hops
|
|
assert.equal(r.headers['x-olp-fallback-hops'], '0', `Expected 0 fallback hops`);
|
|
|
|
// Second request — should hit cache, content identical
|
|
const r2 = await fetch({ port: e2ePort, method: 'POST', path: '/v1/chat/completions', body });
|
|
assert.equal(r2.status, 200, `Cache hit request should be 200`);
|
|
assert.equal(r2.headers['x-olp-cache'], 'hit', `Second request should be cache hit`);
|
|
|
|
const resp2Body = JSON.parse(r2.body);
|
|
const content2 = resp2Body?.choices?.[0]?.message?.content ?? '';
|
|
assert.ok(
|
|
content2.toLowerCase().includes('ok'),
|
|
`Cache hit response should also contain "OK", got: ${content2}`,
|
|
);
|
|
});
|
|
});
|
|
|
|
// ── Suite 12: Mistral Vibe plugin (D8) ───────────────────────────────────
|
|
//
|
|
// All tests are UNIT tests. No real `vibe` binary is invoked.
|
|
// Mock spawn is injected via mistralSetSpawnImpl / mistralResetSpawnImpl.
|
|
//
|
|
// Authority: Mistral Vibe docs (WebFetched 2026-05-23):
|
|
// DOCS-1: https://docs.mistral.ai/mistral-vibe/terminal/quickstart
|
|
// § "--prompt TEXT" + "--output FORMAT" — programmatic mode syntax
|
|
// DOCS-2: https://docs.mistral.ai/mistral-vibe/terminal/configuration
|
|
// § "~/.vibe/.env" — auth file; MISTRAL_API_KEY env var
|
|
// DOCS-5: https://mistral.ai/news/devstral-2-vibe-cli
|
|
// § "Devstral 2", "Devstral Small 2" — model names
|
|
// DOCS-6: https://help.mistral.ai/en/articles/347532
|
|
// § "Mistral Vibe is included in every Le Chat Pro subscription"
|
|
// DOCS-7: https://legal.mistral.ai/terms/usage-policy
|
|
// § No anti-third-party clauses (Tier D confirmed for ADR 0006)
|
|
//
|
|
// Spec assumption acknowledgements (see mistral.mjs header):
|
|
// A1 CONFIRMED: `vibe --prompt "PROMPT" --output json` spawn shape
|
|
// A2 CONFIRMED: MISTRAL_API_KEY env var is the auth mechanism
|
|
// A3 CONFIRMED: ~/.vibe/.env is the auth file path
|
|
// A4 UNPINNED: JSON output event schema — defensive 4-shape parser used
|
|
// A5 UNPINNED: --model flag existence not confirmed by docs
|
|
// A6 UNPINNED: exact model IDs (devstral-2, devstral-small-2) best-effort
|
|
// A7 UNPINNED: --output json is NDJSON not single blob
|
|
// A8 UNPINNED: multi-line prompt handling via --prompt flag
|
|
//
|
|
// Lossy-translation acknowledgements per ADR 0003 (documented in mistral.mjs header):
|
|
// top_p, temperature, stop, max_tokens, tools[], tool_calls → all dropped.
|
|
|
|
/**
|
|
* Creates a fake JSON-line-emitting mock spawn for Mistral Vibe.
|
|
* Lines are emitted as they would arrive from `vibe --output json`
|
|
* (D8 assumption A7: NDJSON lines — D-later will pin the real format).
|
|
*
|
|
* @param {string[]} jsonLines — raw JSON lines emitted in order (no trailing \n needed)
|
|
* @param {number} [exitCode=0]
|
|
*/
|
|
function makeMockMistralSpawn(jsonLines, exitCode = 0) {
|
|
return function mockMistralSpawnImpl(_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(async () => {
|
|
for (const line of jsonLines) {
|
|
proc.stdout.emit('data', Buffer.from(line + '\n'));
|
|
}
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', exitCode, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
};
|
|
}
|
|
|
|
describe('Mistral Vibe plugin (D8)', () => {
|
|
|
|
// ── Test 1: Contract conformance ──────────────────────────────────────
|
|
it('mistral module satisfies validateProvider() — all 10 fields present', () => {
|
|
const { valid, errors } = validateProvider(mistral);
|
|
assert.equal(valid, true, `Validation errors: ${errors.join('; ')}`);
|
|
assert.ok('name' in mistral, 'missing: name');
|
|
assert.ok('displayName' in mistral, 'missing: displayName');
|
|
assert.ok('contractVersion' in mistral, 'missing: contractVersion');
|
|
assert.ok('models' in mistral, 'missing: models');
|
|
assert.ok('auth' in mistral, 'missing: auth');
|
|
assert.ok(typeof mistral.spawn === 'function', 'missing: spawn');
|
|
assert.ok(typeof mistral.estimateCost === 'function', 'missing: estimateCost');
|
|
assert.ok(typeof mistral.quotaStatus === 'function', 'missing: quotaStatus');
|
|
assert.ok(typeof mistral.healthCheck === 'function', 'missing: healthCheck');
|
|
assert.ok('hints' in mistral, 'missing: hints');
|
|
});
|
|
|
|
// ── Test 2: contractVersion === '1.0' ────────────────────────────────
|
|
it('mistral declares contractVersion === "1.0"', () => {
|
|
assert.equal(mistral.contractVersion, '1.0');
|
|
});
|
|
|
|
// ── Test 3: name and displayName ─────────────────────────────────────
|
|
it('mistral.name === "mistral" and displayName is set', () => {
|
|
assert.equal(mistral.name, 'mistral');
|
|
assert.equal(typeof mistral.displayName, 'string');
|
|
assert.ok(mistral.displayName.length > 0);
|
|
assert.ok(mistral.displayName.toLowerCase().includes('mistral'));
|
|
});
|
|
|
|
// ── Test 4: models is canonical-only (D17 Finding 12 fix) ───────────
|
|
it('mistral.models contains only canonical registry IDs — no alias strings', () => {
|
|
// D17 fix: models[] is canonical-only across all plugins. Alias routing
|
|
// is the responsibility of getProviderForModel() in lib/providers/index.mjs.
|
|
const registryIds = modelsRegistry.providers.mistral.models.map(m => m.id);
|
|
const registryAliases = Object.keys(modelsRegistry.providers.mistral.aliases ?? {});
|
|
for (const id of registryIds) {
|
|
assert.ok(mistral.models.includes(id), `canonical id ${id} missing from mistral.models`);
|
|
}
|
|
for (const alias of registryAliases) {
|
|
assert.ok(!mistral.models.includes(alias), `alias ${alias} must NOT appear in mistral.models (D17)`);
|
|
}
|
|
assert.equal(mistral.models.length, registryIds.length, `Expected ${registryIds.length} canonical IDs, got ${mistral.models.length}`);
|
|
});
|
|
|
|
it('mistral.models contains exactly the two canonical date-stamped IDs', () => {
|
|
// D17 fix: canonical-only shape. Length: 2 canonical IDs.
|
|
assert.ok(mistral.models.includes('devstral-2-25-12'), 'missing canonical devstral-2-25-12');
|
|
assert.ok(mistral.models.includes('devstral-small-2-25-12'), 'missing canonical devstral-small-2-25-12');
|
|
assert.ok(!mistral.models.includes('devstral-2'), 'alias devstral-2 must NOT be in models[] (D17)');
|
|
assert.ok(!mistral.models.includes('devstral-small-2'), 'alias devstral-small-2 must NOT be in models[] (D17)');
|
|
assert.ok(!mistral.models.includes('devstral'), 'alias devstral must NOT be in models[] (D17)');
|
|
assert.ok(!mistral.models.includes('devstral-small'), 'alias devstral-small must NOT be in models[] (D17)');
|
|
// Length: 2 canonical only
|
|
assert.equal(mistral.models.length, 2, `Expected 2 canonical IDs, got ${mistral.models.length}`);
|
|
});
|
|
|
|
// ── Test 5: getProviderForModel finds mistral for each model ──────────
|
|
it('getProviderForModel finds mistral for canonical devstral-2-25-12', () => {
|
|
const loaded = new Map([['mistral', mistral]]);
|
|
const result = getProviderForModel(loaded, 'devstral-2-25-12');
|
|
assert.ok(result !== null);
|
|
assert.equal(result.name, 'mistral');
|
|
assert.equal(result.canonicalModel, 'devstral-2-25-12');
|
|
});
|
|
|
|
it('getProviderForModel finds mistral for alias devstral-2 → canonical devstral-2-25-12 (D17)', () => {
|
|
// D17: alias resolution now handled in getProviderForModel, not in models[].
|
|
const loaded = new Map([['mistral', mistral]]);
|
|
const result = getProviderForModel(loaded, 'devstral-2');
|
|
assert.ok(result !== null);
|
|
assert.equal(result.name, 'mistral');
|
|
assert.equal(result.canonicalModel, 'devstral-2-25-12');
|
|
});
|
|
|
|
it('getProviderForModel finds mistral for alias devstral → canonical devstral-2-25-12 (D17)', () => {
|
|
const loaded = new Map([['mistral', mistral]]);
|
|
const result = getProviderForModel(loaded, 'devstral');
|
|
assert.ok(result !== null);
|
|
assert.equal(result.name, 'mistral');
|
|
assert.equal(result.canonicalModel, 'devstral-2-25-12');
|
|
});
|
|
|
|
it('getProviderForModel finds mistral for alias devstral-small-2 → canonical devstral-small-2-25-12 (D17)', () => {
|
|
const loaded = new Map([['mistral', mistral]]);
|
|
const result = getProviderForModel(loaded, 'devstral-small-2');
|
|
assert.ok(result !== null);
|
|
assert.equal(result.name, 'mistral');
|
|
assert.equal(result.canonicalModel, 'devstral-small-2-25-12');
|
|
});
|
|
|
|
// ── Test 6: irToMistral translation ──────────────────────────────────
|
|
it('irToMistral: user message → args with --prompt and --output streaming', () => {
|
|
// Authority: DOCS-1 § "Output Format Options" — `streaming` is the
|
|
// newline-delimited JSON per message mode. `json` emits a single blob
|
|
// at the end (D8 review-2 finding: original draft used `json`,
|
|
// incompatible with the line-buffered stdout parser; corrected to
|
|
// `streaming` per docs verbatim).
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
messages: [{ role: 'user', content: 'Hello world' }],
|
|
});
|
|
const { args, prompt } = irToMistral(ir);
|
|
assert.ok(args.includes('--prompt'), 'args must include "--prompt"');
|
|
assert.ok(args.includes('--output'), 'args must include "--output"');
|
|
assert.ok(args.includes('streaming'), 'args must include "streaming" output format (NDJSON per docs)');
|
|
assert.ok(!args.includes('json'), 'args must NOT include "json" (single-blob mode, incompatible with line-buffered parser)');
|
|
// D8 assumption A5: --model NOT in args (flag existence unconfirmed from docs)
|
|
assert.ok(!args.includes('--model'), 'args must NOT include "--model" at D8 (A5 UNPINNED)');
|
|
assert.ok(typeof prompt === 'string', 'prompt must be a string');
|
|
assert.ok(prompt.includes('Hello world'), 'prompt must contain user text');
|
|
});
|
|
|
|
it('irToMistral: system + user → system annotation + user text in prompt', () => {
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
messages: [
|
|
{ role: 'system', content: 'You are a coding assistant.' },
|
|
{ role: 'user', content: 'Write a function.' },
|
|
],
|
|
});
|
|
const { prompt } = irToMistral(ir);
|
|
assert.ok(prompt.includes('[System] You are a coding assistant.'));
|
|
assert.ok(prompt.includes('Write a function.'));
|
|
});
|
|
|
|
it('irToMistral: assistant prior turn → [Assistant] annotation', () => {
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
messages: [
|
|
{ role: 'user', content: 'Hi' },
|
|
{ role: 'assistant', content: 'Hello!' },
|
|
{ role: 'user', content: 'Thanks' },
|
|
],
|
|
});
|
|
const { prompt } = irToMistral(ir);
|
|
assert.ok(prompt.includes('[Assistant] Hello!'));
|
|
assert.ok(prompt.includes('Thanks'));
|
|
});
|
|
|
|
it('irToMistral: tool result turn → [Tool Result] annotation', () => {
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
messages: [
|
|
{ role: 'user', content: 'Search for X' },
|
|
{ role: 'tool', content: '{"results":[]}', name: 'search' },
|
|
],
|
|
});
|
|
const { prompt } = irToMistral(ir);
|
|
assert.ok(prompt.includes('[Tool Result'));
|
|
assert.ok(prompt.includes('search'));
|
|
});
|
|
|
|
it('irToMistral: response_format json_object injects system prompt (lossy)', () => {
|
|
// ADR 0003 § Lossy: Vibe CLI does not honor response_format natively.
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
messages: [{ role: 'user', content: 'Give JSON' }],
|
|
response_format: { type: 'json_object' },
|
|
});
|
|
const { prompt } = irToMistral(ir);
|
|
assert.ok(prompt.includes('Reply with valid JSON only'));
|
|
});
|
|
|
|
it('irToMistral: tool_calls in assistant message — content preserved, metadata dropped (lossy)', () => {
|
|
// ADR 0003 § Lossy: structured tool_calls dropped; textual content preserved.
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
messages: [
|
|
{ role: 'user', content: 'Search for X' },
|
|
{
|
|
role: 'assistant',
|
|
content: 'Searching...',
|
|
tool_calls: [{ id: 'tc1', type: 'function', function: { name: 'search', arguments: '{"q":"X"}' } }],
|
|
},
|
|
],
|
|
});
|
|
const { prompt } = irToMistral(ir);
|
|
assert.ok(prompt.includes('Searching...'), 'assistant text content must be preserved');
|
|
// tool_calls metadata documented as dropped (lossy) — do NOT assert it's present
|
|
});
|
|
|
|
it('irToMistral: array content is JSON-stringified', () => {
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }],
|
|
});
|
|
const { prompt } = irToMistral(ir);
|
|
assert.ok(typeof prompt === 'string');
|
|
assert.ok(prompt.includes('text'));
|
|
});
|
|
|
|
// ── Test 7: mistralChunkToIR translation ──────────────────────────────
|
|
it('mistralChunkToIR: text field → delta chunk (Mistral La Plateforme shape)', () => {
|
|
// D8 assumption A4: "text" is the preferred field name.
|
|
const chunk = mistralChunkToIR('{"text":"Hello world"}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'delta');
|
|
assert.equal(chunk.content, 'Hello world');
|
|
});
|
|
|
|
it('mistralChunkToIR: content field → delta chunk (OpenAI-compat shape)', () => {
|
|
// D8 assumption A4: "content" as fallback field name.
|
|
const chunk = mistralChunkToIR('{"content":"Hello world"}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'delta');
|
|
assert.equal(chunk.content, 'Hello world');
|
|
});
|
|
|
|
it('mistralChunkToIR: delta field shape → delta chunk', () => {
|
|
// D8 assumption A4: { type: 'delta', delta: '...' } shape.
|
|
const chunk = mistralChunkToIR('{"type":"delta","delta":"token text"}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'delta');
|
|
assert.equal(chunk.content, 'token text');
|
|
});
|
|
|
|
it('mistralChunkToIR: OpenAI streaming choices[0].delta.content shape → delta', () => {
|
|
// D8 assumption A4: Vibe may use OpenAI streaming event shape.
|
|
const chunk = mistralChunkToIR('{"choices":[{"delta":{"content":"output"},"finish_reason":null}]}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'delta');
|
|
assert.equal(chunk.content, 'output');
|
|
});
|
|
|
|
it('mistralChunkToIR: type === "stop" → stop chunk', () => {
|
|
const chunk = mistralChunkToIR('{"type":"stop"}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'stop');
|
|
assert.equal(chunk.finish_reason, 'stop');
|
|
});
|
|
|
|
it('mistralChunkToIR: done === true → stop chunk', () => {
|
|
const chunk = mistralChunkToIR('{"done":true}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'stop');
|
|
assert.equal(chunk.finish_reason, 'stop');
|
|
});
|
|
|
|
it('mistralChunkToIR: OpenAI streaming stop via choices[0].finish_reason === "stop"', () => {
|
|
// D8 assumption A4: OpenAI streaming stop shape.
|
|
const chunk = mistralChunkToIR('{"choices":[{"delta":{},"finish_reason":"stop"}]}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'stop');
|
|
assert.equal(chunk.finish_reason, 'stop');
|
|
});
|
|
|
|
it('mistralChunkToIR: type === "error" → error chunk', () => {
|
|
const chunk = mistralChunkToIR('{"type":"error","error":"quota exceeded"}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'error');
|
|
assert.equal(chunk.error, 'quota exceeded');
|
|
});
|
|
|
|
it('mistralChunkToIR: error field present → error chunk', () => {
|
|
const chunk = mistralChunkToIR('{"error":"something went wrong","code":429}');
|
|
assert.ok(chunk !== null);
|
|
assert.equal(chunk.type, 'error');
|
|
});
|
|
|
|
it('mistralChunkToIR: unknown/progress event type → null (silently ignored)', () => {
|
|
const chunk = mistralChunkToIR('{"type":"progress","step":1}');
|
|
assert.equal(chunk, null);
|
|
});
|
|
|
|
it('mistralChunkToIR: empty line → null', () => {
|
|
assert.equal(mistralChunkToIR(''), null);
|
|
assert.equal(mistralChunkToIR(' '), null);
|
|
});
|
|
|
|
it('mistralChunkToIR: malformed JSON → null (no throw)', () => {
|
|
assert.doesNotThrow(() => {
|
|
const result = mistralChunkToIR('{bad json');
|
|
assert.equal(result, null);
|
|
});
|
|
});
|
|
|
|
// ── Test 8: mock spawn — JSON stream yields correct IR chunks ──────────
|
|
it('spawn with mock: JSON lines → delta chunks then stop chunk', async () => {
|
|
const fakeSpawn = makeMockMistralSpawn([
|
|
'{"text":"Hello"}',
|
|
'{"text":" world"}',
|
|
'{"type":"stop"}',
|
|
]);
|
|
mistralSetSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
stream: true,
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
});
|
|
const authCtx = { apiKey: '<fake-mistral-api-key>' };
|
|
const chunks = [];
|
|
for await (const chunk of mistral.spawn(ir, authCtx)) {
|
|
chunks.push(chunk);
|
|
}
|
|
const deltas = chunks.filter(c => c.type === 'delta');
|
|
const stops = chunks.filter(c => c.type === 'stop');
|
|
assert.ok(deltas.length >= 1, `Expected at least 1 delta, got ${deltas.length}`);
|
|
assert.equal(stops.length, 1, `Expected 1 stop, got ${stops.length}`);
|
|
const allContent = deltas.map(c => c.content).join('');
|
|
assert.equal(allContent, 'Hello world');
|
|
} finally {
|
|
mistralResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('spawn with mock: first delta chunk has role=assistant', async () => {
|
|
const fakeSpawn = makeMockMistralSpawn(['{"text":"Test output"}']);
|
|
mistralSetSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
stream: true,
|
|
messages: [{ role: 'user', content: 'Hello' }],
|
|
});
|
|
const authCtx = { apiKey: '<fake-mistral-api-key>' };
|
|
const chunks = [];
|
|
for await (const chunk of mistral.spawn(ir, authCtx)) {
|
|
chunks.push(chunk);
|
|
}
|
|
const firstDelta = chunks.find(c => c.type === 'delta');
|
|
assert.ok(firstDelta, 'No delta chunk found');
|
|
assert.equal(firstDelta.role, 'assistant');
|
|
} finally {
|
|
mistralResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('spawn with mock: stop event present → no extra synthetic stop appended', async () => {
|
|
const fakeSpawn = makeMockMistralSpawn([
|
|
'{"text":"done"}',
|
|
'{"type":"stop"}',
|
|
]);
|
|
mistralSetSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
stream: false,
|
|
messages: [{ role: 'user', content: 'Hello' }],
|
|
});
|
|
const authCtx = { apiKey: '<fake-mistral-api-key>' };
|
|
const chunks = [];
|
|
for await (const chunk of mistral.spawn(ir, authCtx)) {
|
|
chunks.push(chunk);
|
|
}
|
|
const stops = chunks.filter(c => c.type === 'stop');
|
|
assert.equal(stops.length, 1, `Expected exactly 1 stop chunk, got ${stops.length}`);
|
|
} finally {
|
|
mistralResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('spawn with mock: synthetic stop emitted when JSON stream has no stop event', async () => {
|
|
const fakeSpawn = makeMockMistralSpawn([
|
|
'{"text":"only content, no stop"}',
|
|
]);
|
|
mistralSetSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
stream: true,
|
|
messages: [{ role: 'user', content: 'Hello' }],
|
|
});
|
|
const authCtx = { apiKey: '<fake-mistral-api-key>' };
|
|
const chunks = [];
|
|
for await (const chunk of mistral.spawn(ir, authCtx)) {
|
|
chunks.push(chunk);
|
|
}
|
|
const stops = chunks.filter(c => c.type === 'stop');
|
|
assert.equal(stops.length, 1, 'Should have exactly 1 synthetic stop chunk');
|
|
assert.equal(stops[0].finish_reason, 'stop');
|
|
} finally {
|
|
mistralResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('spawn with mock: non-zero exit code throws ProviderError(SPAWN_FAILED)', async () => {
|
|
const fakeSpawn = makeMockMistralSpawn([], 1);
|
|
mistralSetSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
stream: true,
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
});
|
|
const authCtx = { apiKey: '<fake-mistral-api-key>' };
|
|
let caught = null;
|
|
try {
|
|
for await (const _chunk of mistral.spawn(ir, authCtx)) { // eslint-disable-line no-unused-vars
|
|
// drain
|
|
}
|
|
} catch (e) {
|
|
caught = e;
|
|
}
|
|
assert.ok(caught instanceof ProviderError, `Expected ProviderError, got ${caught?.constructor?.name}`);
|
|
assert.equal(caught.code, 'SPAWN_FAILED');
|
|
} finally {
|
|
mistralResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('spawn throws ProviderError(AUTH_MISSING) when no auth context and no env/file', async () => {
|
|
const fakeSpawn = makeMockMistralSpawn(['{"text":"test"}']);
|
|
mistralSetSpawnImpl(fakeSpawn);
|
|
// Override auth path to nonexistent + clear MISTRAL_API_KEY env to guarantee missing auth
|
|
const savedApiKey = process.env.MISTRAL_API_KEY;
|
|
const savedAuthPath = process.env.MISTRAL_VIBE_AUTH_PATH;
|
|
delete process.env.MISTRAL_API_KEY;
|
|
process.env.MISTRAL_VIBE_AUTH_PATH = '/nonexistent/path/.env';
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
stream: false,
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
});
|
|
let caught = null;
|
|
try {
|
|
for await (const _chunk of mistral.spawn(ir, null)) { // eslint-disable-line no-unused-vars
|
|
// drain
|
|
}
|
|
} catch (e) {
|
|
caught = e;
|
|
}
|
|
assert.ok(caught instanceof ProviderError, `Expected ProviderError, got ${caught?.constructor?.name}`);
|
|
assert.equal(caught.code, 'AUTH_MISSING');
|
|
} finally {
|
|
if (savedApiKey !== undefined) process.env.MISTRAL_API_KEY = savedApiKey;
|
|
else delete process.env.MISTRAL_API_KEY;
|
|
if (savedAuthPath !== undefined) process.env.MISTRAL_VIBE_AUTH_PATH = savedAuthPath;
|
|
else delete process.env.MISTRAL_VIBE_AUTH_PATH;
|
|
mistralResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('spawn with mock: progress events silently ignored, only content emitted', async () => {
|
|
const fakeSpawn = makeMockMistralSpawn([
|
|
'{"type":"progress","step":1}',
|
|
'{"type":"progress","step":2}',
|
|
'{"text":"actual response"}',
|
|
'{"type":"stop"}',
|
|
]);
|
|
mistralSetSpawnImpl(fakeSpawn);
|
|
try {
|
|
const ir = makeIR({
|
|
model: 'devstral-2',
|
|
stream: true,
|
|
messages: [{ role: 'user', content: 'Do something' }],
|
|
});
|
|
const authCtx = { apiKey: '<fake-mistral-api-key>' };
|
|
const chunks = [];
|
|
for await (const chunk of mistral.spawn(ir, authCtx)) {
|
|
chunks.push(chunk);
|
|
}
|
|
const deltas = chunks.filter(c => c.type === 'delta');
|
|
assert.equal(deltas.length, 1);
|
|
assert.equal(deltas[0].content, 'actual response');
|
|
} finally {
|
|
mistralResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
// ── Test 9: healthCheck ───────────────────────────────────────────────
|
|
it('healthCheck returns {ok: false, error: "vibe binary not found"} when binary absent', async () => {
|
|
const result = await mistralHealthCheck({
|
|
_binaryExistsFn: () => false,
|
|
_authReadFn: () => ({ apiKey: '<fake-key>' }),
|
|
});
|
|
assert.equal(result.ok, false);
|
|
assert.equal(result.error, 'vibe binary not found');
|
|
assert.ok(typeof result.latencyMs === 'number');
|
|
});
|
|
|
|
it('healthCheck returns {ok: false, error: "auth artifact missing"} when auth missing', async () => {
|
|
const result = await mistralHealthCheck({
|
|
_binaryExistsFn: () => true,
|
|
_authReadFn: () => null,
|
|
});
|
|
assert.equal(result.ok, false);
|
|
assert.equal(result.error, 'auth artifact missing');
|
|
assert.ok(typeof result.latencyMs === 'number');
|
|
});
|
|
|
|
it('healthCheck returns {ok: true} when binary and auth both present', async () => {
|
|
const result = await mistralHealthCheck({
|
|
_binaryExistsFn: () => true,
|
|
_authReadFn: () => ({ apiKey: '<fake-key>' }),
|
|
});
|
|
assert.equal(result.ok, true);
|
|
assert.ok(typeof result.latencyMs === 'number');
|
|
});
|
|
|
|
// ── Test 10: estimateCost ─────────────────────────────────────────────
|
|
it('estimateCost returns shape with currency USD', () => {
|
|
const request = makeIR({
|
|
model: 'devstral-2',
|
|
messages: [
|
|
{ role: 'system', content: 'You are a coding assistant.' },
|
|
{ role: 'user', content: 'Write hello world in Python.' },
|
|
],
|
|
});
|
|
const result = mistralEstimateCost(request);
|
|
assert.ok(result !== null, 'estimateCost returned null');
|
|
assert.ok('inputTokens' in result, 'missing inputTokens');
|
|
assert.ok('outputTokensEstimate' in result, 'missing outputTokensEstimate');
|
|
assert.ok('currency' in result, 'missing currency');
|
|
assert.ok('usd' in result, 'missing usd');
|
|
assert.equal(result.currency, 'USD');
|
|
assert.equal(result.usd, null); // not pinned at D8
|
|
assert.ok(result.inputTokens > 0, 'inputTokens should be > 0');
|
|
assert.ok(result.outputTokensEstimate >= 0);
|
|
});
|
|
|
|
it('estimateCost returns null for null/missing request', () => {
|
|
assert.equal(mistralEstimateCost(null), null);
|
|
assert.equal(mistralEstimateCost({}), null);
|
|
});
|
|
|
|
// ── Test 11: quotaStatus ──────────────────────────────────────────────
|
|
it('quotaStatus returns null at D8 (Le Chat Pro budget not exposed via API)', async () => {
|
|
const result = await mistralQuotaStatus({});
|
|
assert.equal(result, null);
|
|
});
|
|
|
|
// ── Test 12: auth object shape ────────────────────────────────────────
|
|
it('mistral.auth has correct shape', () => {
|
|
// Authority: DOCS-2 § auth type is api-key, path is ~/.vibe/.env
|
|
assert.equal(mistral.auth.type, 'api-key');
|
|
assert.equal(mistral.auth.storage, 'file');
|
|
assert.equal(typeof mistral.auth.path, 'string');
|
|
assert.ok(mistral.auth.path.includes('.vibe'), 'auth.path should reference .vibe directory');
|
|
assert.ok(mistral.auth.path.includes('.env'), 'auth.path should reference .env file');
|
|
// Portability check: path must start with homedir() (no hardcoded literal)
|
|
assert.ok(
|
|
mistral.auth.path.startsWith(homedir()),
|
|
`auth.path "${mistral.auth.path}" should start with homedir() "${homedir()}"`,
|
|
);
|
|
});
|
|
|
|
// ── Test 13: hints shape ──────────────────────────────────────────────
|
|
it('mistral.hints has correct shape', () => {
|
|
assert.equal(typeof mistral.hints.requiresTTY, 'boolean');
|
|
assert.equal(typeof mistral.hints.concurrentSpawnSafe, 'boolean');
|
|
assert.ok(Number.isInteger(mistral.hints.maxConcurrent) && mistral.hints.maxConcurrent > 0);
|
|
assert.equal(mistral.hints.requiresTTY, false);
|
|
assert.equal(mistral.hints.concurrentSpawnSafe, true);
|
|
});
|
|
|
|
// ── Test 14: STATIC_REGISTRY length after D8 ─────────────────────────
|
|
it('STATIC_REGISTRY.length === 3 after D8 (anthropic + openai + mistral)', () => {
|
|
assert.equal(listAllProviderNames().length, 3);
|
|
assert.ok(listAllProviderNames().includes('anthropic'));
|
|
assert.ok(listAllProviderNames().includes('openai'));
|
|
assert.ok(listAllProviderNames().includes('mistral'));
|
|
});
|
|
|
|
// ── Test 15: loadProviders with mistral enabled ───────────────────────
|
|
it('loadProviders with {enabled: {mistral: true}} returns Map of size 1 with mistral', () => {
|
|
const loaded = loadProviders({ enabled: { mistral: true } });
|
|
assert.equal(loaded.size, 1);
|
|
assert.ok(loaded.has('mistral'));
|
|
});
|
|
|
|
it('loadProviders with all 3 providers enabled returns Map of size 3', () => {
|
|
const loaded = loadProviders({ enabled: { anthropic: true, openai: true, mistral: true } });
|
|
assert.equal(loaded.size, 3);
|
|
assert.ok(loaded.has('anthropic'));
|
|
assert.ok(loaded.has('openai'));
|
|
assert.ok(loaded.has('mistral'));
|
|
});
|
|
|
|
it('mistral loaded via loadProviders passes contract validation', () => {
|
|
const loaded = loadProviders({ enabled: { mistral: true } });
|
|
const p = loaded.get('mistral');
|
|
const { valid, errors } = validateProvider(p);
|
|
assert.equal(valid, true, `Contract errors: ${errors.join('; ')}`);
|
|
// D17: models[] is canonical-only. Verify via getProviderForModel instead of direct inclusion.
|
|
assert.ok(p.models.includes('devstral-2-25-12'), 'canonical devstral-2-25-12 must be in models[]');
|
|
const r = getProviderForModel(loaded, 'devstral-2');
|
|
assert.ok(r !== null, 'alias devstral-2 must route to mistral via getProviderForModel (D17)');
|
|
assert.equal(r.name, 'mistral');
|
|
});
|
|
|
|
// ── Test 16: auth artifact reading helpers ────────────────────────────
|
|
it('readAuthArtifact: MISTRAL_API_KEY env var → returns {apiKey}', () => {
|
|
const saved = process.env.MISTRAL_API_KEY;
|
|
process.env.MISTRAL_API_KEY = '<fake-mistral-api-key>';
|
|
try {
|
|
const result = mistralReadAuthArtifact();
|
|
assert.ok(result !== null, 'Expected auth result');
|
|
assert.equal(result.apiKey, '<fake-mistral-api-key>');
|
|
} finally {
|
|
if (saved !== undefined) process.env.MISTRAL_API_KEY = saved;
|
|
else delete process.env.MISTRAL_API_KEY;
|
|
}
|
|
});
|
|
|
|
it('readAuthArtifact: MISTRAL_VIBE_AUTH_PATH pointing to nonexistent file → null', () => {
|
|
const savedApiKey = process.env.MISTRAL_API_KEY;
|
|
const savedAuthPath = process.env.MISTRAL_VIBE_AUTH_PATH;
|
|
delete process.env.MISTRAL_API_KEY;
|
|
process.env.MISTRAL_VIBE_AUTH_PATH = '/definitely/not/a/real/path/.env';
|
|
try {
|
|
const result = mistralReadAuthArtifact();
|
|
assert.equal(result, null);
|
|
} finally {
|
|
if (savedApiKey !== undefined) process.env.MISTRAL_API_KEY = savedApiKey;
|
|
else delete process.env.MISTRAL_API_KEY;
|
|
if (savedAuthPath !== undefined) process.env.MISTRAL_VIBE_AUTH_PATH = savedAuthPath;
|
|
else delete process.env.MISTRAL_VIBE_AUTH_PATH;
|
|
}
|
|
});
|
|
|
|
// ── Test 17: Suite 5 registry test updated for D8 ─────────────────────
|
|
// (This re-tests the registry length assertion which now expects 3.)
|
|
it('listAllProviderNames() now returns 3-element array with mistral included', () => {
|
|
const names = listAllProviderNames();
|
|
assert.equal(names.length, 3);
|
|
assert.deepEqual(names, ['anthropic', 'openai', 'mistral']);
|
|
});
|
|
|
|
});
|
|
|
|
// ── Suite 13: Fallback engine (D9) ───────────────────────────────────────
|
|
//
|
|
// All tests are unit tests. No real provider CLIs are invoked.
|
|
// Tests cover:
|
|
// 13a: Trigger taxonomy (evaluateHardTriggers / evaluateSoftTriggers)
|
|
// 13b: executeWithFallback engine behaviour
|
|
// 13c: First-chunk safety (buffering semantics at D9)
|
|
// 13d: Soft trigger skipping spawn
|
|
// 13e: Header annotation (providerUsed / modelUsed / fallbackHops / originalError)
|
|
// 13f: HTTP integration with __setFallbackConfig test seam
|
|
//
|
|
// Authority: ADR 0004 — Fallback Engine Semantics and Safety
|
|
|
|
import {
|
|
evaluateHardTriggers,
|
|
evaluateSoftTriggers,
|
|
executeWithFallback,
|
|
buildDefaultChain,
|
|
loadFallbackConfigSync,
|
|
isClientError,
|
|
} from './lib/fallback/engine.mjs';
|
|
|
|
import {
|
|
createOlpServer,
|
|
__setFallbackConfig,
|
|
__resetFallbackConfig,
|
|
__clearCache,
|
|
__setAuthConfig,
|
|
__resetAuthConfig,
|
|
__setStreamingConfig,
|
|
__resetStreamingConfig,
|
|
__clearRecentErrors,
|
|
__snapshotRecentErrors,
|
|
__resetRequestCounters,
|
|
} from './server.mjs';
|
|
|
|
// ── Phase 2 / D45+D46 server-side default override ────────────────────────
|
|
// Override the production-off defaults so that existing pre-D45 HTTP
|
|
// integration tests (Suite 18 etc.) continue to pass:
|
|
// - allow_anonymous: true → /v1/* requests without Authorization → anonymous
|
|
// - owner_only_endpoints: [] → /health full payload (D46 trimming opt-out)
|
|
// - fallback_detail_header_policy: 'all' → X-OLP-Fallback-Detail emitted to
|
|
// all identities (D40 v0.1.1 behaviour)
|
|
// New Suite 20 (D45) + Suite 21 (D46) explicitly call __setAuthConfig per-case
|
|
// to exercise the production-default paths.
|
|
__setAuthConfig({
|
|
allow_anonymous: true,
|
|
owner_only_endpoints: [],
|
|
fallback_detail_header_policy: 'all',
|
|
});
|
|
|
|
// ── 13a: Trigger taxonomy ────────────────────────────────────────────────
|
|
|
|
describe('Fallback engine — trigger taxonomy (D9)', () => {
|
|
|
|
// ── evaluateHardTriggers ─────────────────────────────────────────────
|
|
|
|
it('evaluateHardTriggers: HTTP 500 → fires (5xx hard trigger)', () => {
|
|
const err = Object.assign(new Error('Server error'), { statusCode: 500 });
|
|
assert.equal(evaluateHardTriggers(err), true);
|
|
});
|
|
|
|
it('evaluateHardTriggers: HTTP 503 → fires (5xx hard trigger)', () => {
|
|
const err = Object.assign(new Error('Service unavailable'), { statusCode: 503 });
|
|
assert.equal(evaluateHardTriggers(err), true);
|
|
});
|
|
|
|
it('evaluateHardTriggers: HTTP 400 → does NOT fire (client error)', () => {
|
|
const err = Object.assign(new Error('Bad request'), { statusCode: 400 });
|
|
assert.equal(evaluateHardTriggers(err), false);
|
|
});
|
|
|
|
it('evaluateHardTriggers: HTTP 401 → does NOT fire (client error)', () => {
|
|
const err = Object.assign(new Error('Unauthorized'), { statusCode: 401 });
|
|
assert.equal(evaluateHardTriggers(err), false);
|
|
});
|
|
|
|
it('evaluateHardTriggers: HTTP 403 → does NOT fire (client error)', () => {
|
|
const err = Object.assign(new Error('Forbidden'), { statusCode: 403 });
|
|
assert.equal(evaluateHardTriggers(err), false);
|
|
});
|
|
|
|
it('evaluateHardTriggers: HTTP 404 → does NOT fire (client error)', () => {
|
|
const err = Object.assign(new Error('Not found'), { statusCode: 404 });
|
|
assert.equal(evaluateHardTriggers(err), false);
|
|
});
|
|
|
|
it('evaluateHardTriggers: HTTP 422 → does NOT fire (client error)', () => {
|
|
const err = Object.assign(new Error('Unprocessable'), { statusCode: 422 });
|
|
assert.equal(evaluateHardTriggers(err), false);
|
|
});
|
|
|
|
it('evaluateHardTriggers: HTTP 429 → fires (quota-like 4xx not in client-error set)', () => {
|
|
const err = Object.assign(new Error('Too Many Requests'), { statusCode: 429 });
|
|
assert.equal(evaluateHardTriggers(err), true);
|
|
});
|
|
|
|
// QUOTA_EXHAUSTED and RATE_LIMITED tests removed (D34 F7): those codes were
|
|
// removed from PROVIDER_ERROR_CODES and HARD_TRIGGER_CODES — no v0.1 plugin
|
|
// emits them. Re-add tests via ADR 0004 amendment when HTTP-status parsing lands.
|
|
|
|
it('evaluateHardTriggers: ProviderError SPAWN_FAILED → fires', () => {
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
assert.equal(evaluateHardTriggers(err), true);
|
|
});
|
|
|
|
it('evaluateHardTriggers: ProviderError CLI_NOT_FOUND → fires', () => {
|
|
const err = new ProviderError('CLI not found', 'CLI_NOT_FOUND');
|
|
assert.equal(evaluateHardTriggers(err), true);
|
|
});
|
|
|
|
it('evaluateHardTriggers: ProviderError AUTH_MISSING → does NOT fire (user must fix)', () => {
|
|
const err = new ProviderError('Auth missing', 'AUTH_MISSING');
|
|
assert.equal(evaluateHardTriggers(err), false);
|
|
});
|
|
|
|
// D32 F4: OUTPUT_PARSE_ERROR removed from PROVIDER_ERROR_CODES and
|
|
// HARD_TRIGGER_CODES — no plugin emits it; it was dead code. Test removed.
|
|
|
|
it('evaluateHardTriggers: generic Error with no statusCode → does NOT fire', () => {
|
|
const err = new Error('Something went wrong');
|
|
assert.equal(evaluateHardTriggers(err), false);
|
|
});
|
|
|
|
it('evaluateHardTriggers: null error → does NOT fire', () => {
|
|
assert.equal(evaluateHardTriggers(null), false);
|
|
});
|
|
|
|
// ── evaluateSoftTriggers ─────────────────────────────────────────────
|
|
|
|
it('evaluateSoftTriggers: credit_pool_percent_threshold at exactly threshold → fires', () => {
|
|
const cfg = { credit_pool_percent_threshold: 90 };
|
|
const quota = { percentUsed: 90 };
|
|
assert.equal(evaluateSoftTriggers(cfg, quota), true);
|
|
});
|
|
|
|
it('evaluateSoftTriggers: credit_pool_percent_threshold above threshold → fires', () => {
|
|
const cfg = { credit_pool_percent_threshold: 90 };
|
|
const quota = { percentUsed: 95 };
|
|
assert.equal(evaluateSoftTriggers(cfg, quota), true);
|
|
});
|
|
|
|
it('evaluateSoftTriggers: credit_pool_percent_threshold below threshold → does NOT fire', () => {
|
|
const cfg = { credit_pool_percent_threshold: 90 };
|
|
const quota = { percentUsed: 89 };
|
|
assert.equal(evaluateSoftTriggers(cfg, quota), false);
|
|
});
|
|
|
|
it('evaluateSoftTriggers: daily_request_count_threshold at threshold → fires', () => {
|
|
const cfg = { daily_request_count_threshold: 500 };
|
|
const quota = { dailyCount: 500 };
|
|
assert.equal(evaluateSoftTriggers(cfg, quota), true);
|
|
});
|
|
|
|
it('evaluateSoftTriggers: daily_request_count_threshold below threshold → does NOT fire', () => {
|
|
const cfg = { daily_request_count_threshold: 500 };
|
|
const quota = { dailyCount: 499 };
|
|
assert.equal(evaluateSoftTriggers(cfg, quota), false);
|
|
});
|
|
|
|
it('evaluateSoftTriggers: five_hour_window_percent_threshold at threshold → fires', () => {
|
|
const cfg = { five_hour_window_percent_threshold: 85 };
|
|
const quota = { fiveHourWindowPercent: 85 };
|
|
assert.equal(evaluateSoftTriggers(cfg, quota), true);
|
|
});
|
|
|
|
it('evaluateSoftTriggers: null quotaSnapshot → does NOT fire (graceful degrade)', () => {
|
|
const cfg = { credit_pool_percent_threshold: 90 };
|
|
assert.equal(evaluateSoftTriggers(cfg, null), false);
|
|
});
|
|
|
|
it('evaluateSoftTriggers: undefined quotaSnapshot → does NOT fire (graceful degrade)', () => {
|
|
const cfg = { credit_pool_percent_threshold: 90 };
|
|
assert.equal(evaluateSoftTriggers(cfg, undefined), false);
|
|
});
|
|
|
|
it('evaluateSoftTriggers: null triggerConfig → does NOT fire', () => {
|
|
const quota = { percentUsed: 95 };
|
|
assert.equal(evaluateSoftTriggers(null, quota), false);
|
|
});
|
|
|
|
it('evaluateSoftTriggers: empty triggerConfig → does NOT fire', () => {
|
|
const quota = { percentUsed: 95 };
|
|
assert.equal(evaluateSoftTriggers({}, quota), false);
|
|
});
|
|
|
|
// ── isClientError ────────────────────────────────────────────────────
|
|
|
|
it('isClientError: 400 → true', () => { assert.equal(isClientError(400), true); });
|
|
it('isClientError: 401 → true', () => { assert.equal(isClientError(401), true); });
|
|
it('isClientError: 403 → true', () => { assert.equal(isClientError(403), true); });
|
|
it('isClientError: 404 → true', () => { assert.equal(isClientError(404), true); });
|
|
it('isClientError: 422 → true', () => { assert.equal(isClientError(422), true); });
|
|
it('isClientError: 429 → false (quota, not client error)', () => { assert.equal(isClientError(429), false); });
|
|
it('isClientError: 500 → false', () => { assert.equal(isClientError(500), false); });
|
|
|
|
});
|
|
|
|
// ── 13b: executeWithFallback engine ─────────────────────────────────────
|
|
|
|
describe('Fallback engine — executeWithFallback (D9)', () => {
|
|
|
|
// helper: build a mock executeHopFn that succeeds or throws per provider name
|
|
function makeHopFn(outcomes) {
|
|
// outcomes: { [provider]: 'success' | Error }
|
|
return async function (provider, model, _ir) {
|
|
const outcome = outcomes[provider];
|
|
if (!outcome || outcome === 'success') {
|
|
return [{ type: 'delta', role: 'assistant', content: `response from ${provider}` },
|
|
{ type: 'stop', finish_reason: 'stop' }];
|
|
}
|
|
throw outcome;
|
|
};
|
|
}
|
|
|
|
const dummyIR = makeIR({ model: 'test-model' });
|
|
|
|
it('empty chain → throws Error', async () => {
|
|
let caught = null;
|
|
try {
|
|
await executeWithFallback([], dummyIR, makeHopFn({}));
|
|
} catch (e) {
|
|
caught = e;
|
|
}
|
|
assert.ok(caught instanceof Error, 'Expected Error for empty chain');
|
|
assert.ok(caught.message.includes('chain'));
|
|
});
|
|
|
|
it('single-hop chain with success → returns chunks + fallbackHops=0', async () => {
|
|
const chain = [{ provider: 'anthropic', model: 'claude-sonnet-4-6' }];
|
|
const result = await executeWithFallback(chain, dummyIR, makeHopFn({ anthropic: 'success' }));
|
|
assert.ok(Array.isArray(result.chunks), 'Expected chunks array');
|
|
assert.equal(result.fallbackHops, 0);
|
|
assert.equal(result.providerUsed, 'anthropic');
|
|
assert.equal(result.originalError, null);
|
|
assert.deepEqual(result.triedProviders, ['anthropic']);
|
|
});
|
|
|
|
it('single-hop chain with hard-triggered error → exhausted, returns originalError', async () => {
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const chain = [{ provider: 'anthropic', model: 'claude-sonnet-4-6' }];
|
|
const result = await executeWithFallback(chain, dummyIR, makeHopFn({ anthropic: err }));
|
|
assert.equal(result.chunks, null, 'Expected null chunks on exhausted chain');
|
|
assert.equal(result.originalError, err);
|
|
assert.equal(result.fallbackHops, 1); // chain.length = 1, all exhausted
|
|
assert.deepEqual(result.triedProviders, ['anthropic']);
|
|
});
|
|
|
|
it('two-hop chain, primary fails with hard trigger → falls back to secondary, fallbackHops=1', async () => {
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const result = await executeWithFallback(chain, dummyIR, makeHopFn({ anthropic: err, openai: 'success' }));
|
|
assert.ok(Array.isArray(result.chunks), 'Expected chunks from secondary');
|
|
assert.equal(result.fallbackHops, 1);
|
|
assert.equal(result.providerUsed, 'openai');
|
|
assert.equal(result.originalError, null);
|
|
assert.deepEqual(result.triedProviders, ['anthropic', 'openai']);
|
|
});
|
|
|
|
it('three-hop chain, primary + secondary fail → tertiary returns chunks, fallbackHops=2', async () => {
|
|
const errA = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const errB = Object.assign(new Error('Service unavailable'), { statusCode: 503 });
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
{ provider: 'mistral', model: 'devstral-2' },
|
|
];
|
|
const result = await executeWithFallback(
|
|
chain, dummyIR,
|
|
makeHopFn({ anthropic: errA, openai: errB, mistral: 'success' }),
|
|
);
|
|
assert.ok(Array.isArray(result.chunks));
|
|
assert.equal(result.fallbackHops, 2);
|
|
assert.equal(result.providerUsed, 'mistral');
|
|
assert.equal(result.originalError, null);
|
|
assert.deepEqual(result.triedProviders, ['anthropic', 'openai', 'mistral']);
|
|
});
|
|
|
|
it('three-hop chain, all fail → exhausted, originalError is from FIRST hop (not last)', async () => {
|
|
const errA = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const errB = new ProviderError('Spawn timeout', 'SPAWN_TIMEOUT');
|
|
const errC = new ProviderError('CLI not found', 'CLI_NOT_FOUND');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
{ provider: 'mistral', model: 'devstral-2' },
|
|
];
|
|
const result = await executeWithFallback(
|
|
chain, dummyIR,
|
|
makeHopFn({ anthropic: errA, openai: errB, mistral: errC }),
|
|
);
|
|
assert.equal(result.chunks, null);
|
|
assert.equal(result.originalError, errA, 'Should be first hop error, not last');
|
|
assert.equal(result.fallbackHops, 3); // chain.length = 3
|
|
assert.deepEqual(result.triedProviders, ['anthropic', 'openai', 'mistral']);
|
|
});
|
|
|
|
it('client error (400) on primary → does NOT fall back, surfaces error immediately', async () => {
|
|
const err = Object.assign(new Error('Bad request'), { statusCode: 400 });
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
let openaiCalled = false;
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
openaiCalled = true;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
const result = await executeWithFallback(chain, dummyIR, hopFn);
|
|
assert.equal(result.chunks, null);
|
|
assert.equal(result.originalError, err);
|
|
assert.equal(openaiCalled, false, 'Second provider must NOT be called on client error');
|
|
assert.deepEqual(result.triedProviders, ['anthropic']);
|
|
});
|
|
|
|
it('AUTH_MISSING on primary → does NOT fall back (user must fix config)', async () => {
|
|
const err = new ProviderError('Auth missing', 'AUTH_MISSING');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
let openaiCalled = false;
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
openaiCalled = true;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
const result = await executeWithFallback(chain, dummyIR, hopFn);
|
|
assert.equal(result.chunks, null);
|
|
assert.equal(result.originalError, err);
|
|
assert.equal(openaiCalled, false, 'Second provider must NOT be called on AUTH_MISSING');
|
|
});
|
|
|
|
it('non-trigger error (generic) on primary → does NOT fall back', async () => {
|
|
const err = new Error('Unexpected internal error');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
let openaiCalled = false;
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
openaiCalled = true;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
const result = await executeWithFallback(chain, dummyIR, hopFn);
|
|
assert.equal(result.chunks, null);
|
|
assert.equal(openaiCalled, false, 'Must not fall back on non-trigger error');
|
|
});
|
|
|
|
});
|
|
|
|
// ── 13c: First-chunk safety ──────────────────────────────────────────────
|
|
|
|
describe('Fallback engine — first-chunk safety (D9)', () => {
|
|
|
|
// At D9, executeHopFn = collectAllChunks() which is fully buffered.
|
|
// Safety is enforced by construction: if executeHopFn returns successfully,
|
|
// chunks are already in memory (none written to res yet). Once chunks are
|
|
// returned, fallback is no longer attempted — we return immediately.
|
|
// If executeHopFn throws, zero bytes went to the client.
|
|
|
|
it('successful executeHopFn returns chunks immediately without retrying secondary', async () => {
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
let callCount = 0;
|
|
const hopFn = async (_provider) => {
|
|
callCount++;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
const result = await executeWithFallback(chain, makeIR({ model: 'test' }), hopFn);
|
|
assert.equal(result.fallbackHops, 0, 'Primary served — no fallback');
|
|
assert.equal(callCount, 1, 'executeHopFn called exactly once (first-chunk safety)');
|
|
});
|
|
|
|
it('error from executeHopFn (hard trigger) means zero chunks emitted — advance is safe', async () => {
|
|
// Simulate: anthropic throws (no bytes emitted), openai succeeds
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
let writtenToClient = false;
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') {
|
|
// Throw BEFORE any "write" — simulates first-chunk rule
|
|
throw err;
|
|
}
|
|
// openai succeeds — simulate "writtenToClient" only AFTER returning
|
|
// (in real server, chunks are written after executeWithFallback returns)
|
|
writtenToClient = true;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
const result = await executeWithFallback(chain, makeIR({ model: 'test' }), hopFn);
|
|
// openai served, which means writtenToClient is set (simulates post-return write)
|
|
assert.equal(result.fallbackHops, 1, 'Fell back to openai');
|
|
// Crucially: the "write" happened AFTER executeHopFn returned (not during chain iteration)
|
|
assert.equal(writtenToClient, true, 'Client write happens post-return, not during chain iteration');
|
|
});
|
|
|
|
});
|
|
|
|
// ── 13d: Soft triggers skipping spawn ────────────────────────────────────
|
|
|
|
describe('Fallback engine — soft trigger skipping (D9)', () => {
|
|
|
|
const dummyIR = makeIR({ model: 'test-model' });
|
|
|
|
it('chain [A, B], A soft trigger fires → engine skips A entirely, calls B; fallbackHops=1', async () => {
|
|
let aCalled = false;
|
|
let bCalled = false;
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'a') { aCalled = true; return [{ type: 'stop', finish_reason: 'stop' }]; }
|
|
if (provider === 'b') { bCalled = true; return [{ type: 'stop', finish_reason: 'stop' }]; }
|
|
throw new Error('Unexpected provider');
|
|
};
|
|
const chain = [
|
|
{
|
|
provider: 'a',
|
|
model: 'model-a',
|
|
softTriggers: { credit_pool_percent_threshold: 90 },
|
|
quotaSnapshot: { percentUsed: 95 }, // fires the trigger
|
|
},
|
|
{ provider: 'b', model: 'model-b' },
|
|
];
|
|
const result = await executeWithFallback(chain, dummyIR, hopFn);
|
|
assert.equal(aCalled, false, 'A must NOT be called (soft trigger skipped it)');
|
|
assert.equal(bCalled, true, 'B must be called');
|
|
assert.equal(result.fallbackHops, 1, 'B served as fallback hop 1');
|
|
assert.equal(result.providerUsed, 'b');
|
|
});
|
|
|
|
it('chain [A, B], neither soft trigger fires → A is called; if A succeeds, B never called', async () => {
|
|
let aCalled = false;
|
|
let bCalled = false;
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'a') { aCalled = true; return [{ type: 'stop', finish_reason: 'stop' }]; }
|
|
if (provider === 'b') { bCalled = true; return [{ type: 'stop', finish_reason: 'stop' }]; }
|
|
throw new Error('Unexpected provider');
|
|
};
|
|
const chain = [
|
|
{
|
|
provider: 'a',
|
|
model: 'model-a',
|
|
softTriggers: { credit_pool_percent_threshold: 90 },
|
|
quotaSnapshot: { percentUsed: 85 }, // below threshold, trigger does NOT fire
|
|
},
|
|
{ provider: 'b', model: 'model-b' },
|
|
];
|
|
const result = await executeWithFallback(chain, dummyIR, hopFn);
|
|
assert.equal(aCalled, true, 'A must be called');
|
|
assert.equal(bCalled, false, 'B must NOT be called (A succeeded)');
|
|
assert.equal(result.fallbackHops, 0);
|
|
assert.equal(result.providerUsed, 'a');
|
|
});
|
|
|
|
it('chain [A], A soft trigger fires (null quotaSnapshot) → trigger does NOT fire, A called', async () => {
|
|
// Per ADR 0004: null quotaStatus → treat as "don't fire"
|
|
let aCalled = false;
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'a') { aCalled = true; return [{ type: 'stop', finish_reason: 'stop' }]; }
|
|
throw new Error('Unexpected provider');
|
|
};
|
|
const chain = [
|
|
{
|
|
provider: 'a',
|
|
model: 'model-a',
|
|
softTriggers: { credit_pool_percent_threshold: 90 },
|
|
quotaSnapshot: null, // null → trigger never fires
|
|
},
|
|
];
|
|
const result = await executeWithFallback(chain, dummyIR, hopFn);
|
|
assert.equal(aCalled, true, 'A must be called (null quota → trigger does not fire)');
|
|
assert.equal(result.fallbackHops, 0);
|
|
});
|
|
|
|
});
|
|
|
|
// ── 13e: Header annotation ───────────────────────────────────────────────
|
|
|
|
describe('Fallback engine — observability / header annotation (D9)', () => {
|
|
|
|
const dummyIR = makeIR({ model: 'test-model' });
|
|
|
|
it('success on primary: providerUsed + modelUsed match primary hop', async () => {
|
|
const chain = [{ provider: 'anthropic', model: 'claude-sonnet-4-6' }];
|
|
const result = await executeWithFallback(chain, dummyIR, async () => [{ type: 'stop', finish_reason: 'stop' }]);
|
|
assert.equal(result.providerUsed, 'anthropic');
|
|
assert.equal(result.modelUsed, 'claude-sonnet-4-6');
|
|
});
|
|
|
|
it('success on fallback: providerUsed + modelUsed match the serving hop', async () => {
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
const result = await executeWithFallback(chain, dummyIR, hopFn);
|
|
assert.equal(result.providerUsed, 'openai');
|
|
assert.equal(result.modelUsed, 'gpt-5.5');
|
|
assert.equal(result.fallbackHops, 1);
|
|
});
|
|
|
|
it('chain exhausted: originalError is from FIRST hop, not second or third', async () => {
|
|
const errA = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const errB = new ProviderError('CLI not found', 'CLI_NOT_FOUND');
|
|
const errC = new ProviderError('Spawn timeout', 'SPAWN_TIMEOUT');
|
|
const chain = [
|
|
{ provider: 'a', model: 'model-a' },
|
|
{ provider: 'b', model: 'model-b' },
|
|
{ provider: 'c', model: 'model-c' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'a') throw errA;
|
|
if (provider === 'b') throw errB;
|
|
if (provider === 'c') throw errC;
|
|
throw new Error('Unexpected');
|
|
};
|
|
const result = await executeWithFallback(chain, dummyIR, hopFn);
|
|
assert.equal(result.originalError, errA, 'originalError must be first-hop error');
|
|
assert.notEqual(result.originalError, errB, 'Must NOT be second-hop error');
|
|
assert.notEqual(result.originalError, errC, 'Must NOT be third-hop error');
|
|
});
|
|
|
|
it('triedProviders lists all attempted hops in chain order', async () => {
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const chain = [
|
|
{ provider: 'a', model: 'model-a' },
|
|
{ provider: 'b', model: 'model-b' },
|
|
{ provider: 'c', model: 'model-c' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'a') throw err;
|
|
if (provider === 'b') throw err;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
const result = await executeWithFallback(chain, dummyIR, hopFn);
|
|
assert.deepEqual(result.triedProviders, ['a', 'b', 'c']);
|
|
assert.equal(result.fallbackHops, 2);
|
|
assert.equal(result.providerUsed, 'c');
|
|
});
|
|
|
|
});
|
|
|
|
// ── 13f: HTTP integration ────────────────────────────────────────────────
|
|
|
|
describe('Fallback engine — HTTP integration (D9)', () => {
|
|
let server;
|
|
let port;
|
|
let savedAnthropicToken;
|
|
let savedCodexAuthPath;
|
|
let suiteCodexAuthFile;
|
|
|
|
before(async () => {
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
mistralResetSpawnImpl();
|
|
__resetFallbackConfig();
|
|
|
|
// CRITICAL — inject suite-level fake auth tokens for anthropic + codex.
|
|
// Without this, on hosts that lack the real auth artifacts (CI Linux
|
|
// runners are the load-bearing case; they have no macOS keychain and no
|
|
// ~/.claude/.credentials.json), the provider plugins throw
|
|
// ProviderError(AUTH_MISSING) BEFORE the mock spawn function fires. The
|
|
// chain then stops at the first hop because AUTH_MISSING is NOT a hard
|
|
// trigger (per ADR 0004 § Decision § No fallback for client-side
|
|
// errors), so the fallback path under test never executes. Bug
|
|
// discovered on D9 CI 2026-05-23: 3 tests in this suite failed on
|
|
// Node 20 Linux while passing on macOS Node 25 because the macOS
|
|
// keychain provided real anthropic auth and the test never reached
|
|
// the fake-auth-required code path.
|
|
savedAnthropicToken = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-anthropic-token-for-d9-http-suite';
|
|
|
|
const { writeFileSync } = await import('node:fs');
|
|
const { tmpdir } = await import('node:os');
|
|
const { join: pathJoin } = await import('node:path');
|
|
suiteCodexAuthFile = pathJoin(tmpdir(), `olp-test-d9-codex-auth-${Date.now()}.json`);
|
|
writeFileSync(suiteCodexAuthFile, JSON.stringify({ accessToken: 'fake-codex-token-for-d9-http-suite' }), 'utf8');
|
|
savedCodexAuthPath = process.env.OPENAI_CODEX_AUTH_PATH;
|
|
process.env.OPENAI_CODEX_AUTH_PATH = suiteCodexAuthFile;
|
|
|
|
// Start test server with all 3 providers enabled
|
|
const testProviders = loadProviders({ enabled: { anthropic: true, openai: true, mistral: true } });
|
|
const { loadedProviders: lp, cacheStore: cs } = await import('./server.mjs');
|
|
// Inject test providers
|
|
for (const [name, p] of testProviders) {
|
|
lp.set(name, p);
|
|
}
|
|
cs.clear();
|
|
|
|
server = createOlpServer();
|
|
port = 20456 + Math.floor(Math.random() * 500);
|
|
await new Promise((resolve, reject) => {
|
|
server.listen(port, '127.0.0.1', resolve);
|
|
server.once('error', (e) => {
|
|
if (e.code === 'EADDRINUSE') {
|
|
port++;
|
|
server.listen(port, '127.0.0.1', resolve);
|
|
server.once('error', reject);
|
|
} else reject(e);
|
|
});
|
|
});
|
|
});
|
|
|
|
after(async () => {
|
|
__resetFallbackConfig();
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
mistralResetSpawnImpl();
|
|
|
|
// Restore env vars injected by before()
|
|
if (savedAnthropicToken !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedAnthropicToken;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
if (savedCodexAuthPath !== undefined) {
|
|
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexAuthPath;
|
|
} else {
|
|
delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
}
|
|
if (suiteCodexAuthFile) {
|
|
const { unlinkSync } = await import('node:fs');
|
|
try { unlinkSync(suiteCodexAuthFile); } catch { /* ignore */ }
|
|
}
|
|
|
|
if (!server) return;
|
|
return new Promise(r => server.close(r));
|
|
});
|
|
|
|
// Shared mock spawn builder that returns a successful response
|
|
function makeSuccessSpawn(content = 'Hello from mock') {
|
|
return function mockSpawnImpl(_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
// Emit a minimal SSE-like response that anthropic plugin parses
|
|
// (Anthropic mock format: JSON lines per the real anthropic.mjs parser)
|
|
const lines = [
|
|
JSON.stringify({ type: 'content_block_delta', index: 0, delta: { type: 'text_delta', text: content } }),
|
|
JSON.stringify({ type: 'message_stop' }),
|
|
];
|
|
for (const line of lines) {
|
|
proc.stdout.emit('data', Buffer.from(`data: ${line}\n\n`));
|
|
}
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
};
|
|
}
|
|
|
|
it('no fallback config: POST with no enabled provider → 503', async () => {
|
|
// Remove all providers temporarily
|
|
const { loadedProviders: lp } = await import('./server.mjs');
|
|
const savedMap = new Map(lp);
|
|
lp.clear();
|
|
try {
|
|
const r = await fetch({
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: { model: 'unknown-model', messages: [{ role: 'user', content: 'Hi' }] },
|
|
});
|
|
assert.equal(r.status, 503);
|
|
} finally {
|
|
for (const [name, p] of savedMap) lp.set(name, p);
|
|
}
|
|
});
|
|
|
|
it('no fallback config + mock anthropic: POST claude-sonnet-4-6 → 200 + X-OLP-Fallback-Hops: 0', async () => {
|
|
__setSpawnImpl(makeSuccessSpawn('Test response'));
|
|
try {
|
|
const r = await fetch({
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
max_tokens: 10,
|
|
},
|
|
});
|
|
assert.equal(r.status, 200, `Expected 200, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
assert.equal(r.headers['x-olp-fallback-hops'], '0');
|
|
assert.equal(r.headers['x-olp-provider-used'], 'anthropic');
|
|
} finally {
|
|
__resetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('fallback config: mock anthropic fails (SPAWN_FAILED), chain [anthropic→openai], openai mock succeeds → 200 + X-OLP-Fallback-Hops: 1 + X-OLP-Provider-Used: openai', async () => {
|
|
// Clear cache to prevent cache-hit from previous test polluting this one.
|
|
// ADR 0005: each (provider, model) pair is independently cached; a hit
|
|
// from the previous test would cause executeHopFn to skip the spawn and
|
|
// return cached chunks, masking the SPAWN_FAILED mock.
|
|
__clearCache();
|
|
|
|
// Provide fake codex auth so AUTH_MISSING doesn't stop the chain before
|
|
// the codex mock can respond. Codex checks readAuthArtifact() before spawnImpl.
|
|
// Write a temp file and set OPENAI_CODEX_AUTH_PATH to point at it.
|
|
const { writeFileSync, unlinkSync } = await import('node:fs');
|
|
const { tmpdir } = await import('node:os');
|
|
const { join: pathJoin } = await import('node:path');
|
|
const tmpAuthFile = pathJoin(tmpdir(), `olp-test-codex-auth-${Date.now()}.json`);
|
|
writeFileSync(tmpAuthFile, JSON.stringify({ accessToken: 'fake-test-token-for-codex' }), 'utf8');
|
|
const savedCodexAuthPath = process.env.OPENAI_CODEX_AUTH_PATH;
|
|
process.env.OPENAI_CODEX_AUTH_PATH = tmpAuthFile;
|
|
|
|
// Set up a 2-hop chain: anthropic → openai for claude-sonnet-4-6
|
|
__setFallbackConfig({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
});
|
|
|
|
// Anthropic mock: always exits with code 1 → SPAWN_FAILED (hard trigger)
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stderr.emit('data', Buffer.from('spawn failed\n'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 1, null); // non-zero exit → SPAWN_FAILED
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
// Codex (openai) mock: succeeds with a delta+stop response
|
|
codexSetSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('data', Buffer.from('{"content":"Fallback response from openai"}\n'));
|
|
proc.stdout.emit('data', Buffer.from('{"type":"stop"}\n'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
max_tokens: 10,
|
|
},
|
|
});
|
|
assert.equal(r.status, 200, `Expected 200 on fallback, got ${r.status}: ${r.body.slice(0, 300)}`);
|
|
assert.equal(r.headers['x-olp-fallback-hops'], '1', `Expected fallback-hops=1, got: ${r.headers['x-olp-fallback-hops']}`);
|
|
assert.equal(r.headers['x-olp-provider-used'], 'openai', `Expected provider-used=openai, got: ${r.headers['x-olp-provider-used']}`);
|
|
} finally {
|
|
__resetFallbackConfig();
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
if (savedCodexAuthPath !== undefined) {
|
|
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexAuthPath;
|
|
} else {
|
|
delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
}
|
|
try { unlinkSync(tmpAuthFile); } catch { /* ignore */ }
|
|
}
|
|
});
|
|
|
|
it('fallback config: two-hop chain with both providers failing SPAWN_FAILED → exhausted-chain path → error response with X-OLP-Fallback-Exhausted header', async () => {
|
|
// NOTE on this test's renaming (D9 review-2): the original sonnet draft
|
|
// titled this as a "client error (400) → no fallback" HTTP test. But the
|
|
// anthropic plugin surfaces non-zero exit codes as ProviderError
|
|
// SPAWN_FAILED (a HARD trigger), not as a typed-400 client error. So at
|
|
// the HTTP integration layer we can't easily inject a synthetic 400
|
|
// without monkey-patching executeHopFn. The "client error stops chain
|
|
// immediately" semantic IS covered at the UNIT level in
|
|
// "Engine: two-hop chain primary client error 400 → secondary NOT
|
|
// called" (line 3289+ in this file). This HTTP test now exercises the
|
|
// complementary path: both hops fail with SPAWN_FAILED → chain
|
|
// exhausted → originalError surfaces with X-OLP-Fallback-Exhausted
|
|
// header listing both providers.
|
|
__clearCache();
|
|
|
|
// Set up a 2-hop chain
|
|
__setFallbackConfig({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
});
|
|
|
|
// Both providers fail with SPAWN_FAILED
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 1, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
codexSetSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 1, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
max_tokens: 10,
|
|
},
|
|
});
|
|
// Both providers fail → exhausted chain → 502 error response with exhausted header
|
|
assert.ok(r.status >= 400 && r.status < 600, `Expected error status, got ${r.status}`);
|
|
// X-OLP-Fallback-Exhausted header should be present since both providers tried
|
|
assert.ok(
|
|
r.headers['x-olp-fallback-exhausted'] !== undefined,
|
|
`Expected X-OLP-Fallback-Exhausted header, got headers: ${JSON.stringify(r.headers)}`,
|
|
);
|
|
} finally {
|
|
__resetFallbackConfig();
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
// ── D16 tests: SPAWN_FAILED partial-response salvage (ADR 0004 Amendment 1) ─
|
|
|
|
it('D16/Case-A regression guard: SPAWN_FAILED with no chunks → hard fallback fires → openai serves (2-hop)', async () => {
|
|
// Case A: provider exits non-zero BEFORE emitting any content.
|
|
// Fallback engine must still advance the chain — this is the pre-D16 behavior
|
|
// that must remain unchanged. Emitting zero chunks then exit-1 must still trigger
|
|
// the hard fallback.
|
|
__clearCache();
|
|
|
|
const { writeFileSync, unlinkSync } = await import('node:fs');
|
|
const { tmpdir } = await import('node:os');
|
|
const { join: pathJoin } = await import('node:path');
|
|
const tmpAuthFile = pathJoin(tmpdir(), `olp-test-d16-caseA-auth-${Date.now()}.json`);
|
|
writeFileSync(tmpAuthFile, JSON.stringify({ accessToken: 'fake-d16-caseA-codex-token' }), 'utf8');
|
|
const savedCodexAuthPath = process.env.OPENAI_CODEX_AUTH_PATH;
|
|
process.env.OPENAI_CODEX_AUTH_PATH = tmpAuthFile;
|
|
|
|
__setFallbackConfig({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
});
|
|
|
|
// Anthropic mock: emits NO stdout, exits with code 1 → SPAWN_FAILED with chunks=[]
|
|
// This is Case A — hard trigger must fire, chain must advance.
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
// No stdout data emitted before close
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 1, null); // exit 1 → SPAWN_FAILED, no usable chunks
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
// OpenAI (codex) mock: succeeds with a real response
|
|
codexSetSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('data', Buffer.from('{"content":"D16-caseA-fallback-openai-response"}\n'));
|
|
proc.stdout.emit('data', Buffer.from('{"type":"stop"}\n'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
max_tokens: 10,
|
|
},
|
|
});
|
|
assert.equal(r.status, 200, `Expected 200 from openai fallback, got ${r.status}: ${r.body.slice(0, 300)}`);
|
|
// Hard trigger fired → chain advanced → openai served → hops=1
|
|
assert.equal(r.headers['x-olp-fallback-hops'], '1', `Expected fallback-hops=1 (Case A hard trigger), got: ${r.headers['x-olp-fallback-hops']}`);
|
|
assert.equal(r.headers['x-olp-provider-used'], 'openai', `Expected openai served, got: ${r.headers['x-olp-provider-used']}`);
|
|
} finally {
|
|
__resetFallbackConfig();
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
if (savedCodexAuthPath !== undefined) {
|
|
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexAuthPath;
|
|
} else {
|
|
delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
}
|
|
try { unlinkSync(tmpAuthFile); } catch { /* ignore */ }
|
|
}
|
|
});
|
|
|
|
it('D16/Case-B fix: SPAWN_FAILED after N chunks → fallback does NOT fire → anthropic partial response served (2-hop)', async () => {
|
|
// Case B (the D16 fix): provider emits content then exits non-zero.
|
|
// The partial chunks are usable — fallback must NOT advance to openai.
|
|
// Instead, the partial response is returned with finish_reason='length'.
|
|
// X-OLP-Fallback-Hops must be 0 (anthropic served, openai not called).
|
|
__clearCache();
|
|
|
|
const { writeFileSync, unlinkSync } = await import('node:fs');
|
|
const { tmpdir } = await import('node:os');
|
|
const { join: pathJoin } = await import('node:path');
|
|
const tmpAuthFile = pathJoin(tmpdir(), `olp-test-d16-caseB-auth-${Date.now()}.json`);
|
|
writeFileSync(tmpAuthFile, JSON.stringify({ accessToken: 'fake-d16-caseB-codex-token' }), 'utf8');
|
|
const savedCodexAuthPath = process.env.OPENAI_CODEX_AUTH_PATH;
|
|
process.env.OPENAI_CODEX_AUTH_PATH = tmpAuthFile;
|
|
|
|
let openaiMockCalled = false;
|
|
|
|
__setFallbackConfig({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
});
|
|
|
|
// Anthropic mock: emits 2 raw text chunks then exits non-zero (cleanup error).
|
|
// The anthropic plugin processes raw stdout as text → IR delta chunks.
|
|
// exit code 1 after yielding content → Case B: SPAWN_FAILED with chunks.length>0.
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
// Emit 2 content chunks as raw text (anthropic --output-format text)
|
|
proc.stdout.emit('data', Buffer.from('Hello '));
|
|
proc.stdout.emit('data', Buffer.from('world'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
// Non-zero exit AFTER emitting content → SPAWN_FAILED, chunks.length=2
|
|
proc.emit('close', 1, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
// OpenAI mock: should NOT be called (D16 must prevent fallback here)
|
|
codexSetSpawnImpl(function (_bin, _args, _opts) {
|
|
openaiMockCalled = true;
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('data', Buffer.from('{"content":"unexpected-openai-response"}\n'));
|
|
proc.stdout.emit('data', Buffer.from('{"type":"stop"}\n'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
max_tokens: 10,
|
|
},
|
|
});
|
|
// D16: anthropic partial response surfaced instead of falling back
|
|
assert.equal(r.status, 200, `Expected 200 with partial anthropic response, got ${r.status}: ${r.body.slice(0, 300)}`);
|
|
// No fallback should have fired — anthropic's partial chunks were usable
|
|
assert.equal(r.headers['x-olp-fallback-hops'], '0', `Expected fallback-hops=0 (no fallback, Case B), got: ${r.headers['x-olp-fallback-hops']}`);
|
|
assert.equal(r.headers['x-olp-provider-used'], 'anthropic', `Expected anthropic served, got: ${r.headers['x-olp-provider-used']}`);
|
|
// OpenAI mock must NOT have been invoked
|
|
assert.equal(openaiMockCalled, false, 'OpenAI mock should NOT have been called (D16 prevents fallback on Case B)');
|
|
// Verify finish_reason='length' (synthesized stop from D16 salvage path)
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.choices?.[0]?.finish_reason, 'length', `Expected finish_reason='length' from D16 salvage, got: ${body.choices?.[0]?.finish_reason}`);
|
|
// Verify the 2 content chunks are present in the response
|
|
const content = body.choices?.[0]?.message?.content ?? '';
|
|
assert.ok(content.includes('Hello'), `Expected 'Hello' in partial content, got: '${content}'`);
|
|
assert.ok(content.includes('world'), `Expected 'world' in partial content, got: '${content}'`);
|
|
} finally {
|
|
__resetFallbackConfig();
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
if (savedCodexAuthPath !== undefined) {
|
|
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexAuthPath;
|
|
} else {
|
|
delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
}
|
|
try { unlinkSync(tmpAuthFile); } catch { /* ignore */ }
|
|
}
|
|
});
|
|
|
|
it('D16/Case-B single-hop: SPAWN_FAILED after 1 chunk → HTTP 200 with partial response + finish_reason=length (no fallback chain)', async () => {
|
|
// Case B single-hop variant: only one provider in the chain.
|
|
// Provider emits 1 content chunk then exits non-zero.
|
|
// D16 must surface the partial response (HTTP 200, not 502).
|
|
__clearCache();
|
|
|
|
__setFallbackConfig({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
});
|
|
|
|
// Anthropic mock: emits 1 raw text chunk then exits non-zero
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('data', Buffer.from('Partial answer'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 1, null); // exit 1 after content → Case B
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
max_tokens: 10,
|
|
},
|
|
});
|
|
// D16: must be 200 with the partial content, not 502
|
|
assert.equal(r.status, 200, `Expected 200 with partial response (not 502), got ${r.status}: ${r.body.slice(0, 300)}`);
|
|
assert.equal(r.headers['x-olp-fallback-hops'], '0', `Expected fallback-hops=0, got: ${r.headers['x-olp-fallback-hops']}`);
|
|
assert.equal(r.headers['x-olp-provider-used'], 'anthropic', `Expected anthropic served, got: ${r.headers['x-olp-provider-used']}`);
|
|
const body = JSON.parse(r.body);
|
|
// finish_reason='length' indicates truncation (D16 synthesized stop)
|
|
assert.equal(body.choices?.[0]?.finish_reason, 'length', `Expected finish_reason='length', got: ${body.choices?.[0]?.finish_reason}`);
|
|
// Partial content must be present
|
|
const content = body.choices?.[0]?.message?.content ?? '';
|
|
assert.ok(content.includes('Partial answer'), `Expected 'Partial answer' in content, got: '${content}'`);
|
|
} finally {
|
|
__resetFallbackConfig();
|
|
__resetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
// ── D39 (issue #3 Part 2): cache_evicted_truncated observability log ──
|
|
// Authority: D39 design — surface salvage frequency to dashboards. The log
|
|
// event fires in executeHopFn (server.mjs) immediately after the explicit
|
|
// cacheStore.delete() that replaces the prior set-with-TTL-0 tombstone.
|
|
// The event is level=info → routed to stdout per logEvent in server.mjs.
|
|
it('D39: cache_evicted_truncated log event fires on SPAWN_FAILED salvage with correct provider+model', async () => {
|
|
__clearCache();
|
|
|
|
__setFallbackConfig({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
});
|
|
|
|
// Reuse the D16 Case-B single-hop mock: emit 1 raw text chunk, then exit-1.
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('data', Buffer.from('D39-evict-log-content'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 1, null); // Case B salvage path
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
// Capture stdout writes to find the JSON log line emitted by logEvent.
|
|
const stdoutWrites = [];
|
|
const origStdoutWrite = process.stdout.write.bind(process.stdout);
|
|
process.stdout.write = (chunk, ...rest) => {
|
|
const s = typeof chunk === 'string'
|
|
? chunk
|
|
: (Buffer.isBuffer(chunk) ? chunk.toString() : String(chunk));
|
|
stdoutWrites.push(s);
|
|
return origStdoutWrite(chunk, ...rest);
|
|
};
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'D39 cache_evicted_truncated log' }],
|
|
max_tokens: 10,
|
|
},
|
|
});
|
|
assert.equal(r.status, 200, `Expected 200 from salvage, got ${r.status}`);
|
|
} finally {
|
|
process.stdout.write = origStdoutWrite;
|
|
__resetFallbackConfig();
|
|
__resetSpawnImpl();
|
|
}
|
|
|
|
// Find the cache_evicted_truncated event in the captured stdout.
|
|
const evictedLines = [];
|
|
for (const w of stdoutWrites) {
|
|
for (const line of w.split('\n')) {
|
|
if (!line.includes('"event":"cache_evicted_truncated"')) continue;
|
|
try {
|
|
const parsed = JSON.parse(line);
|
|
if (parsed?.event === 'cache_evicted_truncated') evictedLines.push(parsed);
|
|
} catch { /* not JSON — ignore */ }
|
|
}
|
|
}
|
|
assert.equal(evictedLines.length, 1,
|
|
`Expected exactly one cache_evicted_truncated event, got ${evictedLines.length}: ${JSON.stringify(evictedLines)}`);
|
|
assert.equal(evictedLines[0].provider, 'anthropic',
|
|
`Expected provider=anthropic, got ${evictedLines[0].provider}`);
|
|
assert.equal(evictedLines[0].model, 'claude-sonnet-4-6',
|
|
`Expected model=claude-sonnet-4-6, got ${evictedLines[0].model}`);
|
|
assert.equal(evictedLines[0].level, 'info',
|
|
`Expected level=info, got ${evictedLines[0].level}`);
|
|
});
|
|
|
|
// ── D39 (issue #3 Part 3): truncated response is not sticky-cached ────
|
|
// Defense-in-depth around the D16 eviction code path. Two consecutive
|
|
// identical buffered requests that both trigger SPAWN_FAILED-with-chunks
|
|
// (Case B salvage). The eviction in executeHopFn must ensure the second
|
|
// request is a FRESH spawn (not served from a sticky cache entry written
|
|
// by the singleflight populate during the first request's salvage). If the
|
|
// eviction were ever lost (e.g., the delete were silently dropped or the
|
|
// condition gate were wrong), this test would catch it: spawnCount would
|
|
// become 1 instead of 2 and X-OLP-Cache would be `hit` on r2.
|
|
it('D39: truncated response is not sticky-cached (two identical requests → two spawns)', async () => {
|
|
__clearCache();
|
|
|
|
__setFallbackConfig({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
});
|
|
|
|
let spawnCount = 0;
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
spawnCount++;
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
// Case B: emit partial content then exit non-zero on every spawn.
|
|
proc.stdout.emit('data', Buffer.from('D39-sticky-content'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 1, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
try {
|
|
// The two requests must be byte-identical so they share a cache key.
|
|
const reqBody = {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'D39 sticky-cache regression — identical body' }],
|
|
max_tokens: 10,
|
|
};
|
|
|
|
const r1 = await fetch({ port, method: 'POST', path: '/v1/chat/completions', body: reqBody });
|
|
assert.equal(r1.status, 200, `r1 must succeed via salvage, got ${r1.status}`);
|
|
assert.equal(r1.headers['x-olp-cache'], 'miss', `r1 must be cache miss, got ${r1.headers['x-olp-cache']}`);
|
|
|
|
const r2 = await fetch({ port, method: 'POST', path: '/v1/chat/completions', body: reqBody });
|
|
assert.equal(r2.status, 200, `r2 must succeed via salvage, got ${r2.status}`);
|
|
// The decisive assertions: a FRESH spawn must have fired for r2.
|
|
assert.equal(spawnCount, 2,
|
|
`Expected 2 spawns across two identical truncated requests (sticky-cache regression guard), got ${spawnCount}`);
|
|
assert.equal(r2.headers['x-olp-cache'], 'miss',
|
|
`r2 must also be cache miss (truncated salvage must not be sticky), got ${r2.headers['x-olp-cache']}`);
|
|
} finally {
|
|
__resetFallbackConfig();
|
|
__resetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
// ── D40 HTTP integration tests (issue #7): X-OLP-Fallback-Detail on the wire ─
|
|
|
|
it('D40: 1-hop chain succeeds → X-OLP-Fallback-Detail header is absent (no failures to report)', async () => {
|
|
__clearCache();
|
|
__setSpawnImpl(makeSuccessSpawn('D40 clean success'));
|
|
try {
|
|
const r = await fetch({
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
max_tokens: 10,
|
|
},
|
|
});
|
|
assert.equal(r.status, 200, `Expected 200, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
assert.equal(r.headers['x-olp-fallback-hops'], '0');
|
|
assert.equal(
|
|
r.headers['x-olp-fallback-detail'],
|
|
undefined,
|
|
`Header must be absent on clean primary success, got: ${r.headers['x-olp-fallback-detail']}`,
|
|
);
|
|
} finally {
|
|
__resetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('D40: 2-hop chain, both fail → response carries X-OLP-Fallback-Detail with 2 tuples', async () => {
|
|
__clearCache();
|
|
__setFallbackConfig({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
});
|
|
|
|
// Anthropic mock: exits non-zero with stderr message → SPAWN_FAILED
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stderr.emit('data', Buffer.from('anthropic-stderr\n'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 1, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
// Codex (openai) mock: also fails → exhausted chain
|
|
codexSetSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stderr.emit('data', Buffer.from('codex-stderr\n'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 1, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
max_tokens: 10,
|
|
},
|
|
});
|
|
assert.ok(r.status >= 400 && r.status < 600, `Expected error status, got ${r.status}`);
|
|
assert.ok(
|
|
r.headers['x-olp-fallback-detail'] !== undefined,
|
|
`Expected X-OLP-Fallback-Detail header on exhausted chain, headers: ${JSON.stringify(r.headers)}`,
|
|
);
|
|
const parsed = JSON.parse(r.headers['x-olp-fallback-detail']);
|
|
assert.ok(Array.isArray(parsed), 'Header must JSON-parse to array');
|
|
assert.equal(parsed.length, 2, `Expected 2 tuples on 2-hop exhaustion, got ${parsed.length}`);
|
|
assert.equal(parsed[0].hop, 0);
|
|
assert.equal(parsed[0].provider, 'anthropic');
|
|
assert.equal(parsed[0].code, 'SPAWN_FAILED');
|
|
assert.equal(parsed[0].trigger_type, 'hard');
|
|
assert.equal(parsed[1].hop, 1);
|
|
assert.equal(parsed[1].provider, 'openai');
|
|
} finally {
|
|
__resetFallbackConfig();
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('D40: 2-hop chain, primary fails + secondary succeeds → success response carries X-OLP-Fallback-Detail with 1 tuple (the failed primary)', async () => {
|
|
__clearCache();
|
|
|
|
const { writeFileSync, unlinkSync } = await import('node:fs');
|
|
const { tmpdir } = await import('node:os');
|
|
const { join: pathJoin } = await import('node:path');
|
|
const tmpAuthFile = pathJoin(tmpdir(), `olp-test-d40-success-with-detail-${Date.now()}.json`);
|
|
writeFileSync(tmpAuthFile, JSON.stringify({ accessToken: 'fake-d40-codex-token' }), 'utf8');
|
|
const savedCodexAuthPath = process.env.OPENAI_CODEX_AUTH_PATH;
|
|
process.env.OPENAI_CODEX_AUTH_PATH = tmpAuthFile;
|
|
|
|
__setFallbackConfig({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
});
|
|
|
|
// Anthropic mock: SPAWN_FAILED (no chunks)
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 1, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
// Codex (openai) mock: succeeds
|
|
codexSetSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('data', Buffer.from('{"content":"D40 success-after-fail"}\n'));
|
|
proc.stdout.emit('data', Buffer.from('{"type":"stop"}\n'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'Hi' }],
|
|
max_tokens: 10,
|
|
},
|
|
});
|
|
assert.equal(r.status, 200, `Expected 200 from fallback success, got ${r.status}: ${r.body.slice(0, 300)}`);
|
|
assert.equal(r.headers['x-olp-fallback-hops'], '1');
|
|
assert.equal(r.headers['x-olp-provider-used'], 'openai');
|
|
assert.ok(
|
|
r.headers['x-olp-fallback-detail'] !== undefined,
|
|
`Expected X-OLP-Fallback-Detail on success-with-prior-failure, got headers: ${JSON.stringify(r.headers)}`,
|
|
);
|
|
const parsed = JSON.parse(r.headers['x-olp-fallback-detail']);
|
|
assert.equal(parsed.length, 1, `Expected 1 tuple (the failed primary), got ${parsed.length}`);
|
|
assert.equal(parsed[0].hop, 0);
|
|
assert.equal(parsed[0].provider, 'anthropic');
|
|
assert.equal(parsed[0].code, 'SPAWN_FAILED');
|
|
assert.equal(parsed[0].trigger_type, 'hard');
|
|
} finally {
|
|
__resetFallbackConfig();
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
if (savedCodexAuthPath !== undefined) {
|
|
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexAuthPath;
|
|
} else {
|
|
delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
}
|
|
try { unlinkSync(tmpAuthFile); } catch { /* ignore */ }
|
|
}
|
|
});
|
|
|
|
});
|
|
|
|
// ── Suite 13g: D28 round-3 F2 — structured log observability fields ─────────
|
|
//
|
|
// Tests that all executeWithFallback logEvent calls carry the 4 new fields:
|
|
// chain_id, ir_request_hash, trigger_type, next_provider
|
|
//
|
|
// Also tests computeIRRequestHash (stability, provider-agnosticism, per-field sensitivity).
|
|
//
|
|
// Authority: ADR 0004 § Observability headers
|
|
|
|
import { computeIRRequestHash } from './lib/cache/keys.mjs';
|
|
|
|
describe('Fallback engine — D28 observability fields (chain_id / ir_request_hash / trigger_type / next_provider)', () => {
|
|
|
|
// ── Helper: capture all logEvent calls ────────────────────────────────────
|
|
|
|
function makeLogCapture() {
|
|
const events = [];
|
|
return {
|
|
logEvent: (level, event, data) => events.push({ level, event, data }),
|
|
events,
|
|
};
|
|
}
|
|
|
|
// ── chain_id: present + 16-char hex + consistent across all hops ──────────
|
|
|
|
it('chain_id is present in all events and is 16-char hex', async () => {
|
|
const { logEvent, events } = makeLogCapture();
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
await executeWithFallback(chain, makeIR({ model: 'test-model' }), hopFn, { logEvent });
|
|
assert.ok(events.length >= 2, 'Expected at least 2 log events');
|
|
for (const e of events) {
|
|
assert.ok(typeof e.data.chain_id === 'string', `chain_id must be string, event: ${e.event}`);
|
|
assert.match(e.data.chain_id, /^[0-9a-f]{16}$/, `chain_id must be 16-char hex, event: ${e.event}`);
|
|
}
|
|
});
|
|
|
|
it('chain_id is consistent across all hops from one executeWithFallback call', async () => {
|
|
const { logEvent, events } = makeLogCapture();
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const chain = [
|
|
{ provider: 'a', model: 'model-a' },
|
|
{ provider: 'b', model: 'model-b' },
|
|
{ provider: 'c', model: 'model-c' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'a') throw err;
|
|
if (provider === 'b') throw err;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
await executeWithFallback(chain, makeIR({ model: 'test-model' }), hopFn, { logEvent });
|
|
assert.ok(events.length >= 3, 'Expected at least 3 log events');
|
|
const ids = new Set(events.map(e => e.data.chain_id));
|
|
assert.equal(ids.size, 1, `All events in one call must share the same chain_id, got: ${[...ids]}`);
|
|
});
|
|
|
|
it('chain_id differs between separate executeWithFallback calls', async () => {
|
|
const chain = [{ provider: 'anthropic', model: 'claude-sonnet-4-6' }];
|
|
const hopFn = async () => [{ type: 'stop', finish_reason: 'stop' }];
|
|
const { logEvent: log1, events: events1 } = makeLogCapture();
|
|
const { logEvent: log2, events: events2 } = makeLogCapture();
|
|
await executeWithFallback(chain, makeIR({ model: 'test-model' }), hopFn, { logEvent: log1 });
|
|
await executeWithFallback(chain, makeIR({ model: 'test-model' }), hopFn, { logEvent: log2 });
|
|
assert.ok(events1.length >= 1);
|
|
assert.ok(events2.length >= 1);
|
|
// With overwhelming probability two random 8-byte values are different
|
|
assert.notEqual(events1[0].data.chain_id, events2[0].data.chain_id,
|
|
'Separate calls must have different chain_ids');
|
|
});
|
|
|
|
// ── ir_request_hash: provider-agnostic + consistent across hops ───────────
|
|
|
|
it('ir_request_hash is the same across all hops within one chain', async () => {
|
|
const { logEvent, events } = makeLogCapture();
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
await executeWithFallback(chain, makeIR({ model: 'some-model' }), hopFn, { logEvent });
|
|
const hashes = new Set(events.map(e => e.data.ir_request_hash));
|
|
assert.equal(hashes.size, 1, `All hops must share the same ir_request_hash, got: ${[...hashes]}`);
|
|
});
|
|
|
|
it('ir_request_hash does not change when provider or model changes but content is same', async () => {
|
|
const ir = makeIR({ model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: 'Hello' }] });
|
|
const hash1 = computeIRRequestHash(ir);
|
|
// Same IR, different model field (computeIRRequestHash is provider-agnostic)
|
|
const ir2 = makeIR({ model: 'gpt-5.5', messages: [{ role: 'user', content: 'Hello' }] });
|
|
const hash2 = computeIRRequestHash(ir2);
|
|
// model is NOT in the computeIRRequestHash subset, so hashes must be equal
|
|
assert.equal(hash1, hash2, 'ir_request_hash must be provider/model-agnostic');
|
|
});
|
|
|
|
// ── trigger_type: classification per error type ───────────────────────────
|
|
|
|
it('trigger_type is "hard" for SPAWN_FAILED (fallback_hard_trigger event)', async () => {
|
|
const { logEvent, events } = makeLogCapture();
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
await executeWithFallback(chain, makeIR({ model: 'test-model' }), hopFn, { logEvent });
|
|
const hardEvent = events.find(e => e.event === 'fallback_hard_trigger');
|
|
assert.ok(hardEvent, 'Expected fallback_hard_trigger event');
|
|
assert.equal(hardEvent.data.trigger_type, 'hard');
|
|
});
|
|
|
|
it('trigger_type is "auth_missing" for AUTH_MISSING (fallback_auth_missing_no_fallback event)', async () => {
|
|
const { logEvent, events } = makeLogCapture();
|
|
const err = new ProviderError('Auth missing', 'AUTH_MISSING');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
await executeWithFallback(chain, makeIR({ model: 'test-model' }), hopFn, { logEvent });
|
|
const authEvent = events.find(e => e.event === 'fallback_auth_missing_no_fallback');
|
|
assert.ok(authEvent, 'Expected fallback_auth_missing_no_fallback event');
|
|
assert.equal(authEvent.data.trigger_type, 'auth_missing');
|
|
});
|
|
|
|
it('trigger_type is "client_error" for HTTP 400 (fallback_client_error_no_fallback event)', async () => {
|
|
const { logEvent, events } = makeLogCapture();
|
|
const err = Object.assign(new Error('Bad request'), { statusCode: 400 });
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
await executeWithFallback(chain, makeIR({ model: 'test-model' }), hopFn, { logEvent });
|
|
const clientEvent = events.find(e => e.event === 'fallback_client_error_no_fallback');
|
|
assert.ok(clientEvent, 'Expected fallback_client_error_no_fallback event');
|
|
assert.equal(clientEvent.data.trigger_type, 'client_error');
|
|
});
|
|
|
|
it('trigger_type is "non_trigger" for generic error (fallback_non_trigger_error event)', async () => {
|
|
const { logEvent, events } = makeLogCapture();
|
|
const err = new Error('Unexpected generic error');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
await executeWithFallback(chain, makeIR({ model: 'test-model' }), hopFn, { logEvent });
|
|
const nonTriggerEvent = events.find(e => e.event === 'fallback_non_trigger_error');
|
|
assert.ok(nonTriggerEvent, 'Expected fallback_non_trigger_error event');
|
|
assert.equal(nonTriggerEvent.data.trigger_type, 'non_trigger');
|
|
});
|
|
|
|
it('trigger_type is null on success (fallback_hop_success event)', async () => {
|
|
const { logEvent, events } = makeLogCapture();
|
|
const chain = [{ provider: 'anthropic', model: 'claude-sonnet-4-6' }];
|
|
const hopFn = async () => [{ type: 'stop', finish_reason: 'stop' }];
|
|
await executeWithFallback(chain, makeIR({ model: 'test-model' }), hopFn, { logEvent });
|
|
const successEvent = events.find(e => e.event === 'fallback_hop_success');
|
|
assert.ok(successEvent, 'Expected fallback_hop_success event');
|
|
assert.equal(successEvent.data.trigger_type, null);
|
|
});
|
|
|
|
// ── next_provider: lookahead routing ──────────────────────────────────────
|
|
|
|
it('next_provider is chain[i+1].provider when chain advances on hard trigger', async () => {
|
|
const { logEvent, events } = makeLogCapture();
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
await executeWithFallback(chain, makeIR({ model: 'test-model' }), hopFn, { logEvent });
|
|
const hardEvent = events.find(e => e.event === 'fallback_hard_trigger');
|
|
assert.ok(hardEvent, 'Expected fallback_hard_trigger event');
|
|
assert.equal(hardEvent.data.next_provider, 'openai', 'next_provider must be the next chain hop');
|
|
});
|
|
|
|
it('next_provider is null on the last hop (chain exhausted)', async () => {
|
|
const { logEvent, events } = makeLogCapture();
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
];
|
|
const hopFn = async () => { throw err; };
|
|
await executeWithFallback(chain, makeIR({ model: 'test-model' }), hopFn, { logEvent });
|
|
const exhaustedEvent = events.find(e => e.event === 'fallback_chain_exhausted');
|
|
assert.ok(exhaustedEvent, 'Expected fallback_chain_exhausted event');
|
|
assert.equal(exhaustedEvent.data.next_provider, null, 'next_provider must be null when chain is exhausted');
|
|
});
|
|
|
|
it('next_provider is null on client error stop (no advancement)', async () => {
|
|
const { logEvent, events } = makeLogCapture();
|
|
const err = Object.assign(new Error('Bad request'), { statusCode: 422 });
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
await executeWithFallback(chain, makeIR({ model: 'test-model' }), hopFn, { logEvent });
|
|
const clientEvent = events.find(e => e.event === 'fallback_client_error_no_fallback');
|
|
assert.ok(clientEvent, 'Expected fallback_client_error_no_fallback event');
|
|
assert.equal(clientEvent.data.next_provider, null, 'next_provider must be null when not advancing');
|
|
});
|
|
|
|
it('next_provider is null on success (no advancement needed)', async () => {
|
|
const { logEvent, events } = makeLogCapture();
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const hopFn = async () => [{ type: 'stop', finish_reason: 'stop' }];
|
|
await executeWithFallback(chain, makeIR({ model: 'test-model' }), hopFn, { logEvent });
|
|
const successEvent = events.find(e => e.event === 'fallback_hop_success');
|
|
assert.ok(successEvent, 'Expected fallback_hop_success event');
|
|
assert.equal(successEvent.data.next_provider, null, 'next_provider must be null on success');
|
|
});
|
|
|
|
// ── computeIRRequestHash: stability + per-field sensitivity ───────────────
|
|
|
|
it('computeIRRequestHash: identical IR inputs → identical hash', () => {
|
|
const ir1 = makeIR({ messages: [{ role: 'user', content: 'Hello' }], model: 'model-a' });
|
|
const ir2 = makeIR({ messages: [{ role: 'user', content: 'Hello' }], model: 'model-a' });
|
|
assert.equal(computeIRRequestHash(ir1), computeIRRequestHash(ir2));
|
|
});
|
|
|
|
it('computeIRRequestHash: returns 16-char hex string', () => {
|
|
const ir = makeIR({ messages: [{ role: 'user', content: 'Hello' }] });
|
|
const hash = computeIRRequestHash(ir);
|
|
assert.equal(typeof hash, 'string');
|
|
assert.equal(hash.length, 16);
|
|
assert.match(hash, /^[0-9a-f]{16}$/);
|
|
});
|
|
|
|
it('computeIRRequestHash: changes when messages content changes', () => {
|
|
const ir1 = makeIR({ messages: [{ role: 'user', content: 'Hello' }] });
|
|
const ir2 = makeIR({ messages: [{ role: 'user', content: 'Goodbye' }] });
|
|
assert.notEqual(computeIRRequestHash(ir1), computeIRRequestHash(ir2));
|
|
});
|
|
|
|
it('computeIRRequestHash: changes when temperature changes', () => {
|
|
const ir1 = makeIR({ temperature: 0.5 });
|
|
const ir2 = makeIR({ temperature: 1.0 });
|
|
assert.notEqual(computeIRRequestHash(ir1), computeIRRequestHash(ir2));
|
|
});
|
|
|
|
it('computeIRRequestHash: changes when max_tokens changes', () => {
|
|
const ir1 = makeIR({ max_tokens: 100 });
|
|
const ir2 = makeIR({ max_tokens: 200 });
|
|
assert.notEqual(computeIRRequestHash(ir1), computeIRRequestHash(ir2));
|
|
});
|
|
|
|
it('computeIRRequestHash: changes when top_p changes', () => {
|
|
const ir1 = makeIR({ top_p: 0.9 });
|
|
const ir2 = makeIR({ top_p: 0.5 });
|
|
assert.notEqual(computeIRRequestHash(ir1), computeIRRequestHash(ir2));
|
|
});
|
|
|
|
it('computeIRRequestHash: changes when stop changes', () => {
|
|
const ir1 = makeIR({ stop: ['END'] });
|
|
const ir2 = makeIR({ stop: ['STOP'] });
|
|
assert.notEqual(computeIRRequestHash(ir1), computeIRRequestHash(ir2));
|
|
});
|
|
|
|
it('computeIRRequestHash: changes when tool_choice changes', () => {
|
|
const ir1 = makeIR({ tool_choice: 'auto' });
|
|
const ir2 = makeIR({ tool_choice: 'none' });
|
|
assert.notEqual(computeIRRequestHash(ir1), computeIRRequestHash(ir2));
|
|
});
|
|
|
|
it('computeIRRequestHash: changes when response_format changes', () => {
|
|
const ir1 = makeIR({ response_format: { type: 'text' } });
|
|
const ir2 = makeIR({ response_format: { type: 'json_object' } });
|
|
assert.notEqual(computeIRRequestHash(ir1), computeIRRequestHash(ir2));
|
|
});
|
|
|
|
it('computeIRRequestHash: changes when tools changes', () => {
|
|
const ir1 = makeIR({ tools: [{ type: 'function', function: { name: 'foo' } }] });
|
|
const ir2 = makeIR({ tools: [{ type: 'function', function: { name: 'bar' } }] });
|
|
assert.notEqual(computeIRRequestHash(ir1), computeIRRequestHash(ir2));
|
|
});
|
|
|
|
});
|
|
|
|
// ── D40: X-OLP-Fallback-Detail header (issue #7) ─────────────────────────
|
|
//
|
|
// Tests for D40 fold the per-hop fallback failure detail into a response
|
|
// header. Authority: ADR 0004 § Decision § Chain advancement step 4 +
|
|
// § Observability headers.
|
|
|
|
import {
|
|
serializeFallbackDetailHeader,
|
|
FALLBACK_DETAIL_BYTE_CAP,
|
|
} from './server.mjs';
|
|
|
|
describe('D40 — X-OLP-Fallback-Detail header (issue #7)', () => {
|
|
|
|
// ── Engine: fallbackDetail returned in FallbackResult ─────────────────
|
|
|
|
it('engine: 2-hop chain, both fail → fallbackDetail has 2 tuples with correct shape', async () => {
|
|
const errA = new ProviderError('Anthropic spawn failed', 'SPAWN_FAILED');
|
|
const errB = new ProviderError('Codex spawn timeout', 'SPAWN_TIMEOUT');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw errA;
|
|
throw errB;
|
|
};
|
|
const result = await executeWithFallback(chain, makeIR({ model: 'claude-sonnet-4-6' }), hopFn);
|
|
assert.equal(result.chunks, null, 'Expected exhausted chain');
|
|
assert.ok(Array.isArray(result.fallbackDetail), 'fallbackDetail must be array');
|
|
assert.equal(result.fallbackDetail.length, 2);
|
|
assert.deepEqual(result.fallbackDetail[0], {
|
|
hop: 0, provider: 'anthropic', model: 'claude-sonnet-4-6',
|
|
code: 'SPAWN_FAILED', error_message: 'Anthropic spawn failed', trigger_type: 'hard',
|
|
});
|
|
assert.deepEqual(result.fallbackDetail[1], {
|
|
hop: 1, provider: 'openai', model: 'gpt-5.5',
|
|
code: 'SPAWN_TIMEOUT', error_message: 'Codex spawn timeout', trigger_type: 'hard',
|
|
});
|
|
});
|
|
|
|
it('engine: 2-hop chain, primary fails + secondary succeeds → fallbackDetail has 1 tuple (the failed primary)', async () => {
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
const result = await executeWithFallback(chain, makeIR({ model: 'claude-sonnet-4-6' }), hopFn);
|
|
assert.ok(Array.isArray(result.chunks), 'Expected success chunks');
|
|
assert.equal(result.fallbackHops, 1);
|
|
assert.equal(result.fallbackDetail.length, 1);
|
|
assert.equal(result.fallbackDetail[0].hop, 0);
|
|
assert.equal(result.fallbackDetail[0].provider, 'anthropic');
|
|
assert.equal(result.fallbackDetail[0].code, 'SPAWN_FAILED');
|
|
assert.equal(result.fallbackDetail[0].trigger_type, 'hard');
|
|
});
|
|
|
|
it('engine: 1-hop chain succeeds → fallbackDetail is empty array', async () => {
|
|
const chain = [{ provider: 'anthropic', model: 'claude-sonnet-4-6' }];
|
|
const hopFn = async () => [{ type: 'stop', finish_reason: 'stop' }];
|
|
const result = await executeWithFallback(chain, makeIR({ model: 'claude-sonnet-4-6' }), hopFn);
|
|
assert.ok(Array.isArray(result.chunks));
|
|
assert.deepEqual(result.fallbackDetail, [], 'Empty fallbackDetail on clean primary success');
|
|
});
|
|
|
|
it('engine: 1-hop chain fails → fallbackDetail has 1 tuple', async () => {
|
|
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
|
|
const chain = [{ provider: 'anthropic', model: 'claude-sonnet-4-6' }];
|
|
const hopFn = async () => { throw err; };
|
|
const result = await executeWithFallback(chain, makeIR({ model: 'claude-sonnet-4-6' }), hopFn);
|
|
assert.equal(result.chunks, null);
|
|
assert.equal(result.fallbackDetail.length, 1);
|
|
assert.equal(result.fallbackDetail[0].hop, 0);
|
|
assert.equal(result.fallbackDetail[0].code, 'SPAWN_FAILED');
|
|
});
|
|
|
|
it('engine: AUTH_MISSING terminates chain, fallbackDetail tuple records trigger_type:"auth_missing" (D56, v1.x roadmap #7)', async () => {
|
|
// v1.x roadmap #7 (D40 follow-up): explicit pin that the AUTH_MISSING
|
|
// path produces a fallbackDetail tuple with trigger_type:"auth_missing"
|
|
// BEFORE the engine's early-return at engine.mjs:486. Was implicit via
|
|
// other engine-path tests; D56 makes it explicit so any future refactor
|
|
// that moves the tuple-push past the auth_missing branch fails this.
|
|
const err = new ProviderError('No OAuth token found', 'AUTH_MISSING');
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' }, // present to verify AUTH_MISSING does NOT advance
|
|
];
|
|
const hopFn = async () => { throw err; };
|
|
const result = await executeWithFallback(chain, makeIR({ model: 'claude-sonnet-4-6' }), hopFn);
|
|
// AUTH_MISSING is HARD_TRIGGER_CODES[AUTH_MISSING]=false (engine.mjs L52);
|
|
// chain stops at hop 0 instead of advancing to openai.
|
|
assert.equal(result.chunks, null, 'AUTH_MISSING terminates chain');
|
|
assert.equal(result.fallbackHops, 0, 'AUTH_MISSING does NOT advance — stays at hop 0');
|
|
assert.equal(result.fallbackDetail.length, 1, 'fallbackDetail has exactly 1 tuple (the AUTH_MISSING hop)');
|
|
assert.equal(result.fallbackDetail[0].code, 'AUTH_MISSING');
|
|
assert.equal(result.fallbackDetail[0].trigger_type, 'auth_missing');
|
|
assert.equal(result.fallbackDetail[0].provider, 'anthropic');
|
|
assert.equal(result.fallbackDetail[0].hop, 0);
|
|
});
|
|
|
|
it('engine: non-ProviderError exception → tuple code is "UNKNOWN"', async () => {
|
|
const err = new Error('Something unexpected'); // no .code field
|
|
const chain = [{ provider: 'anthropic', model: 'claude-sonnet-4-6' }];
|
|
const hopFn = async () => { throw err; };
|
|
const result = await executeWithFallback(chain, makeIR({ model: 'claude-sonnet-4-6' }), hopFn);
|
|
assert.equal(result.fallbackDetail.length, 1);
|
|
assert.equal(result.fallbackDetail[0].code, 'UNKNOWN');
|
|
assert.equal(result.fallbackDetail[0].error_message, 'Something unexpected');
|
|
assert.equal(result.fallbackDetail[0].trigger_type, 'non_trigger');
|
|
});
|
|
|
|
it('engine: 500-char error message → tuple.error_message is truncated to 200 chars ending in ellipsis', async () => {
|
|
const longMsg = 'x'.repeat(500);
|
|
const err = new ProviderError(longMsg, 'SPAWN_FAILED');
|
|
const chain = [{ provider: 'anthropic', model: 'claude-sonnet-4-6' }];
|
|
const hopFn = async () => { throw err; };
|
|
const result = await executeWithFallback(chain, makeIR({ model: 'claude-sonnet-4-6' }), hopFn);
|
|
const tuple = result.fallbackDetail[0];
|
|
assert.equal(tuple.error_message.length, 200, 'Truncated message must be exactly 200 chars');
|
|
assert.equal(tuple.error_message.slice(-1), '…', 'Truncated message must end with ellipsis');
|
|
assert.equal(tuple.error_message.slice(0, 199), 'x'.repeat(199), 'First 199 chars preserved');
|
|
});
|
|
|
|
it('engine: client error 400 → fallbackDetail has 1 tuple with trigger_type "client_error" (no advance)', async () => {
|
|
const err = Object.assign(new Error('Bad request'), { statusCode: 400 });
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const hopFn = async (provider) => {
|
|
if (provider === 'anthropic') throw err;
|
|
return [{ type: 'stop', finish_reason: 'stop' }];
|
|
};
|
|
const result = await executeWithFallback(chain, makeIR({ model: 'claude-sonnet-4-6' }), hopFn);
|
|
assert.equal(result.fallbackDetail.length, 1, 'Client error stops chain — only 1 tuple');
|
|
assert.equal(result.fallbackDetail[0].trigger_type, 'client_error');
|
|
});
|
|
|
|
// ── serializeFallbackDetailHeader: cap + RFC 7230 hygiene ─────────────
|
|
|
|
it('serialize: empty array → null (no header emitted)', () => {
|
|
assert.equal(serializeFallbackDetailHeader([]), null);
|
|
});
|
|
|
|
it('serialize: null/undefined → null (no header emitted)', () => {
|
|
assert.equal(serializeFallbackDetailHeader(null), null);
|
|
assert.equal(serializeFallbackDetailHeader(undefined), null);
|
|
});
|
|
|
|
it('serialize: small array → JSON.stringified array under cap', () => {
|
|
const detail = [{
|
|
hop: 0, provider: 'anthropic', model: 'claude-sonnet-4-6',
|
|
code: 'SPAWN_FAILED', error_message: 'oops', trigger_type: 'hard',
|
|
}];
|
|
const v = serializeFallbackDetailHeader(detail);
|
|
assert.ok(typeof v === 'string');
|
|
assert.ok(Buffer.byteLength(v, 'utf8') <= FALLBACK_DETAIL_BYTE_CAP);
|
|
const parsed = JSON.parse(v);
|
|
assert.deepEqual(parsed, detail);
|
|
});
|
|
|
|
it('serialize: array exceeding 4KB → truncated with { truncated:true, omitted_hops:N } sentinel; total stays <= 4096 bytes', () => {
|
|
// Build many tuples each just under the cap individually but cumulatively
|
|
// over the cap. Each tuple here is roughly 250 bytes serialised.
|
|
const baseMsg = 'x'.repeat(180); // leaves ~20 chars for JSON braces/key padding
|
|
const tuples = [];
|
|
for (let i = 0; i < 50; i++) {
|
|
tuples.push({
|
|
hop: i,
|
|
provider: 'anthropic',
|
|
model: 'claude-sonnet-4-6',
|
|
code: 'SPAWN_FAILED',
|
|
error_message: baseMsg,
|
|
trigger_type: 'hard',
|
|
});
|
|
}
|
|
const v = serializeFallbackDetailHeader(tuples);
|
|
assert.ok(typeof v === 'string');
|
|
assert.ok(
|
|
Buffer.byteLength(v, 'utf8') <= FALLBACK_DETAIL_BYTE_CAP,
|
|
`Header value must be <= ${FALLBACK_DETAIL_BYTE_CAP} bytes, got ${Buffer.byteLength(v, 'utf8')}`,
|
|
);
|
|
const parsed = JSON.parse(v);
|
|
assert.ok(Array.isArray(parsed), 'Must JSON-parse to array');
|
|
const tail = parsed[parsed.length - 1];
|
|
assert.equal(tail.truncated, true, 'Last entry must be the truncation sentinel');
|
|
assert.equal(typeof tail.omitted_hops, 'number');
|
|
assert.ok(tail.omitted_hops > 0, 'omitted_hops must be positive when truncation fired');
|
|
assert.equal(
|
|
parsed.length - 1 + tail.omitted_hops,
|
|
tuples.length,
|
|
'kept tuples + omitted_hops must equal total original tuples',
|
|
);
|
|
});
|
|
|
|
it('serialize: JSON.stringify escapes newlines in error_message (RFC 7230 hygiene — no raw CR/LF in header)', () => {
|
|
const detail = [{
|
|
hop: 0, provider: 'anthropic', model: 'claude-sonnet-4-6',
|
|
code: 'UNKNOWN', error_message: 'line1\nline2\rline3', trigger_type: 'non_trigger',
|
|
}];
|
|
const v = serializeFallbackDetailHeader(detail);
|
|
assert.ok(typeof v === 'string');
|
|
// Raw \n / \r in a header value would break RFC 7230. JSON.stringify escapes them.
|
|
assert.equal(v.indexOf('\n'), -1, 'Header value must not contain raw newline');
|
|
assert.equal(v.indexOf('\r'), -1, 'Header value must not contain raw CR');
|
|
// Parsed back, the original message round-trips correctly
|
|
assert.equal(JSON.parse(v)[0].error_message, 'line1\nline2\rline3');
|
|
});
|
|
|
|
it('serialize: non-ASCII characters in error_message (e.g. em dash U+2014) → escaped as \\uXXXX so Node HTTP header validator accepts the value', () => {
|
|
// The D38 CONCURRENCY_LIMIT synthesised error message in server.mjs contains
|
|
// an em dash. Without \uXXXX escaping, res.writeHead would throw
|
|
// "Invalid character in header content" because Node's HTTP layer rejects
|
|
// multi-byte UTF-8 in header values (RFC 7230 §3.2.6 field-vchar = ASCII VCHAR).
|
|
const detail = [{
|
|
hop: 0, provider: 'anthropic', model: 'claude-sonnet-4-6',
|
|
code: 'CONCURRENCY_LIMIT',
|
|
error_message: 'provider anthropic at maxConcurrent (2) — advancing to next hop',
|
|
trigger_type: 'hard',
|
|
}];
|
|
const v = serializeFallbackDetailHeader(detail);
|
|
assert.ok(typeof v === 'string');
|
|
// Every byte must be ASCII (0x00-0x7F) — verified via Buffer round-trip equality.
|
|
const utf8Len = Buffer.byteLength(v, 'utf8');
|
|
assert.equal(utf8Len, v.length, 'Header value must be pure ASCII (1 byte per char)');
|
|
for (let i = 0; i < v.length; i++) {
|
|
assert.ok(v.charCodeAt(i) < 0x80, `Char at ${i} (${v[i]}) must be ASCII`);
|
|
}
|
|
// JSON.parse must still round-trip the original em dash.
|
|
assert.equal(
|
|
JSON.parse(v)[0].error_message,
|
|
'provider anthropic at maxConcurrent (2) — advancing to next hop',
|
|
);
|
|
});
|
|
|
|
});
|
|
|
|
// ── Suite 14: providers.enabled config wiring ──────────────────────────────
|
|
//
|
|
// Tests that:
|
|
// 14a: Empty config → 0 providers loaded
|
|
// 14b: providers.enabled.anthropic=true → anthropic in loaded map
|
|
// 14c: HTTP 503 disappears once config has the right provider enabled
|
|
//
|
|
// Authority: ADR 0002 § Disable model; ALIGNMENT.md § Provider Inventory
|
|
|
|
import {
|
|
__setProvidersEnabled,
|
|
__resetProvidersEnabled,
|
|
} from './server.mjs';
|
|
|
|
describe('providers.enabled config wiring (Suite 14)', () => {
|
|
|
|
it('14a: loadFallbackConfigSync returns providersEnabled field', () => {
|
|
// loadFallbackConfigSync with no config file → returns empty providersEnabled
|
|
// (This tests the schema change to loadFallbackConfigSync.)
|
|
const cfg = loadFallbackConfigSync('/nonexistent/path/that/does/not/exist.json');
|
|
assert.ok(typeof cfg === 'object', 'result must be an object');
|
|
assert.ok('providersEnabled' in cfg, 'result must have providersEnabled field');
|
|
assert.deepEqual(cfg.providersEnabled, {}, 'missing file → empty providersEnabled');
|
|
assert.deepEqual(cfg.chains, {}, 'missing file → empty chains');
|
|
});
|
|
|
|
it('14b: __setProvidersEnabled({ anthropic: true }) → anthropic in loadedProviders', async () => {
|
|
const { loadedProviders: lp } = await import('./server.mjs');
|
|
const originalSize = lp.size;
|
|
try {
|
|
__setProvidersEnabled({ anthropic: true });
|
|
assert.ok(lp.has('anthropic'), 'anthropic must be in loadedProviders after enable');
|
|
} finally {
|
|
__resetProvidersEnabled();
|
|
// Restore to whatever state it was (may vary depending on config file)
|
|
}
|
|
});
|
|
|
|
it('14c: __setProvidersEnabled({}) → no providers loaded → HTTP 503', async () => {
|
|
const { createOlpServer: createServer14 } = await import('./server.mjs');
|
|
__setProvidersEnabled({});
|
|
const s = createServer14();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({
|
|
port: p,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: 'hi' }] },
|
|
});
|
|
assert.equal(r.status, 503, `Expected 503 with no providers, got ${r.status}`);
|
|
} finally {
|
|
__resetProvidersEnabled();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
it('14d: __setProvidersEnabled({ anthropic: true }) → HTTP 200 with mock spawn', async () => {
|
|
const savedToken = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-suite-14d';
|
|
__setProvidersEnabled({ anthropic: true });
|
|
|
|
// Install mock spawn that immediately emits text + close
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('data', Buffer.from('Hello from suite 14d'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
const { createOlpServer: createServer14d, __clearCache: clearCache14d } = await import('./server.mjs');
|
|
clearCache14d();
|
|
const s = createServer14d();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port: p,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: 'hi' }] },
|
|
});
|
|
assert.equal(r.status, 200, `Expected 200 with anthropic enabled, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
} finally {
|
|
__resetProvidersEnabled();
|
|
__resetSpawnImpl();
|
|
if (savedToken !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
});
|
|
|
|
// ── Suite 15: Streaming cache-miss real-time (P1.2) ─────────────────────────
|
|
//
|
|
// Tests that the single-hop streaming path (P1.2) emits chunks in real-time
|
|
// rather than buffering. Also tests the cache-hit burst-replay path for
|
|
// the second identical request.
|
|
//
|
|
// Authority: ADR 0003 entry adapter pattern; ADR 0004 § first-chunk rule
|
|
|
|
describe('Streaming cache-miss real-time (Suite 15)', () => {
|
|
let server15;
|
|
let port15;
|
|
let savedToken15;
|
|
|
|
before(async () => {
|
|
savedToken15 = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-suite-15';
|
|
__setProvidersEnabled({ anthropic: true });
|
|
|
|
const { createOlpServer: s15, __clearCache: cc15 } = await import('./server.mjs');
|
|
cc15();
|
|
server15 = s15();
|
|
await new Promise((resolve, reject) => {
|
|
server15.listen(0, '127.0.0.1', resolve);
|
|
server15.once('error', reject);
|
|
});
|
|
port15 = server15.address().port;
|
|
});
|
|
|
|
after(async () => {
|
|
__resetProvidersEnabled();
|
|
__resetSpawnImpl();
|
|
if (savedToken15 !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken15;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
if (!server15) return;
|
|
return new Promise(r => server15.close(r));
|
|
});
|
|
|
|
it('15a: streaming cache-miss → res.write fires per chunk, not all-at-once', async () => {
|
|
// Mock: emits 3 deltas with a 20ms gap between each, then stop.
|
|
// We verify by recording arrival timestamps on the client side.
|
|
const DELTA_GAP_MS = 20;
|
|
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
// Emit 3 deltas with gaps
|
|
let idx = 0;
|
|
const texts = ['chunk1', 'chunk2', 'chunk3'];
|
|
const emitNext = () => {
|
|
if (idx < texts.length) {
|
|
proc.stdout.emit('data', Buffer.from(texts[idx]));
|
|
idx++;
|
|
setTimeout(emitNext, DELTA_GAP_MS);
|
|
} else {
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
}
|
|
};
|
|
setImmediate(emitNext);
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
const { __clearCache: cc15a } = await import('./server.mjs');
|
|
cc15a();
|
|
|
|
// Make SSE request and collect timestamps of each data: line arrival
|
|
const arrivalTimestamps = [];
|
|
const chunks = await new Promise((resolve, reject) => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port: port15,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
}, res => {
|
|
const collectedLines = [];
|
|
res.on('data', (d) => {
|
|
const text = d.toString();
|
|
const lines = text.split('\n').filter(l => l.startsWith('data: ') && l !== 'data: [DONE]');
|
|
if (lines.length > 0) {
|
|
arrivalTimestamps.push(Date.now());
|
|
collectedLines.push(...lines);
|
|
}
|
|
});
|
|
res.on('end', () => resolve(collectedLines));
|
|
res.on('error', reject);
|
|
});
|
|
req.on('error', reject);
|
|
req.write(JSON.stringify({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'streaming test' }],
|
|
stream: true,
|
|
}));
|
|
req.end();
|
|
});
|
|
|
|
// We should have received some data events
|
|
assert.ok(chunks.length > 0, 'should have received streaming chunks');
|
|
// With 20ms inter-chunk gaps the OS should flush at least twice; a buffered
|
|
// implementation would surface as a single arrival event. This assertion
|
|
// proves real-streaming architecturally (per D10 P1.2 review).
|
|
assert.ok(
|
|
arrivalTimestamps.length >= 2,
|
|
`real streaming should produce > 1 client arrival event, got ${arrivalTimestamps.length}`
|
|
);
|
|
const allText = chunks.join('');
|
|
assert.ok(allText.length > 0, 'should have received non-empty data');
|
|
|
|
// Verify response has SSE headers
|
|
// (We already verified by receiving 'data: ' lines)
|
|
assert.ok(chunks.some(c => c.includes('"delta"') || c.includes('"content"') || c.includes('finish_reason')),
|
|
'Should contain delta or stop chunks');
|
|
});
|
|
|
|
it('15b: second identical streaming request → cache hit, content identical', async () => {
|
|
// Re-use the same mock (last setSpawnImpl from 15a may have been reset).
|
|
// Install a mock that records how many times it was called.
|
|
let spawnCallCount = 0;
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
spawnCallCount++;
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('data', Buffer.from('cached-content'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
const { __clearCache: cc15b } = await import('./server.mjs');
|
|
cc15b();
|
|
|
|
const makeStreamRequest = () => new Promise((resolve, reject) => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port: port15,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
}, res => {
|
|
let data = '';
|
|
res.on('data', d => { data += d.toString(); });
|
|
res.on('end', () => resolve({ status: res.statusCode, body: data, headers: res.headers }));
|
|
res.on('error', reject);
|
|
});
|
|
req.on('error', reject);
|
|
req.write(JSON.stringify({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'cache-test-15b' }],
|
|
stream: true,
|
|
}));
|
|
req.end();
|
|
});
|
|
|
|
// First request: cache miss → real spawn
|
|
const first = await makeStreamRequest();
|
|
assert.equal(first.status, 200, `Expected 200, got ${first.status}: ${first.body.slice(0, 200)}`);
|
|
assert.equal(first.headers['x-olp-cache'], 'miss', 'First request should be cache miss');
|
|
assert.equal(spawnCallCount, 1, 'Spawn should be called exactly once for first request');
|
|
|
|
// Second identical request: should be cache hit (burst-replay, no spawn)
|
|
const second = await makeStreamRequest();
|
|
assert.equal(second.status, 200, `Expected 200 on cache hit, got ${second.status}`);
|
|
assert.equal(second.headers['x-olp-cache'], 'hit', 'Second request should be cache hit');
|
|
assert.equal(spawnCallCount, 1, 'Spawn should NOT be called again for cache-hit request');
|
|
|
|
// Both responses should contain the same content
|
|
assert.ok(first.body.includes('cached-content'), 'First response should contain content');
|
|
assert.ok(second.body.includes('cached-content'), 'Second response should contain same content');
|
|
});
|
|
|
|
it('15c: streaming + multi-hop chain → uses buffered path (not real-streaming)', async () => {
|
|
// A 2-hop chain forces the code to fall through to executeWithFallback (buffered).
|
|
// With mock anthropic succeeding, the result should still be 200 with streaming headers,
|
|
// but X-OLP-Cache should not reflect real-streaming behavior (it may be miss or hit
|
|
// depending on buffered cache).
|
|
// The critical assertion: single-hop condition NOT met → buffered path is taken.
|
|
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('data', Buffer.from('multihop-content'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
// Add openai to loaded providers and configure a 2-hop chain
|
|
const { loadedProviders: lp15c, __clearCache: cc15c } = await import('./server.mjs');
|
|
const extraProviders = loadProviders({ enabled: { anthropic: true, openai: true } });
|
|
for (const [name, p] of extraProviders) lp15c.set(name, p);
|
|
cc15c();
|
|
|
|
// Inject codex fake auth so the 2nd hop doesn't immediately AUTH_MISSING
|
|
const savedCodexPath = process.env.OPENAI_CODEX_AUTH_PATH;
|
|
const { writeFileSync: wfs15c, unlinkSync: uls15c } = await import('node:fs');
|
|
const { tmpdir: td15c } = await import('node:os');
|
|
const { join: pj15c } = await import('node:path');
|
|
const tmpAuth15c = pj15c(td15c(), `olp-test-15c-${Date.now()}.json`);
|
|
wfs15c(tmpAuth15c, JSON.stringify({ accessToken: 'fake-codex-15c' }), 'utf8');
|
|
process.env.OPENAI_CODEX_AUTH_PATH = tmpAuth15c;
|
|
|
|
__setFallbackConfig({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
});
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port: port15,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'multihop streaming test' }],
|
|
stream: true,
|
|
},
|
|
});
|
|
// Multi-hop chain → buffered path → still returns 200 with streaming headers
|
|
assert.equal(r.status, 200, `Expected 200 for multi-hop streaming, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
// X-OLP-Fallback-Hops: 0 since first hop succeeds
|
|
assert.equal(r.headers['x-olp-fallback-hops'], '0', 'Primary hop should serve');
|
|
assert.ok(r.body.includes('multihop-content'), 'Response should contain content');
|
|
} finally {
|
|
__resetFallbackConfig();
|
|
__resetSpawnImpl();
|
|
if (savedCodexPath !== undefined) {
|
|
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexPath;
|
|
} else {
|
|
delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
}
|
|
try { uls15c(tmpAuth15c); } catch { /* ignore */ }
|
|
// Clean up the extra openai provider
|
|
lp15c.delete('openai');
|
|
}
|
|
});
|
|
|
|
it('15d: streaming early error → 502 JSON not 200 empty body (D14 defect fix)', async () => {
|
|
// Authority: D14 defect fix — deferred writeHead in real-streaming path.
|
|
// Pre-D14: writeHead(200) fired unconditionally before spawn, so a throw
|
|
// before first chunk produced HTTP 200 + empty SSE body (silent loss).
|
|
// Post-D14: writeHead is deferred to just before the first res.write;
|
|
// a throw before any chunk fires sendError(502, ...) identical to the
|
|
// buffered path. This test would FAIL on pre-D14 code.
|
|
|
|
// Inject a mock anthropic provider whose spawn throws immediately (no yields).
|
|
const { loadedProviders: lp15d, __clearCache: cc15d } = await import('./server.mjs');
|
|
const savedAnthropicProvider = lp15d.get('anthropic');
|
|
|
|
const earlyErrorMessage = 'Simulated provider failure before first chunk';
|
|
const throwingProvider = {
|
|
...savedAnthropicProvider,
|
|
spawn: async function* () {
|
|
throw new ProviderError(earlyErrorMessage, 'SPAWN_FAILED');
|
|
},
|
|
};
|
|
lp15d.set('anthropic', throwingProvider);
|
|
cc15d();
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port: port15,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'streaming early error test' }],
|
|
stream: true,
|
|
},
|
|
});
|
|
|
|
// Must be 502, NOT 200 (the pre-D14 silent-loss behaviour)
|
|
assert.equal(r.status, 502, `Expected 502, got ${r.status}. Body: ${r.body.slice(0, 400)}`);
|
|
|
|
// Must be JSON error body, NOT SSE
|
|
assert.ok(
|
|
(r.headers['content-type'] ?? '').includes('application/json'),
|
|
`Expected application/json, got: ${r.headers['content-type']}`
|
|
);
|
|
|
|
// Body must be parseable as {error: {message, type}}
|
|
let parsed;
|
|
try {
|
|
parsed = JSON.parse(r.body);
|
|
} catch {
|
|
assert.fail(`Response body is not valid JSON: ${r.body.slice(0, 400)}`);
|
|
}
|
|
assert.ok(parsed.error, 'Response must have an error field');
|
|
assert.equal(parsed.error.type, 'provider_error', `Expected type provider_error, got ${parsed.error.type}`);
|
|
assert.ok(
|
|
parsed.error.message.includes(earlyErrorMessage),
|
|
`Expected error message to contain "${earlyErrorMessage}", got: ${parsed.error.message}`
|
|
);
|
|
} finally {
|
|
// Restore the original anthropic provider
|
|
if (savedAnthropicProvider !== undefined) {
|
|
lp15d.set('anthropic', savedAnthropicProvider);
|
|
} else {
|
|
lp15d.delete('anthropic');
|
|
}
|
|
}
|
|
});
|
|
|
|
it('15e: streaming generator exhausted without stop chunk → response delivered but NOT cached (F9)', async () => {
|
|
// Authority: ADR 0005 § "Cache write conditions" item 1 — "response completed
|
|
// successfully (no truncation, no error mid-stream)". A generator that exhausts
|
|
// without emitting a stop chunk is a truncated response and must NOT be cached.
|
|
// Compare D16's buffered-path truncation eviction.
|
|
//
|
|
// Implementation note: __setSpawnImpl cannot produce a no-stop response because
|
|
// the anthropic provider plugin synthesizes a stop chunk on clean proc exit
|
|
// (lib/providers/anthropic.mjs line ~376: `yield anthropicStopToIR('stop')`).
|
|
// Instead we inject a custom provider whose spawn() async generator yields a
|
|
// delta chunk and then RETURNS without yielding a stop chunk, bypassing the
|
|
// plugin layer entirely. Same injection pattern as 15d.
|
|
//
|
|
// Verifies: (a) client receives the chunks (200); (b) the response is NOT
|
|
// written to cache — a second identical request triggers a fresh spawn (cache
|
|
// miss), proving no caching happened after request 1.
|
|
|
|
const { loadedProviders: lp15e, __clearCache: cc15e } = await import('./server.mjs');
|
|
const savedAnthropicProvider = lp15e.get('anthropic');
|
|
|
|
let spawnCallCount = 0;
|
|
const noStopProvider = {
|
|
...savedAnthropicProvider,
|
|
spawn: async function* () {
|
|
spawnCallCount++;
|
|
// Yield one delta chunk but NO stop chunk — generator exhausts here.
|
|
yield { type: 'delta', content: 'no-stop-content', finish_reason: null };
|
|
// (implicit return — no stop chunk emitted)
|
|
},
|
|
};
|
|
lp15e.set('anthropic', noStopProvider);
|
|
cc15e();
|
|
|
|
const makeRequest = () => new Promise((resolve, reject) => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port: port15,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
}, res => {
|
|
let data = '';
|
|
res.on('data', d => { data += d.toString(); });
|
|
res.on('end', () => resolve({ status: res.statusCode, body: data, headers: res.headers }));
|
|
res.on('error', reject);
|
|
});
|
|
req.on('error', reject);
|
|
req.write(JSON.stringify({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'no-stop-chunk-f9-test' }],
|
|
stream: true,
|
|
}));
|
|
req.end();
|
|
});
|
|
|
|
try {
|
|
// First request: generator exhausts without stop → client receives response (200)
|
|
const r1 = await makeRequest();
|
|
assert.equal(r1.status, 200, `First request failed: ${r1.status} ${r1.body.slice(0, 200)}`);
|
|
assert.equal(spawnCallCount, 1, 'Spawn called once for first request');
|
|
assert.equal(r1.headers['x-olp-cache'], 'miss', 'First request must be cache miss');
|
|
|
|
// Second identical request: must ALSO be cache miss + fresh spawn (proves no caching after r1)
|
|
const r2 = await makeRequest();
|
|
assert.equal(r2.status, 200, `Second request failed: ${r2.status} ${r2.body.slice(0, 200)}`);
|
|
assert.equal(spawnCallCount, 2,
|
|
`Expected spawn called again for r2 (no-stop response must not be cached), got spawnCallCount=${spawnCallCount}`);
|
|
assert.equal(r2.headers['x-olp-cache'], 'miss', 'Second request must also be cache miss (no caching of truncated response)');
|
|
} finally {
|
|
// Restore the original anthropic provider
|
|
if (savedAnthropicProvider !== undefined) {
|
|
lp15e.set('anthropic', savedAnthropicProvider);
|
|
} else {
|
|
lp15e.delete('anthropic');
|
|
}
|
|
}
|
|
});
|
|
|
|
});
|
|
|
|
// ── D35 batch: issues #4 / #9 / #10 / #11 / #12 ─────────────────────────────
|
|
//
|
|
// #4 — Uniform X-OLP-* headers on in-handler error paths (audit confirms D32 complete)
|
|
// #9 — Zero-chunk empty stream → writeHead fires with text/event-stream + X-OLP-* headers
|
|
// #10 — Post-first-chunk error emits truncation marker (finish_reason:'length') + [DONE]
|
|
// #11 — validateIRRequest irVersion enforcement
|
|
// #12 — alignment.yml scripts/** path removed (CI only, no test needed)
|
|
//
|
|
// Authority: ADR 0004 § Observability headers; ADR 0003 IR contract
|
|
|
|
// ── D35-#11 unit tests (no server needed) ────────────────────────────────────
|
|
|
|
describe('D35 #11 — validateIRRequest irVersion enforcement', () => {
|
|
|
|
it('#11a: undefined irVersion is accepted (omitted field)', () => {
|
|
const ir = makeIR();
|
|
delete ir.irVersion;
|
|
const r = validateIRRequest(ir);
|
|
assert.equal(r.valid, true, `Expected valid when irVersion is absent, got errors: ${r.errors.join('; ')}`);
|
|
});
|
|
|
|
it("#11b: irVersion '1.0' is accepted (correct version)", () => {
|
|
const r = validateIRRequest(makeIR({ irVersion: '1.0' }));
|
|
assert.equal(r.valid, true, `Expected valid for irVersion '1.0'`);
|
|
});
|
|
|
|
it("#11c: irVersion '2.0' is rejected (wrong version string)", () => {
|
|
const r = validateIRRequest(makeIR({ irVersion: '2.0' }));
|
|
assert.equal(r.valid, false, "Expected invalid for irVersion '2.0'");
|
|
assert.ok(
|
|
r.errors.some(e => e.includes('irVersion')),
|
|
`Expected an irVersion error, got: ${r.errors.join('; ')}`
|
|
);
|
|
});
|
|
|
|
it('#11d: irVersion 1.0 (number, not string) is rejected', () => {
|
|
const r = validateIRRequest(makeIR({ irVersion: 1.0 }));
|
|
assert.equal(r.valid, false, 'Expected invalid for irVersion as number 1.0');
|
|
assert.ok(
|
|
r.errors.some(e => e.includes('irVersion')),
|
|
`Expected an irVersion error, got: ${r.errors.join('; ')}`
|
|
);
|
|
});
|
|
|
|
});
|
|
|
|
// ── D35-#4/#9/#10 integration tests ──────────────────────────────────────────
|
|
|
|
describe('D35 #4/#9/#10 — Streaming error-path header + truncation-marker fixes', () => {
|
|
let serverD35;
|
|
let portD35;
|
|
let savedTokenD35;
|
|
|
|
before(async () => {
|
|
savedTokenD35 = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-d35';
|
|
__setProvidersEnabled({ anthropic: true });
|
|
|
|
const { createOlpServer: sD35, __clearCache: ccD35 } = await import('./server.mjs');
|
|
ccD35();
|
|
serverD35 = sD35();
|
|
await new Promise((resolve, reject) => {
|
|
serverD35.listen(0, '127.0.0.1', resolve);
|
|
serverD35.once('error', reject);
|
|
});
|
|
portD35 = serverD35.address().port;
|
|
});
|
|
|
|
after(async () => {
|
|
__resetProvidersEnabled();
|
|
__resetSpawnImpl();
|
|
if (savedTokenD35 !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedTokenD35;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
if (!serverD35) return;
|
|
return new Promise(r => serverD35.close(r));
|
|
});
|
|
|
|
it('#4-audit: 503 no_enabled_provider carries all 5 X-OLP-* headers (D32 already correct)', async () => {
|
|
// Verifies that the 503 path (no chain built) emits the full 5-header set
|
|
// via olpErrorHeaders — confirming D32 coverage. D35 audit found D32 complete;
|
|
// this test pins the invariant.
|
|
__setProvidersEnabled({});
|
|
const { createOlpServer: s4, __clearCache: cc4 } = await import('./server.mjs');
|
|
cc4();
|
|
const tmpServer = s4();
|
|
await new Promise((resolve, reject) => {
|
|
tmpServer.listen(0, '127.0.0.1', resolve);
|
|
tmpServer.once('error', reject);
|
|
});
|
|
const tmpPort = tmpServer.address().port;
|
|
try {
|
|
const r = await fetch({
|
|
port: tmpPort,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'test' }],
|
|
stream: false,
|
|
},
|
|
});
|
|
assert.equal(r.status, 503, `Expected 503, got ${r.status}`);
|
|
assert.ok(r.headers['x-olp-provider-used'], 'Must have X-OLP-Provider-Used');
|
|
assert.ok(r.headers['x-olp-model-used'], 'Must have X-OLP-Model-Used');
|
|
assert.ok(r.headers['x-olp-fallback-hops'] !== undefined, 'Must have X-OLP-Fallback-Hops');
|
|
assert.ok(r.headers['x-olp-cache'], 'Must have X-OLP-Cache');
|
|
assert.ok(r.headers['x-olp-latency-ms'] !== undefined, 'Must have X-OLP-Latency-Ms');
|
|
assert.equal(r.headers['x-olp-provider-used'], 'none', 'X-OLP-Provider-Used must be "none"');
|
|
assert.equal(r.headers['x-olp-model-used'], 'claude-sonnet-4-6', 'X-OLP-Model-Used must be the requested model');
|
|
} finally {
|
|
__setProvidersEnabled({ anthropic: true });
|
|
await new Promise(r => tmpServer.close(r));
|
|
}
|
|
});
|
|
|
|
it('#9: zero-chunk clean exit → 200 + text/event-stream + all 5 X-OLP-* headers + [DONE]', async () => {
|
|
// Provider spawn yields zero chunks and exits cleanly (empty generator).
|
|
// Before D35 fix: writeHead never fired → Node auto-emitted 200 with default
|
|
// Content-Type + no X-OLP-* headers.
|
|
// After D35 fix: writeHead fires before SSE_DONE with text/event-stream + all 5 headers.
|
|
const { loadedProviders: lpD35_9, __clearCache: ccD35_9 } = await import('./server.mjs');
|
|
const savedProvider = lpD35_9.get('anthropic');
|
|
const emptyProvider = {
|
|
...savedProvider,
|
|
spawn: async function* () {
|
|
// yields nothing — clean generator exit
|
|
},
|
|
};
|
|
lpD35_9.set('anthropic', emptyProvider);
|
|
ccD35_9();
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port: portD35,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'empty-stream-test' }],
|
|
stream: true,
|
|
},
|
|
});
|
|
|
|
assert.equal(r.status, 200, `Expected 200, got ${r.status}`);
|
|
|
|
// Must have text/event-stream Content-Type
|
|
assert.ok(
|
|
(r.headers['content-type'] ?? '').includes('text/event-stream'),
|
|
`Expected text/event-stream, got: ${r.headers['content-type']}`
|
|
);
|
|
|
|
// Must carry all 5 X-OLP-* headers
|
|
assert.ok(r.headers['x-olp-provider-used'], 'Must have X-OLP-Provider-Used');
|
|
assert.ok(r.headers['x-olp-model-used'], 'Must have X-OLP-Model-Used');
|
|
assert.ok(r.headers['x-olp-fallback-hops'] !== undefined, 'Must have X-OLP-Fallback-Hops');
|
|
assert.ok(r.headers['x-olp-cache'], 'Must have X-OLP-Cache');
|
|
assert.ok(r.headers['x-olp-latency-ms'] !== undefined, 'Must have X-OLP-Latency-Ms');
|
|
|
|
// Body must contain [DONE] terminator
|
|
assert.ok(r.body.includes('[DONE]'), `Expected [DONE] in body, got: ${r.body.slice(0, 200)}`);
|
|
} finally {
|
|
if (savedProvider !== undefined) {
|
|
lpD35_9.set('anthropic', savedProvider);
|
|
} else {
|
|
lpD35_9.delete('anthropic');
|
|
}
|
|
}
|
|
});
|
|
|
|
it('#10: post-first-chunk throw → truncation marker (finish_reason:length) + [DONE] in response', async () => {
|
|
// Provider spawn yields 1 delta chunk then throws.
|
|
// Before D35 fix: catch block called res.end() silently — no [DONE], no truncation marker.
|
|
// After D35 fix: emits {type:'stop', finish_reason:'length'} SSE chunk + [DONE] before res.end().
|
|
const { loadedProviders: lpD35_10, __clearCache: ccD35_10 } = await import('./server.mjs');
|
|
const savedProvider = lpD35_10.get('anthropic');
|
|
const oneChunkThenThrow = {
|
|
...savedProvider,
|
|
spawn: async function* () {
|
|
yield { type: 'delta', role: 'assistant', content: 'partial-content' };
|
|
throw new Error('post-first-chunk provider error');
|
|
},
|
|
};
|
|
lpD35_10.set('anthropic', oneChunkThenThrow);
|
|
ccD35_10();
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port: portD35,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'post-chunk-error-test' }],
|
|
stream: true,
|
|
},
|
|
});
|
|
|
|
// Status must be 200 (headers already sent with first chunk)
|
|
assert.equal(r.status, 200, `Expected 200 (headers sent at first chunk), got ${r.status}`);
|
|
|
|
// Body must contain the [DONE] terminator
|
|
assert.ok(r.body.includes('[DONE]'), `Expected [DONE] in body, got: ${r.body.slice(0, 400)}`);
|
|
|
|
// Body must contain the finish_reason:'length' truncation marker
|
|
assert.ok(
|
|
r.body.includes('"length"'),
|
|
`Expected finish_reason:"length" truncation marker in body, got: ${r.body.slice(0, 400)}`
|
|
);
|
|
} finally {
|
|
if (savedProvider !== undefined) {
|
|
lpD35_10.set('anthropic', savedProvider);
|
|
} else {
|
|
lpD35_10.delete('anthropic');
|
|
}
|
|
}
|
|
});
|
|
|
|
it('#10b: post-first-chunk error-chunk → truncation marker + [DONE] (error-chunk variant)', async () => {
|
|
// Provider spawn yields 1 delta then yields an error irChunk (type='error').
|
|
// The error-chunk branch inside the for-await loop mirrors the catch-block fix.
|
|
const { loadedProviders: lpD35_10b, __clearCache: ccD35_10b } = await import('./server.mjs');
|
|
const savedProvider = lpD35_10b.get('anthropic');
|
|
const oneChunkThenErrorChunk = {
|
|
...savedProvider,
|
|
spawn: async function* () {
|
|
yield { type: 'delta', role: 'assistant', content: 'partial-content-b' };
|
|
yield { type: 'error', error: 'provider emitted error chunk after first' };
|
|
},
|
|
};
|
|
lpD35_10b.set('anthropic', oneChunkThenErrorChunk);
|
|
ccD35_10b();
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port: portD35,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'error-chunk-after-first-test' }],
|
|
stream: true,
|
|
},
|
|
});
|
|
|
|
assert.equal(r.status, 200, `Expected 200, got ${r.status}`);
|
|
assert.ok(r.body.includes('[DONE]'), `Expected [DONE] in body, got: ${r.body.slice(0, 400)}`);
|
|
assert.ok(
|
|
r.body.includes('"length"'),
|
|
`Expected finish_reason:"length" in body, got: ${r.body.slice(0, 400)}`
|
|
);
|
|
} finally {
|
|
if (savedProvider !== undefined) {
|
|
lpD35_10b.set('anthropic', savedProvider);
|
|
} else {
|
|
lpD35_10b.delete('anthropic');
|
|
}
|
|
}
|
|
});
|
|
|
|
});
|
|
|
|
// ── Suite 17: D18 — /v1/models population + X-OLP-* headers on errors ────────
|
|
//
|
|
// Finding 10: /v1/models returned empty data regardless of loaded providers.
|
|
// Finding 11: chain-exhausted (and other) error responses omitted the standard
|
|
// X-OLP-* observability headers required by ADR 0004 § Observability.
|
|
//
|
|
// Tests:
|
|
// 17a: /v1/models with anthropic enabled → 200 + 3 entries, owned_by='anthropic'
|
|
// 17b: /v1/models with no providers enabled → 200 + data:[]
|
|
// 17c: /v1/models entries contain only canonical IDs (no alias like 'sonnet')
|
|
// 17d: /v1/models entries match OpenAI spec shape (id/object/created/owned_by only)
|
|
// 17e: chain-exhausted error response carries all 5 X-OLP-* headers
|
|
// 17f: chain-exhausted error X-OLP-Fallback-Hops reflects hops attempted
|
|
// 17g: pre-routing 400 (invalid JSON body) carries X-OLP-Latency-Ms header
|
|
//
|
|
// Authority: OpenAI /v1/models spec (https://platform.openai.com/docs/api-reference/models);
|
|
// ADR 0004 § Observability headers (5-header set required on every response)
|
|
|
|
import {
|
|
createOlpServer as createServer17,
|
|
__setProvidersEnabled as setProviders17,
|
|
__resetProvidersEnabled as resetProviders17,
|
|
__setFallbackConfig as setFallbackConfig17,
|
|
__resetFallbackConfig as resetFallbackConfig17,
|
|
__clearCache as clearCache17,
|
|
createOlpServer as createServer27,
|
|
__setProvidersEnabled as setProviders27,
|
|
__resetProvidersEnabled as resetProviders27,
|
|
createOlpServer as createServer33,
|
|
__setProvidersEnabled as setProviders33,
|
|
__resetProvidersEnabled as resetProviders33,
|
|
__setFallbackConfig as setFallbackConfig33,
|
|
__resetFallbackConfig as resetFallbackConfig33,
|
|
__clearCache as clearCache33,
|
|
loadedProviders as loadedProviders33,
|
|
} from './server.mjs';
|
|
|
|
describe('/v1/models population + X-OLP-* error headers (Suite 17)', () => {
|
|
|
|
// ── 17a: /v1/models with anthropic enabled → 3 canonical + 4 alias entries ─────────────
|
|
|
|
it('17a: /v1/models with anthropic enabled → 200 + 7 entries (3 canonical + 4 aliases) with owned_by="anthropic"', async () => {
|
|
setProviders17({ anthropic: true });
|
|
const s = createServer17();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.object, 'list');
|
|
assert.ok(Array.isArray(body.data), 'data must be an array');
|
|
// Anthropic has 3 canonical models + 4 aliases (claude, sonnet, opus, haiku) in models-registry.json
|
|
assert.equal(body.data.length, 7, `Expected 7 anthropic entries (3 canonical + 4 aliases), got ${body.data.length}`);
|
|
for (const entry of body.data) {
|
|
assert.equal(entry.owned_by, 'anthropic', `Expected owned_by='anthropic', got '${entry.owned_by}'`);
|
|
}
|
|
} finally {
|
|
resetProviders17();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
// ── 17b: /v1/models with no providers enabled → data:[] ──────────────────
|
|
|
|
it('17b: /v1/models with no providers enabled → 200 + data:[]', async () => {
|
|
setProviders17({});
|
|
const s = createServer17();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.object, 'list');
|
|
assert.deepEqual(body.data, [], 'data must be empty when no providers are enabled');
|
|
} finally {
|
|
resetProviders17();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
// ── 17c: /v1/models returns both canonical IDs and alias IDs ───────────────
|
|
// Updated by D27 F15: aliases are now surfaced in /v1/models (canonical-first order).
|
|
|
|
it('17c: /v1/models returns canonical IDs and alias IDs for loaded providers', async () => {
|
|
setProviders17({ anthropic: true });
|
|
const s = createServer17();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
const ids = body.data.map(e => e.id);
|
|
// Canonical IDs must appear
|
|
assert.ok(ids.includes('claude-sonnet-4-6'), 'canonical claude-sonnet-4-6 must appear');
|
|
assert.ok(ids.includes('claude-opus-4-7'), 'canonical claude-opus-4-7 must appear');
|
|
assert.ok(ids.includes('claude-haiku-4-5'), 'canonical claude-haiku-4-5 must appear');
|
|
// Aliases for the loaded (anthropic) provider must also appear
|
|
const anthropicAliases = ['claude', 'sonnet', 'opus', 'haiku'];
|
|
for (const alias of anthropicAliases) {
|
|
assert.ok(ids.includes(alias), `Alias '${alias}' must appear in /v1/models data when anthropic is enabled`);
|
|
}
|
|
// Canonical IDs come before alias IDs (canonical-first ordering)
|
|
const firstAliasIdx = Math.min(...anthropicAliases.map(a => ids.indexOf(a)));
|
|
const lastCanonicalIdx = Math.max(ids.indexOf('claude-sonnet-4-6'), ids.indexOf('claude-opus-4-7'), ids.indexOf('claude-haiku-4-5'));
|
|
assert.ok(lastCanonicalIdx < firstAliasIdx, 'canonical entries must appear before alias entries');
|
|
} finally {
|
|
resetProviders17();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
// ── 17d: /v1/models entries match OpenAI spec shape ───────────────────────
|
|
|
|
it('17d: /v1/models entries match OpenAI spec shape (id/object/created/owned_by)', async () => {
|
|
setProviders17({ anthropic: true });
|
|
const s = createServer17();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.ok(body.data.length > 0, 'Expected at least one model entry');
|
|
for (const entry of body.data) {
|
|
// Must have spec-defined fields
|
|
assert.ok(typeof entry.id === 'string' && entry.id.length > 0, 'id must be a non-empty string');
|
|
assert.equal(entry.object, 'model', "object must be 'model'");
|
|
assert.ok(typeof entry.created === 'number' && entry.created > 0, 'created must be a positive number (Unix epoch seconds)');
|
|
assert.ok(typeof entry.owned_by === 'string' && entry.owned_by.length > 0, 'owned_by must be a non-empty string');
|
|
// Must NOT have invented fields beyond the spec
|
|
const allowedKeys = new Set(['id', 'object', 'created', 'owned_by']);
|
|
for (const key of Object.keys(entry)) {
|
|
assert.ok(allowedKeys.has(key), `Unexpected field '${key}' in /v1/models entry (not in OpenAI spec)`);
|
|
}
|
|
}
|
|
} finally {
|
|
resetProviders17();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
// ── 17e: chain-exhausted error response carries all 5 X-OLP-* headers ─────
|
|
|
|
it('17e: chain-exhausted 502 response carries all 5 standard X-OLP-* observability headers', async () => {
|
|
// Set up a 2-hop chain where both providers fail (SPAWN_FAILED via exit code 1).
|
|
// Both providers must be enabled for the chain to be built.
|
|
setProviders17({ anthropic: true, openai: true });
|
|
setFallbackConfig17({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
providersEnabled: { anthropic: true, openai: true },
|
|
});
|
|
clearCache17();
|
|
|
|
// Fake auth for both providers
|
|
const savedToken = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-17e-anthropic';
|
|
|
|
const { writeFileSync: wfs17e, unlinkSync: uls17e } = await import('node:fs');
|
|
const { tmpdir: td17e } = await import('node:os');
|
|
const { join: pj17e } = await import('node:path');
|
|
const tmpAuth17e = pj17e(td17e(), `olp-test-17e-codex-${Date.now()}.json`);
|
|
wfs17e(tmpAuth17e, JSON.stringify({ accessToken: 'fake-codex-17e' }), 'utf8');
|
|
const savedCodexPath = process.env.OPENAI_CODEX_AUTH_PATH;
|
|
process.env.OPENAI_CODEX_AUTH_PATH = tmpAuth17e;
|
|
|
|
// Both providers: exit code 1 → SPAWN_FAILED hard trigger
|
|
function makeFailSpawn17() {
|
|
return function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 1, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
};
|
|
}
|
|
|
|
__setSpawnImpl(makeFailSpawn17());
|
|
codexSetSpawnImpl(makeFailSpawn17());
|
|
|
|
const s = createServer17();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port: p,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: 'hi' }] },
|
|
});
|
|
assert.ok(r.status >= 400 && r.status < 600, `Expected error status, got ${r.status}`);
|
|
// All 5 standard X-OLP-* headers must be present per ADR 0004 § Observability
|
|
assert.ok(r.headers['x-olp-provider-used'] !== undefined, 'X-OLP-Provider-Used must be present on error response');
|
|
assert.ok(r.headers['x-olp-model-used'] !== undefined, 'X-OLP-Model-Used must be present on error response');
|
|
assert.ok(r.headers['x-olp-fallback-hops'] !== undefined, 'X-OLP-Fallback-Hops must be present on error response');
|
|
assert.ok(r.headers['x-olp-cache'] !== undefined, 'X-OLP-Cache must be present on error response');
|
|
assert.ok(r.headers['x-olp-latency-ms'] !== undefined, 'X-OLP-Latency-Ms must be present on error response');
|
|
} finally {
|
|
resetProviders17();
|
|
resetFallbackConfig17();
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
if (savedToken !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
if (savedCodexPath !== undefined) {
|
|
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexPath;
|
|
} else {
|
|
delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
}
|
|
try { uls17e(tmpAuth17e); } catch { /* ignore */ }
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
// ── 17f: chain-exhausted X-OLP-Fallback-Hops reflects hops attempted ─────
|
|
|
|
it('17f: chain-exhausted error X-OLP-Fallback-Hops reflects hops attempted (2-hop → "2")', async () => {
|
|
// 2-hop chain where both fail → fallbackHops should reflect 2 hops tried
|
|
setProviders17({ anthropic: true, openai: true });
|
|
setFallbackConfig17({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
providersEnabled: { anthropic: true, openai: true },
|
|
});
|
|
clearCache17();
|
|
|
|
const savedToken = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-17f-anthropic';
|
|
|
|
const { writeFileSync: wfs17f, unlinkSync: uls17f } = await import('node:fs');
|
|
const { tmpdir: td17f } = await import('node:os');
|
|
const { join: pj17f } = await import('node:path');
|
|
const tmpAuth17f = pj17f(td17f(), `olp-test-17f-codex-${Date.now()}.json`);
|
|
wfs17f(tmpAuth17f, JSON.stringify({ accessToken: 'fake-codex-17f' }), 'utf8');
|
|
const savedCodexPath17f = process.env.OPENAI_CODEX_AUTH_PATH;
|
|
process.env.OPENAI_CODEX_AUTH_PATH = tmpAuth17f;
|
|
|
|
function makeFailSpawn17f() {
|
|
return function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 1, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
};
|
|
}
|
|
|
|
__setSpawnImpl(makeFailSpawn17f());
|
|
codexSetSpawnImpl(makeFailSpawn17f());
|
|
|
|
const s = createServer17();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
|
|
try {
|
|
const r = await fetch({
|
|
port: p,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: 'hi' }] },
|
|
});
|
|
assert.ok(r.status >= 400, `Expected error status, got ${r.status}`);
|
|
// fallbackHops is set to chain.length (2) when chain exhausted per engine.mjs
|
|
assert.equal(r.headers['x-olp-fallback-hops'], '2',
|
|
`Expected X-OLP-Fallback-Hops: 2 (both hops tried), got: ${r.headers['x-olp-fallback-hops']}`);
|
|
} finally {
|
|
resetProviders17();
|
|
resetFallbackConfig17();
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
if (savedToken !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
if (savedCodexPath17f !== undefined) {
|
|
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexPath17f;
|
|
} else {
|
|
delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
}
|
|
try { uls17f(tmpAuth17f); } catch { /* ignore */ }
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
// ── 17g: pre-routing 400 (invalid JSON) carries full 5 X-OLP-* headers ──────
|
|
// Updated by D32 F8: olpErrorHeaders() now emits all 5 headers with
|
|
// "no provider attempted" defaults (provider='none', model='unknown',
|
|
// hops=0, cache='bypass') on pre-chain error paths. Test updated to assert
|
|
// the full set rather than only X-OLP-Latency-Ms.
|
|
|
|
it('17g: pre-routing 400 (invalid JSON body) carries all 5 X-OLP-* headers with no-provider defaults', async () => {
|
|
// Pre-chain errors inside handleChatCompletions now emit the full 5-header set
|
|
// via olpErrorHeaders() with canonical "no provider attempted" defaults per
|
|
// ADR 0004 § Observability (D32 F8).
|
|
setProviders17({});
|
|
const s = createServer17();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const result = await new Promise((resolve, reject) => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port: p,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json', 'Content-Length': '5' },
|
|
}, res => {
|
|
let data = '';
|
|
res.on('data', c => { data += c; });
|
|
res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: data }));
|
|
});
|
|
req.on('error', reject);
|
|
req.write('{bad}');
|
|
req.end();
|
|
});
|
|
assert.equal(result.status, 400, `Expected 400 for invalid JSON body`);
|
|
// All 5 X-OLP-* headers must be present (D32 F8 olpErrorHeaders)
|
|
assert.ok(result.headers['x-olp-latency-ms'] !== undefined,
|
|
'Pre-routing 400 must carry X-OLP-Latency-Ms');
|
|
const latencyMs = parseInt(result.headers['x-olp-latency-ms'], 10);
|
|
assert.ok(!isNaN(latencyMs) && latencyMs >= 0,
|
|
`X-OLP-Latency-Ms must be non-negative integer, got: ${result.headers['x-olp-latency-ms']}`);
|
|
assert.equal(result.headers['x-olp-provider-used'], 'none',
|
|
'Pre-routing 400 must carry X-OLP-Provider-Used: none (no provider attempted)');
|
|
assert.equal(result.headers['x-olp-model-used'], 'unknown',
|
|
'Pre-routing 400 must carry X-OLP-Model-Used: unknown (IR not parsed yet)');
|
|
assert.equal(result.headers['x-olp-fallback-hops'], '0',
|
|
'Pre-routing 400 must carry X-OLP-Fallback-Hops: 0');
|
|
assert.equal(result.headers['x-olp-cache'], 'bypass',
|
|
'Pre-routing 400 must carry X-OLP-Cache: bypass');
|
|
} finally {
|
|
resetProviders17();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
// ── 17h: 415 wrong Content-Type carries all 5 X-OLP-* headers ────────��─────
|
|
// D32 F8: olpErrorHeaders on 415 pre-chain path.
|
|
|
|
it('17h: 415 wrong Content-Type carries all 5 X-OLP-* headers with no-provider defaults', async () => {
|
|
setProviders17({});
|
|
const s = createServer17();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const result = await new Promise((resolve, reject) => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port: p,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'text/plain', 'Content-Length': '2' },
|
|
}, res => {
|
|
let data = '';
|
|
res.on('data', c => { data += c; });
|
|
res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: data }));
|
|
});
|
|
req.on('error', reject);
|
|
req.write('{}');
|
|
req.end();
|
|
});
|
|
assert.equal(result.status, 415, `Expected 415 for wrong Content-Type, got ${result.status}`);
|
|
assert.equal(result.headers['x-olp-provider-used'], 'none',
|
|
'415 must carry X-OLP-Provider-Used: none');
|
|
assert.equal(result.headers['x-olp-model-used'], 'unknown',
|
|
'415 must carry X-OLP-Model-Used: unknown');
|
|
assert.equal(result.headers['x-olp-fallback-hops'], '0',
|
|
'415 must carry X-OLP-Fallback-Hops: 0');
|
|
assert.equal(result.headers['x-olp-cache'], 'bypass',
|
|
'415 must carry X-OLP-Cache: bypass');
|
|
assert.ok(result.headers['x-olp-latency-ms'] !== undefined,
|
|
'415 must carry X-OLP-Latency-Ms');
|
|
} finally {
|
|
resetProviders17();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
// ── 17i: 503 no-enabled-providers carries all 5 X-OLP-* headers ───────────
|
|
// D32 F8: olpErrorHeaders on 503 no-chain path. X-OLP-Model-Used reflects
|
|
// the requested model (IR was parsed successfully before chain lookup).
|
|
|
|
it('17i: 503 no-enabled-providers carries all 5 X-OLP-* headers with model from IR', async () => {
|
|
setProviders17({});
|
|
const s = createServer17();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({
|
|
port: p,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: 'hi' }] },
|
|
});
|
|
assert.equal(r.status, 503, `Expected 503 for no enabled providers, got ${r.status}`);
|
|
assert.equal(r.headers['x-olp-provider-used'], 'none',
|
|
'503 must carry X-OLP-Provider-Used: none');
|
|
// Model is known (IR was parsed) so model string is propagated
|
|
assert.equal(r.headers['x-olp-model-used'], 'claude-sonnet-4-6',
|
|
'503 must carry X-OLP-Model-Used reflecting the requested model');
|
|
assert.equal(r.headers['x-olp-fallback-hops'], '0',
|
|
'503 must carry X-OLP-Fallback-Hops: 0');
|
|
assert.equal(r.headers['x-olp-cache'], 'bypass',
|
|
'503 must carry X-OLP-Cache: bypass');
|
|
assert.ok(r.headers['x-olp-latency-ms'] !== undefined,
|
|
'503 must carry X-OLP-Latency-Ms');
|
|
} finally {
|
|
resetProviders17();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
});
|
|
|
|
// ── Suite 16: Spawn timeout (P1.3) ───────────────────────────────────────────
|
|
//
|
|
// Tests that:
|
|
// 16a: SPAWN_TIMEOUT is in PROVIDER_ERROR_CODES
|
|
// 16b: SPAWN_TIMEOUT is a hard trigger (evaluateHardTriggers returns true)
|
|
// 16c: Provider that never completes throws SPAWN_TIMEOUT after configured timeout
|
|
// 16d: Fallback chain: primary times out → secondary serves
|
|
//
|
|
// Authority: ADR 0004 § Trigger taxonomy bullet 4; lib/providers/base.mjs
|
|
|
|
import { PROVIDER_ERROR_CODES } from './lib/providers/base.mjs';
|
|
|
|
describe('Spawn timeout (Suite 16)', () => {
|
|
|
|
it('16a: SPAWN_TIMEOUT is in PROVIDER_ERROR_CODES', () => {
|
|
assert.ok(PROVIDER_ERROR_CODES.includes('SPAWN_TIMEOUT'),
|
|
'SPAWN_TIMEOUT must be in PROVIDER_ERROR_CODES');
|
|
});
|
|
|
|
it('16b: evaluateHardTriggers with SPAWN_TIMEOUT code → true (hard trigger)', () => {
|
|
const err = new ProviderError('spawn timed out', 'SPAWN_TIMEOUT');
|
|
assert.equal(evaluateHardTriggers(err), true,
|
|
'SPAWN_TIMEOUT ProviderError must be a hard trigger');
|
|
});
|
|
|
|
it('16c: anthropic plugin with short timeout → throws ProviderError SPAWN_TIMEOUT', async () => {
|
|
// Install a spawn mock that never closes (simulates hanging CLI).
|
|
// Set a very short timeout (100ms) via hints mutation.
|
|
const savedTimeout = anthropic.hints.maxSpawnTimeMs;
|
|
anthropic.hints.maxSpawnTimeMs = 100;
|
|
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
// Never emit close — simulates a hanging process.
|
|
// stdout may emit some data but never closes.
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
try {
|
|
const irReq = makeIR({ model: 'claude-sonnet-4-6', stream: false });
|
|
// Need auth so we don't hit AUTH_MISSING before the timeout path.
|
|
const savedToken = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-16c';
|
|
try {
|
|
const gen = anthropic.spawn(irReq, { accessToken: 'fake-16c' });
|
|
// Drain the generator — it should eventually throw SPAWN_TIMEOUT.
|
|
let caughtError = null;
|
|
try {
|
|
for await (const chunk of gen) { void chunk; }
|
|
} catch (e) {
|
|
caughtError = e;
|
|
}
|
|
assert.ok(caughtError !== null, 'Expected a ProviderError to be thrown');
|
|
assert.ok(caughtError instanceof ProviderError, `Expected ProviderError, got ${caughtError?.constructor?.name}`);
|
|
assert.equal(caughtError.code, 'SPAWN_TIMEOUT', `Expected SPAWN_TIMEOUT code, got ${caughtError?.code}`);
|
|
} finally {
|
|
if (savedToken !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
}
|
|
} finally {
|
|
anthropic.hints.maxSpawnTimeMs = savedTimeout;
|
|
__resetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('16d: fallback chain: primary times out → chain advances to secondary → secondary serves', async () => {
|
|
// Set up: primary = anthropic (will hang + timeout), secondary = openai (will succeed).
|
|
const savedAnthropicTimeout = anthropic.hints.maxSpawnTimeMs;
|
|
anthropic.hints.maxSpawnTimeMs = 100; // short timeout for test speed
|
|
|
|
const savedToken = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-16d-anthropic';
|
|
|
|
// Anthropic mock: hangs forever (triggers SPAWN_TIMEOUT)
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = { write: () => {}, end: () => {} };
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
// Codex mock: succeeds immediately
|
|
codexSetSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stdout.emit('data', Buffer.from('{"content":"fallback-from-timeout-test"}\n'));
|
|
proc.stdout.emit('data', Buffer.from('{"type":"stop"}\n'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
// Set up fake codex auth
|
|
const { writeFileSync: wfs16d, unlinkSync: uls16d } = await import('node:fs');
|
|
const { tmpdir: td16d } = await import('node:os');
|
|
const { join: pj16d } = await import('node:path');
|
|
const tmpAuth16d = pj16d(td16d(), `olp-test-16d-${Date.now()}.json`);
|
|
wfs16d(tmpAuth16d, JSON.stringify({ accessToken: 'fake-codex-16d' }), 'utf8');
|
|
const savedCodexPath = process.env.OPENAI_CODEX_AUTH_PATH;
|
|
process.env.OPENAI_CODEX_AUTH_PATH = tmpAuth16d;
|
|
|
|
// Build a 2-hop chain: anthropic → openai
|
|
const testProviders = loadProviders({ enabled: { anthropic: true, openai: true } });
|
|
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
|
|
const irReq = makeIR({ model: 'claude-sonnet-4-6', stream: false });
|
|
|
|
async function testExecuteHopFn(hopProvider, hopModel, ir) {
|
|
const plugin = testProviders.get(hopProvider);
|
|
if (!plugin) throw Object.assign(new Error(`no provider ${hopProvider}`), { statusCode: 503 });
|
|
const chunks = [];
|
|
for await (const c of plugin.spawn(ir, hopProvider === 'anthropic'
|
|
? { accessToken: 'fake-token-16d-anthropic' }
|
|
: { accessToken: 'fake-codex-16d' })) {
|
|
chunks.push(c);
|
|
if (c.type === 'error') throw new ProviderError(c.error, 'SPAWN_FAILED');
|
|
if (c.type === 'stop') break;
|
|
}
|
|
return chunks;
|
|
}
|
|
|
|
try {
|
|
const result = await executeWithFallback(chain, irReq, testExecuteHopFn, { logEvent: () => {} });
|
|
assert.ok(result.chunks !== null, 'Fallback should have produced chunks');
|
|
assert.equal(result.fallbackHops, 1, 'Should have fallen back to second hop');
|
|
assert.equal(result.providerUsed, 'openai', 'openai should have served after anthropic timed out');
|
|
const content = result.chunks.filter(c => c.type === 'delta').map(c => c.content).join('');
|
|
assert.ok(content.includes('fallback-from-timeout-test'), 'Content from fallback provider expected');
|
|
} finally {
|
|
anthropic.hints.maxSpawnTimeMs = savedAnthropicTimeout;
|
|
__resetSpawnImpl();
|
|
codexResetSpawnImpl();
|
|
if (savedToken !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
if (savedCodexPath !== undefined) {
|
|
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexPath;
|
|
} else {
|
|
delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
}
|
|
try { uls16d(tmpAuth16d); } catch { /* ignore */ }
|
|
}
|
|
});
|
|
|
|
|
|
it('16e: spawn-timeout fires SPAWN_TIMEOUT even when timer races with queued chunks (D24 race fix)', async () => {
|
|
// Exercises the race fixed in D24 (cold-audit round-2 F4).
|
|
//
|
|
// Race path modelled deterministically:
|
|
// 1. Mock stdin.end() schedules data1 + data2 + 'close' via setImmediate
|
|
// (deferred so event handlers registered in the plugin body are ready).
|
|
// 2. Timer is 10ms. Drain loop awaits empty queue. setImmediate fires:
|
|
// push(data1) wakes the drain loop (rejectNext was set, now cleared),
|
|
// push(data2) and push(close) queue up. Drain processes data1, yields it.
|
|
// Generator suspends at yield with rejectNext===null.
|
|
// 3. Consumer (gen.next() resolved) waits 25ms (> 10ms timer).
|
|
// Timer fires: spawnTimedOut=true, rejectNext===null → rejectNext branch
|
|
// SKIPPED. Race condition reproduced.
|
|
// 4. Consumer resumes → drain loop processes data2 then 'close' → breaks.
|
|
// Post-loop: spawnTimedOut=true.
|
|
// Pre-D24: generator returns normally (truncated, silently cacheable).
|
|
// Post-D24: unconditional `if (spawnTimedOut) throw` → SPAWN_TIMEOUT.
|
|
//
|
|
// The timer starts inside the generator body (not at gen-creation time),
|
|
// so it only ticks after the first gen.next() call.
|
|
|
|
const savedTimeout = anthropic.hints.maxSpawnTimeMs;
|
|
anthropic.hints.maxSpawnTimeMs = 10; // fires during the 25ms consumer pause
|
|
|
|
const savedToken = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-16e';
|
|
|
|
__setSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
// Emit data + close via setImmediate so the event handlers (registered
|
|
// AFTER stdin.end() in the plugin body) are already attached when
|
|
// these events fire. All three items land in the same setImmediate
|
|
// callback, so they're all in chunks[] before the drain loop resumes.
|
|
setImmediate(() => {
|
|
proc.stdout.emit('data', Buffer.from('partial-chunk-1'));
|
|
proc.stdout.emit('data', Buffer.from('partial-chunk-2'));
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null); // exitCode=0, done=true
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = (_sig) => { /* already closed */ };
|
|
return proc;
|
|
});
|
|
|
|
try {
|
|
const irReq = makeIR({ model: 'claude-sonnet-4-6', stream: false });
|
|
const gen = anthropic.spawn(irReq, { accessToken: 'fake-16e' });
|
|
|
|
// Pull exactly one chunk. Sequence:
|
|
// - Generator body starts: spawnImpl(), stdin.end() (schedules setImmediate)
|
|
// - Timer armed (10ms)
|
|
// - Drain loop: chunks=[], done=false → enters await (rejectNext set)
|
|
// - setImmediate fires: push(data1), push(data2), close queued, done=true
|
|
// - push(data1) calls resolveNext() → drain resumes
|
|
// - Drain processes data1 → yield → generator suspends. rejectNext=null.
|
|
// - gen.next() promise resolves with the first chunk.
|
|
let firstResult;
|
|
try {
|
|
firstResult = await gen.next();
|
|
} catch (e) {
|
|
throw new Error(`Unexpected throw on first gen.next(): ${e.message}`);
|
|
}
|
|
assert.ok(firstResult && !firstResult.done, 'Expected first gen.next() to yield a chunk');
|
|
|
|
// Wait > 10ms. During this pause:
|
|
// - Timer fires: spawnTimedOut=true, rejectNext===null → branch skipped.
|
|
// - Race condition has now occurred (timer fired with no active rejectNext).
|
|
await new Promise(r => setTimeout(r, 25));
|
|
|
|
// Resume draining. Drain loop processes second data + 'close' → breaks.
|
|
// Post-loop: spawnTimedOut=true.
|
|
// Pre-D24: generator returns normally (done=true, no throw). FAIL.
|
|
// Post-D24: `if (spawnTimedOut) throw ProviderError SPAWN_TIMEOUT`. PASS.
|
|
let caughtError = null;
|
|
try {
|
|
while (true) {
|
|
const { value, done: genDone } = await gen.next();
|
|
if (genDone) break;
|
|
void value;
|
|
}
|
|
} catch (e) {
|
|
caughtError = e;
|
|
}
|
|
|
|
assert.ok(caughtError !== null,
|
|
'Expected ProviderError SPAWN_TIMEOUT (D24 race fix: pre-D24 generator returned normally with partial chunks)');
|
|
assert.ok(caughtError instanceof ProviderError,
|
|
`Expected ProviderError, got ${caughtError?.constructor?.name}`);
|
|
assert.equal(caughtError.code, 'SPAWN_TIMEOUT',
|
|
`Expected SPAWN_TIMEOUT code, got ${caughtError?.code}`);
|
|
} finally {
|
|
anthropic.hints.maxSpawnTimeMs = savedTimeout;
|
|
__resetSpawnImpl();
|
|
if (savedToken !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
}
|
|
});
|
|
|
|
// ── Suite D26: round-3 cleanup batch (F16, F17, F19) ─────────────────────────
|
|
//
|
|
// F16: soft_triggers_deferred_v1x startup warning fires when routing.soft_triggers
|
|
// is non-empty in the fallback config.
|
|
// F17: SPAWN_FAILED from an error chunk in Codex / Mistral plugins includes the
|
|
// accumulated stderr tail in the error message.
|
|
// F19: Streaming path exhausted without stop chunk emits finish_reason:'length'
|
|
// SSE chunk before [DONE] (when partial content was streamed).
|
|
//
|
|
// Authority:
|
|
// F16 — ADR 0004 Amendment 2 § Mitigations (soft-trigger startup warning)
|
|
// F17 — ADR 0004 § Chain advancement step 4 (preserve debug signal)
|
|
// F19 — OpenAI Chat Completions spec (finish_reason:'length');
|
|
// ADR 0005 § Cache write conditions item 1 (D25 F9 no-cache invariant)
|
|
|
|
// ── F16: soft_triggers_deferred_v1x warning ──────────────────────────────
|
|
|
|
describe('D26 F16 — soft_triggers_deferred_v1x startup warning', () => {
|
|
|
|
it('F16: logEvent emits soft_triggers_deferred_v1x warn when soft_triggers is non-empty', () => {
|
|
// Intercept stderr to observe the log event.
|
|
// server.mjs emits the warning at module-evaluation time for _startupConfig.
|
|
// We cannot re-trigger that path without re-loading the module. Instead, we
|
|
// reproduce the exact code path inline (same logEvent call + check) with a
|
|
// synthetic config and monkeypatch process.stderr to capture the output.
|
|
|
|
const written = [];
|
|
const origWrite = process.stderr.write.bind(process.stderr);
|
|
process.stderr.write = (chunk, ...rest) => {
|
|
if (typeof chunk === 'string') written.push(chunk);
|
|
else if (Buffer.isBuffer(chunk)) written.push(chunk.toString());
|
|
return origWrite(chunk, ...rest);
|
|
};
|
|
|
|
try {
|
|
// Simulate what server.mjs does at startup when soft_triggers is non-empty.
|
|
const simulatedConfig = {
|
|
soft_triggers: { anthropic: { p95_latency_ms: 5000 }, openai: { error_rate: 0.1 } },
|
|
};
|
|
const softTriggersConfigured = Object.keys(simulatedConfig.soft_triggers ?? {}).length > 0;
|
|
if (softTriggersConfigured) {
|
|
const entry = {
|
|
ts: new Date().toISOString(),
|
|
level: 'warn',
|
|
event: 'soft_triggers_deferred_v1x',
|
|
configured_providers: Object.keys(simulatedConfig.soft_triggers),
|
|
message: 'routing.soft_triggers configured but soft triggers are deferred to v1.x; ' +
|
|
'thresholds will not fire at v0.1 — see ADR 0004 Amendment 2',
|
|
};
|
|
process.stderr.write(JSON.stringify(entry) + '\n');
|
|
}
|
|
} finally {
|
|
process.stderr.write = origWrite;
|
|
}
|
|
|
|
assert.equal(written.length, 1, `Expected exactly 1 stderr write, got ${written.length}`);
|
|
const parsed = JSON.parse(written[0]);
|
|
assert.equal(parsed.level, 'warn');
|
|
assert.equal(parsed.event, 'soft_triggers_deferred_v1x');
|
|
assert.ok(Array.isArray(parsed.configured_providers), 'configured_providers must be an array');
|
|
assert.ok(parsed.configured_providers.includes('anthropic'), 'must include anthropic provider');
|
|
assert.ok(parsed.configured_providers.includes('openai'), 'must include openai provider');
|
|
assert.ok(typeof parsed.message === 'string' && parsed.message.includes('v1.x'),
|
|
'message must mention v1.x deferral');
|
|
});
|
|
|
|
it('F16: no warning emitted when soft_triggers is empty', () => {
|
|
const written = [];
|
|
const origWrite = process.stderr.write.bind(process.stderr);
|
|
process.stderr.write = (chunk, ...rest) => {
|
|
if (typeof chunk === 'string') written.push(chunk);
|
|
else if (Buffer.isBuffer(chunk)) written.push(chunk.toString());
|
|
return origWrite(chunk, ...rest);
|
|
};
|
|
|
|
try {
|
|
const simulatedConfig = { soft_triggers: {} };
|
|
const softTriggersConfigured = Object.keys(simulatedConfig.soft_triggers ?? {}).length > 0;
|
|
if (softTriggersConfigured) {
|
|
process.stderr.write('should-not-appear\n');
|
|
}
|
|
} finally {
|
|
process.stderr.write = origWrite;
|
|
}
|
|
|
|
assert.equal(written.length, 0, 'No stderr writes expected when soft_triggers is empty');
|
|
});
|
|
|
|
it('F16: no warning emitted when soft_triggers is absent (undefined)', () => {
|
|
const written = [];
|
|
const origWrite = process.stderr.write.bind(process.stderr);
|
|
process.stderr.write = (chunk, ...rest) => {
|
|
if (typeof chunk === 'string') written.push(chunk);
|
|
else if (Buffer.isBuffer(chunk)) written.push(chunk.toString());
|
|
return origWrite(chunk, ...rest);
|
|
};
|
|
|
|
try {
|
|
const simulatedConfig = {};
|
|
const softTriggersConfigured = Object.keys(simulatedConfig.soft_triggers ?? {}).length > 0;
|
|
if (softTriggersConfigured) {
|
|
process.stderr.write('should-not-appear\n');
|
|
}
|
|
} finally {
|
|
process.stderr.write = origWrite;
|
|
}
|
|
|
|
assert.equal(written.length, 0, 'No stderr writes expected when soft_triggers is absent');
|
|
});
|
|
|
|
});
|
|
|
|
// ── F17: stderr propagation on error-chunk SPAWN_FAILED ──────────────────
|
|
|
|
describe('D26 F17 — stderr propagation in error-chunk SPAWN_FAILED (Codex + Mistral)', () => {
|
|
|
|
it('F17 codex: error chunk throw includes stderr tail when stderr is non-empty', async () => {
|
|
// Inject a mock spawn that emits stderr then an NDJSON error event then exits 0.
|
|
// Verify that the thrown ProviderError message contains both the error text
|
|
// and the stderr tail separated by ' | stderr: '.
|
|
const stderrPayload = 'codex: quota exceeded — plan limit reached';
|
|
|
|
codexSetSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stderr.emit('data', Buffer.from(stderrPayload));
|
|
proc.stdout.emit('data', Buffer.from('{"type":"error","error":"upstream 429"}\n'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
try {
|
|
const ir = makeIR({ model: 'gpt-5.5', stream: false });
|
|
const authCtx = { accessToken: 'fake-codex-f17' };
|
|
let caught = null;
|
|
try {
|
|
for await (const _chunk of codex.spawn(ir, authCtx)) { /* drain */ } // eslint-disable-line no-unused-vars
|
|
} catch (e) {
|
|
caught = e;
|
|
}
|
|
assert.ok(caught instanceof ProviderError,
|
|
`Expected ProviderError, got ${caught?.constructor?.name ?? String(caught)}`);
|
|
assert.equal(caught.code, 'SPAWN_FAILED');
|
|
assert.ok(
|
|
caught.message.includes('upstream 429'),
|
|
`Expected message to include 'upstream 429', got: ${caught.message}`,
|
|
);
|
|
assert.ok(
|
|
caught.message.includes('codex: quota exceeded'),
|
|
`Expected message to include stderr tail, got: ${caught.message}`,
|
|
);
|
|
assert.ok(
|
|
caught.message.includes('| stderr:'),
|
|
`Expected '| stderr:' separator in message, got: ${caught.message}`,
|
|
);
|
|
} finally {
|
|
codexResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('F17 codex: error chunk message has no stderr suffix when stderr is empty', async () => {
|
|
codexSetSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
// No stderr — just the error line
|
|
proc.stdout.emit('data', Buffer.from('{"type":"error","error":"plain error"}\n'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
try {
|
|
const ir = makeIR({ model: 'gpt-5.5', stream: false });
|
|
const authCtx = { accessToken: 'fake-codex-f17b' };
|
|
let caught = null;
|
|
try {
|
|
for await (const _chunk of codex.spawn(ir, authCtx)) { /* drain */ } // eslint-disable-line no-unused-vars
|
|
} catch (e) {
|
|
caught = e;
|
|
}
|
|
assert.ok(caught instanceof ProviderError, `Expected ProviderError, got ${caught?.constructor?.name}`);
|
|
assert.equal(caught.code, 'SPAWN_FAILED');
|
|
assert.ok(caught.message.includes('plain error'),
|
|
`Expected 'plain error' in message, got: ${caught.message}`);
|
|
assert.ok(!caught.message.includes('| stderr:'),
|
|
`Expected no stderr suffix when stderr is empty, got: ${caught.message}`);
|
|
} finally {
|
|
codexResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
it('F17 mistral: error chunk throw includes stderr tail when stderr is non-empty', async () => {
|
|
const stderrPayload = 'vibe: api key invalid';
|
|
|
|
mistralSetSpawnImpl(function (_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: () => {
|
|
setImmediate(() => {
|
|
proc.stderr.emit('data', Buffer.from(stderrPayload));
|
|
proc.stdout.emit('data', Buffer.from('{"type":"error","error":"auth failure"}\n'));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
});
|
|
|
|
try {
|
|
const ir = makeIR({ model: 'devstral-2-25-12', stream: false });
|
|
const authCtx = { apiKey: 'fake-mistral-f17' };
|
|
let caught = null;
|
|
try {
|
|
for await (const _chunk of mistral.spawn(ir, authCtx)) { /* drain */ } // eslint-disable-line no-unused-vars
|
|
} catch (e) {
|
|
caught = e;
|
|
}
|
|
assert.ok(caught instanceof ProviderError,
|
|
`Expected ProviderError, got ${caught?.constructor?.name ?? String(caught)}`);
|
|
assert.equal(caught.code, 'SPAWN_FAILED');
|
|
assert.ok(caught.message.includes('auth failure'),
|
|
`Expected 'auth failure' in message, got: ${caught.message}`);
|
|
assert.ok(caught.message.includes('vibe: api key invalid'),
|
|
`Expected stderr tail in message, got: ${caught.message}`);
|
|
assert.ok(caught.message.includes('| stderr:'),
|
|
`Expected '| stderr:' separator, got: ${caught.message}`);
|
|
} finally {
|
|
mistralResetSpawnImpl();
|
|
}
|
|
});
|
|
|
|
});
|
|
|
|
// ── F19: streaming truncation marker before [DONE] ───────────────────────
|
|
|
|
describe('D26 F19 — streaming truncation marker on stop-less exhaustion', () => {
|
|
|
|
let serverF19;
|
|
let portF19;
|
|
let savedTokenF19;
|
|
|
|
before(async () => {
|
|
savedTokenF19 = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-f19';
|
|
__setProvidersEnabled({ anthropic: true });
|
|
|
|
const { createOlpServer: s19, __clearCache: cc19 } = await import('./server.mjs');
|
|
cc19();
|
|
serverF19 = s19();
|
|
await new Promise((resolve, reject) => {
|
|
serverF19.listen(0, '127.0.0.1', resolve);
|
|
serverF19.once('error', reject);
|
|
});
|
|
portF19 = serverF19.address().port;
|
|
});
|
|
|
|
after(async () => {
|
|
__resetProvidersEnabled();
|
|
__resetSpawnImpl();
|
|
if (savedTokenF19 !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedTokenF19;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
if (!serverF19) return;
|
|
return new Promise(r => serverF19.close(r));
|
|
});
|
|
|
|
it('F19: no-stop generator emits finish_reason:length SSE chunk before [DONE]', async () => {
|
|
// Inject a provider that yields partial content then exhausts without a stop chunk.
|
|
const { loadedProviders: lpF19, __clearCache: ccF19 } = await import('./server.mjs');
|
|
const savedProviderA = lpF19.get('anthropic');
|
|
|
|
let spawnCalled = 0;
|
|
const noStopProvider = {
|
|
...savedProviderA,
|
|
hints: { ...savedProviderA?.hints, cacheable: true },
|
|
spawn: async function* () {
|
|
spawnCalled++;
|
|
yield { type: 'delta', role: 'assistant', content: 'partial content' };
|
|
// No stop chunk — generator exhausts here.
|
|
},
|
|
};
|
|
lpF19.set('anthropic', noStopProvider);
|
|
ccF19();
|
|
|
|
const sseLines = await new Promise((resolve, reject) => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port: portF19,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
}, res => {
|
|
let data = '';
|
|
res.on('data', d => { data += d.toString(); });
|
|
res.on('end', () => resolve(data.split('\n').filter(Boolean)));
|
|
res.on('error', reject);
|
|
});
|
|
req.on('error', reject);
|
|
req.write(JSON.stringify({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'f19-truncation-test' }],
|
|
stream: true,
|
|
}));
|
|
req.end();
|
|
});
|
|
|
|
try {
|
|
const dataLines = sseLines.filter(l => l.startsWith('data: '));
|
|
const doneLines = dataLines.filter(l => l === 'data: [DONE]');
|
|
const nonDoneLines = dataLines.filter(l => l !== 'data: [DONE]');
|
|
|
|
// Must have: delta chunk + truncation marker + [DONE]
|
|
assert.ok(nonDoneLines.length >= 2,
|
|
`Expected >= 2 non-[DONE] data lines (delta + truncation), got ${nonDoneLines.length}: ${JSON.stringify(nonDoneLines)}`);
|
|
assert.equal(doneLines.length, 1, 'Expected exactly one [DONE] line');
|
|
|
|
// [DONE] must be the last data line
|
|
const lastDataLine = dataLines[dataLines.length - 1];
|
|
assert.equal(lastDataLine, 'data: [DONE]', `[DONE] must be last data line, got: ${lastDataLine}`);
|
|
|
|
// Line immediately before [DONE] must be the truncation marker
|
|
const lineBeforeDone = dataLines[dataLines.length - 2];
|
|
assert.ok(lineBeforeDone?.startsWith('data: '),
|
|
`Expected data line before [DONE], got: ${lineBeforeDone}`);
|
|
const truncPayload = JSON.parse(lineBeforeDone.slice(6).trim());
|
|
assert.equal(
|
|
truncPayload.choices?.[0]?.finish_reason,
|
|
'length',
|
|
`Expected finish_reason:'length' on truncation marker, got: ${JSON.stringify(truncPayload.choices?.[0])}`,
|
|
);
|
|
assert.equal(spawnCalled, 1, 'Spawn must have been called exactly once');
|
|
} finally {
|
|
if (savedProviderA !== undefined) {
|
|
lpF19.set('anthropic', savedProviderA);
|
|
} else {
|
|
lpF19.delete('anthropic');
|
|
}
|
|
}
|
|
});
|
|
|
|
it('F19: zero-content no-stop generator does NOT emit truncation marker', async () => {
|
|
// When streamedChunks.length === 0, no truncation marker should appear.
|
|
const { loadedProviders: lpF19b, __clearCache: ccF19b } = await import('./server.mjs');
|
|
const savedProviderB = lpF19b.get('anthropic');
|
|
|
|
const emptyProvider = {
|
|
...savedProviderB,
|
|
hints: { ...savedProviderB?.hints, cacheable: true },
|
|
spawn: async function* () {
|
|
// Yields nothing — generator returns immediately.
|
|
},
|
|
};
|
|
lpF19b.set('anthropic', emptyProvider);
|
|
ccF19b();
|
|
|
|
const sseLines = await new Promise((resolve, reject) => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port: portF19,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
}, res => {
|
|
let data = '';
|
|
res.on('data', d => { data += d.toString(); });
|
|
res.on('end', () => resolve(data.split('\n').filter(Boolean)));
|
|
res.on('error', reject);
|
|
});
|
|
req.on('error', reject);
|
|
req.write(JSON.stringify({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'f19-empty-test' }],
|
|
stream: true,
|
|
}));
|
|
req.end();
|
|
});
|
|
|
|
try {
|
|
const dataLines = sseLines.filter(l => l.startsWith('data: '));
|
|
for (const line of dataLines) {
|
|
if (line === 'data: [DONE]') continue;
|
|
const payload = JSON.parse(line.slice(6).trim());
|
|
assert.ok(
|
|
payload.choices?.[0]?.finish_reason !== 'length',
|
|
`Expected no finish_reason:length when no content streamed, got: ${JSON.stringify(payload.choices?.[0])}`,
|
|
);
|
|
}
|
|
} finally {
|
|
if (savedProviderB !== undefined) {
|
|
lpF19b.set('anthropic', savedProviderB);
|
|
} else {
|
|
lpF19b.delete('anthropic');
|
|
}
|
|
}
|
|
});
|
|
|
|
it('F19: D25 F9 no-cache invariant preserved — second request after no-stop triggers fresh spawn', async () => {
|
|
// Verify F19 did not accidentally break D25 F9's no-cache-on-truncation guarantee.
|
|
const { loadedProviders: lpF19c, __clearCache: ccF19c } = await import('./server.mjs');
|
|
const savedProviderC = lpF19c.get('anthropic');
|
|
|
|
let spawnCount = 0;
|
|
const noStopProvider2 = {
|
|
...savedProviderC,
|
|
hints: { ...savedProviderC?.hints, cacheable: true },
|
|
spawn: async function* () {
|
|
spawnCount++;
|
|
yield { type: 'delta', role: 'assistant', content: 'f19-nocache-check' };
|
|
// No stop chunk.
|
|
},
|
|
};
|
|
lpF19c.set('anthropic', noStopProvider2);
|
|
ccF19c();
|
|
|
|
const makeStreamRequest = () => new Promise((resolve, reject) => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port: portF19,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
}, res => {
|
|
let data = '';
|
|
res.on('data', d => { data += d.toString(); });
|
|
res.on('end', () => resolve({ status: res.statusCode, body: data, headers: res.headers }));
|
|
res.on('error', reject);
|
|
});
|
|
req.on('error', reject);
|
|
req.write(JSON.stringify({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'f19-nocache-repeated-query' }],
|
|
stream: true,
|
|
}));
|
|
req.end();
|
|
});
|
|
|
|
try {
|
|
const r1 = await makeStreamRequest();
|
|
assert.equal(r1.status, 200, `r1 status: ${r1.status}`);
|
|
assert.equal(r1.headers['x-olp-cache'], 'miss', 'r1 must be cache miss');
|
|
assert.equal(spawnCount, 1, 'spawn called once for r1');
|
|
|
|
const r2 = await makeStreamRequest();
|
|
assert.equal(r2.status, 200, `r2 status: ${r2.status}`);
|
|
assert.equal(r2.headers['x-olp-cache'], 'miss',
|
|
'r2 must also be cache miss — D25 F9 no-cache invariant preserved');
|
|
assert.equal(spawnCount, 2, 'spawn called again for r2');
|
|
} finally {
|
|
if (savedProviderC !== undefined) {
|
|
lpF19c.set('anthropic', savedProviderC);
|
|
} else {
|
|
lpF19c.delete('anthropic');
|
|
}
|
|
}
|
|
});
|
|
|
|
});
|
|
|
|
// ── Suite D27: round-3 batch (F8, F15) ────────────────────────────────────────
|
|
//
|
|
// F8: validateIRRequest now validates response_format and tool_choice.
|
|
// F15: /v1/models now surfaces alias entries for loaded providers.
|
|
//
|
|
// Authority:
|
|
// F8 — ADR 0003 § Optional fields (response_format object; tool_choice string/object)
|
|
// F15 — OpenAI /v1/models spec (https://platform.openai.com/docs/api-reference/models);
|
|
// models-registry.json alias map (SPOT per D17)
|
|
|
|
// ── F8: response_format and tool_choice validation ────────────────────────────
|
|
|
|
describe('D27 F8 — validateIRRequest response_format + tool_choice validation', () => {
|
|
|
|
it('F8: response_format object with string .type is accepted', () => {
|
|
const r = validateIRRequest(makeIR({ response_format: { type: 'json_object' } }));
|
|
assert.equal(r.valid, true);
|
|
assert.deepEqual(r.errors, []);
|
|
});
|
|
|
|
it('F8: response_format string (not object) is rejected', () => {
|
|
const r = validateIRRequest(makeIR({ response_format: 'json_object' }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('response_format must be an object')),
|
|
`Expected 'response_format must be an object', got: ${JSON.stringify(r.errors)}`);
|
|
});
|
|
|
|
it('F8: response_format object with non-string .type is rejected', () => {
|
|
const r = validateIRRequest(makeIR({ response_format: { type: 42 } }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('response_format.type must be a string')),
|
|
`Expected 'response_format.type must be a string', got: ${JSON.stringify(r.errors)}`);
|
|
});
|
|
|
|
it('F8: response_format undefined is accepted (optional field)', () => {
|
|
const ir = makeIR();
|
|
delete ir.response_format;
|
|
const r = validateIRRequest(ir);
|
|
assert.equal(r.valid, true);
|
|
});
|
|
|
|
it("F8: tool_choice 'auto' is accepted", () => {
|
|
const r = validateIRRequest(makeIR({ tool_choice: 'auto' }));
|
|
assert.equal(r.valid, true);
|
|
assert.deepEqual(r.errors, []);
|
|
});
|
|
|
|
it("F8: tool_choice 'none' is accepted", () => {
|
|
const r = validateIRRequest(makeIR({ tool_choice: 'none' }));
|
|
assert.equal(r.valid, true);
|
|
assert.deepEqual(r.errors, []);
|
|
});
|
|
|
|
it("F8: tool_choice 'required' is accepted", () => {
|
|
const r = validateIRRequest(makeIR({ tool_choice: 'required' }));
|
|
assert.equal(r.valid, true);
|
|
assert.deepEqual(r.errors, []);
|
|
});
|
|
|
|
it('F8: tool_choice unknown string is rejected', () => {
|
|
const r = validateIRRequest(makeIR({ tool_choice: 'any' }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes("tool_choice string must be 'auto' | 'none' | 'required'")),
|
|
`Expected tool_choice string error, got: ${JSON.stringify(r.errors)}`);
|
|
});
|
|
|
|
it('F8: tool_choice {type:"function", function:{name:"X"}} is accepted', () => {
|
|
const r = validateIRRequest(makeIR({ tool_choice: { type: 'function', function: { name: 'my_fn' } } }));
|
|
assert.equal(r.valid, true);
|
|
assert.deepEqual(r.errors, []);
|
|
});
|
|
|
|
it('F8: tool_choice {type:"tool"} (wrong type field) is rejected', () => {
|
|
const r = validateIRRequest(makeIR({ tool_choice: { type: 'tool' } }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes("tool_choice.type must be 'function'")),
|
|
`Expected tool_choice.type error, got: ${JSON.stringify(r.errors)}`);
|
|
});
|
|
|
|
it('F8: tool_choice {} (object but no type field) is rejected', () => {
|
|
const r = validateIRRequest(makeIR({ tool_choice: {} }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes("tool_choice.type must be 'function'")),
|
|
`Expected tool_choice.type error for empty object, got: ${JSON.stringify(r.errors)}`);
|
|
});
|
|
|
|
it('F8: tool_choice undefined is accepted (optional field)', () => {
|
|
const ir = makeIR();
|
|
delete ir.tool_choice;
|
|
const r = validateIRRequest(ir);
|
|
assert.equal(r.valid, true);
|
|
});
|
|
|
|
it('F8: tool_choice number is rejected (not string or object)', () => {
|
|
const r = validateIRRequest(makeIR({ tool_choice: 42 }));
|
|
assert.equal(r.valid, false);
|
|
assert.ok(r.errors.some(e => e.includes('tool_choice must be a string or an object')),
|
|
`Expected 'tool_choice must be a string or an object', got: ${JSON.stringify(r.errors)}`);
|
|
});
|
|
|
|
});
|
|
|
|
// ── F15: /v1/models alias surfacing ───────────────────────────────────────────
|
|
|
|
describe('D27 F15 — /v1/models alias surfacing', () => {
|
|
|
|
it('F15a: /v1/models with anthropic enabled contains all canonical IDs and all 4 anthropic aliases', async () => {
|
|
setProviders27({ anthropic: true });
|
|
const s = createServer27();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
const ids = body.data.map(e => e.id);
|
|
// Canonical IDs
|
|
assert.ok(ids.includes('claude-opus-4-7'), 'canonical claude-opus-4-7 must appear');
|
|
assert.ok(ids.includes('claude-sonnet-4-6'), 'canonical claude-sonnet-4-6 must appear');
|
|
assert.ok(ids.includes('claude-haiku-4-5'), 'canonical claude-haiku-4-5 must appear');
|
|
// Anthropic aliases
|
|
for (const alias of ['claude', 'sonnet', 'opus', 'haiku']) {
|
|
assert.ok(ids.includes(alias), `alias '${alias}' must appear when anthropic is enabled`);
|
|
}
|
|
} finally {
|
|
resetProviders27();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
it('F15b: each alias entry has owned_by equal to its canonical target provider', async () => {
|
|
setProviders27({ anthropic: true });
|
|
const s = createServer27();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
const aliasMap = getAliasMap();
|
|
for (const entry of body.data) {
|
|
if (aliasMap.has(entry.id)) {
|
|
const { providerName } = aliasMap.get(entry.id);
|
|
assert.equal(entry.owned_by, providerName,
|
|
`alias '${entry.id}' must have owned_by='${providerName}', got '${entry.owned_by}'`);
|
|
}
|
|
}
|
|
} finally {
|
|
resetProviders27();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
it('F15c: /v1/models with no providers enabled returns empty data (no aliases)', async () => {
|
|
setProviders27({});
|
|
const s = createServer27();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.deepEqual(body.data, [], 'data must be empty when no providers are enabled (no aliases either)');
|
|
} finally {
|
|
resetProviders27();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
it('F15d: /v1/models with anthropic+mistral enabled contains both providers\' canonicals and aliases', async () => {
|
|
setProviders27({ anthropic: true, mistral: true });
|
|
const s = createServer27();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
const ids = body.data.map(e => e.id);
|
|
// Anthropic canonicals + aliases
|
|
assert.ok(ids.includes('claude-sonnet-4-6'), 'anthropic canonical must appear');
|
|
assert.ok(ids.includes('sonnet'), 'anthropic alias sonnet must appear');
|
|
assert.ok(ids.includes('claude'), 'anthropic alias claude must appear');
|
|
// Mistral canonicals + aliases
|
|
assert.ok(ids.includes('devstral-2-25-12'), 'mistral canonical must appear');
|
|
assert.ok(ids.includes('devstral'), 'mistral alias devstral must appear');
|
|
// Verify minimum total count: 3 anthropic + 4 anthropic aliases + 2 mistral + 4 mistral aliases = 13
|
|
assert.ok(body.data.length >= 13,
|
|
`Expected >=13 entries for anthropic+mistral, got ${body.data.length}`);
|
|
} finally {
|
|
resetProviders27();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
it('F15e: alias entries for disabled providers do not appear', async () => {
|
|
// Only anthropic enabled — codex aliases (codex, codex-spark, gpt5, gpt5-mini) must NOT appear
|
|
setProviders27({ anthropic: true });
|
|
const s = createServer27();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
const ids = body.data.map(e => e.id);
|
|
for (const alias of ['codex', 'codex-spark', 'gpt5', 'gpt5-mini', 'devstral', 'devstral-2']) {
|
|
assert.ok(!ids.includes(alias),
|
|
`Alias '${alias}' must NOT appear when its provider is not enabled`);
|
|
}
|
|
} finally {
|
|
resetProviders27();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
});
|
|
|
|
// ── Suite D33: round-5 cold-audit cleanup batch ───────────────────────────────
|
|
//
|
|
// F3: function_call deprecated field → deterministic IR id (no Date.now())
|
|
// F5: /health returns per-provider healthCheck() snapshots
|
|
// F8_fallback: fallback-hop cache-hit → X-OLP-Cache: hit (not miss)
|
|
// F12: /v1/models `created` is stable per-model timestamp
|
|
//
|
|
// Authority:
|
|
// F3 — ADR 0005 § "same inputs → same key, no random, no timestamp"
|
|
// F5 — ADR 0002 § Provider contract (healthCheck)
|
|
// F8_fallback — ADR 0005 § D1 cache; ADR 0004 § Observability headers
|
|
// F12 — OpenAI /v1/models spec (https://platform.openai.com/docs/api-reference/models);
|
|
// models-registry.json bootstrapCreated fallback
|
|
|
|
describe('D33 round-5 cold-audit cleanup', () => {
|
|
|
|
// ── F3: deterministic function_call ID ────────────────────────────────────
|
|
|
|
describe('F3 — function_call deprecated field → deterministic cache key', () => {
|
|
|
|
it('F3a: two identical function_call requests produce the same IR id', () => {
|
|
const req = {
|
|
model: 'test-model',
|
|
messages: [
|
|
{
|
|
role: 'assistant',
|
|
function_call: { name: 'get_weather', arguments: '{"city":"Paris"}' },
|
|
},
|
|
],
|
|
};
|
|
const ir1 = openAIToIR(req);
|
|
const ir2 = openAIToIR(req);
|
|
const id1 = ir1.messages[0].tool_calls?.[0]?.id;
|
|
const id2 = ir2.messages[0].tool_calls?.[0]?.id;
|
|
assert.ok(id1, 'function_call must produce a tool_calls entry with an id');
|
|
assert.equal(id1, id2,
|
|
`Two identical function_call requests must produce the same id; got ${id1} vs ${id2}`);
|
|
assert.ok(id1.startsWith('fc-'), 'id must start with "fc-"');
|
|
});
|
|
|
|
it('F3b: two identical function_call requests produce the same cache key', () => {
|
|
const req = {
|
|
model: 'test-model',
|
|
messages: [
|
|
{
|
|
role: 'assistant',
|
|
function_call: { name: 'my_tool', arguments: '{}' },
|
|
},
|
|
],
|
|
};
|
|
const ir1 = openAIToIR(req);
|
|
const ir2 = openAIToIR(req);
|
|
const key1 = computeCacheKey('anthropic', 'claude-sonnet-4-6', ir1);
|
|
const key2 = computeCacheKey('anthropic', 'claude-sonnet-4-6', ir2);
|
|
assert.equal(key1, key2,
|
|
'Identical function_call requests must produce identical cache keys');
|
|
});
|
|
|
|
it('F3c: different function_call names produce different cache keys', () => {
|
|
const req1 = openAIToIR({
|
|
model: 'test-model',
|
|
messages: [{ role: 'assistant', function_call: { name: 'tool_a', arguments: '{}' } }],
|
|
});
|
|
const req2 = openAIToIR({
|
|
model: 'test-model',
|
|
messages: [{ role: 'assistant', function_call: { name: 'tool_b', arguments: '{}' } }],
|
|
});
|
|
const key1 = computeCacheKey('anthropic', 'claude-sonnet-4-6', req1);
|
|
const key2 = computeCacheKey('anthropic', 'claude-sonnet-4-6', req2);
|
|
assert.notEqual(key1, key2,
|
|
'Different function_call names must produce different cache keys');
|
|
});
|
|
|
|
});
|
|
|
|
// ── F5: /health per-provider snapshot ─────────────────────────────────────
|
|
|
|
describe('F5 — /health returns per-provider healthCheck() snapshots', () => {
|
|
|
|
it('F5a: /health with no providers enabled returns providers.status = {}', async () => {
|
|
setProviders33({});
|
|
const s = createServer33();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/health' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.ok(body.ok, '/health must return ok:true');
|
|
assert.ok('status' in body.providers,
|
|
'/health providers must include a "status" key');
|
|
assert.deepEqual(body.providers.status, {},
|
|
'status must be empty when no providers are loaded');
|
|
} finally {
|
|
resetProviders33();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
it('F5b: /health with anthropic enabled includes providers.status.anthropic', async () => {
|
|
setProviders33({ anthropic: true });
|
|
const s = createServer33();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/health' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.ok('anthropic' in body.providers.status,
|
|
'providers.status must include anthropic when it is enabled');
|
|
const ahc = body.providers.status.anthropic;
|
|
assert.ok(typeof ahc === 'object' && ahc !== null,
|
|
'providers.status.anthropic must be an object');
|
|
assert.ok('ok' in ahc,
|
|
'providers.status.anthropic must have an "ok" field');
|
|
} finally {
|
|
resetProviders33();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
it('F5c: /health includes status for each loaded provider', async () => {
|
|
setProviders33({ anthropic: true, mistral: true });
|
|
const s = createServer33();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/health' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.providers.enabled, 2, 'enabled must be 2');
|
|
assert.ok('anthropic' in body.providers.status, 'anthropic status must appear');
|
|
assert.ok('mistral' in body.providers.status, 'mistral status must appear');
|
|
assert.ok(!('openai' in body.providers.status),
|
|
'openai must NOT appear in status when not loaded');
|
|
} finally {
|
|
resetProviders33();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
it('F5d: /health status entry has ok:false + error on healthCheck() throw', async () => {
|
|
setProviders33({ anthropic: true });
|
|
const savedProvider = loadedProviders33.get('anthropic');
|
|
// Inject a provider whose healthCheck always throws
|
|
const throwingProvider = {
|
|
...savedProvider,
|
|
healthCheck: async () => { throw new Error('probe-failed'); },
|
|
};
|
|
loadedProviders33.set('anthropic', throwingProvider);
|
|
const s = createServer33();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/health' });
|
|
assert.equal(r.status, 200, '/health must still return 200 even when healthCheck throws');
|
|
const body = JSON.parse(r.body);
|
|
assert.ok(body.ok, 'top-level ok must still be true');
|
|
const ahc = body.providers.status.anthropic;
|
|
assert.equal(ahc.ok, false, 'status.anthropic.ok must be false when healthCheck throws');
|
|
assert.ok(typeof ahc.error === 'string', 'status.anthropic.error must be a string');
|
|
assert.ok(ahc.error.includes('probe-failed'), 'error must include the thrown message');
|
|
} finally {
|
|
if (savedProvider !== undefined) {
|
|
loadedProviders33.set('anthropic', savedProvider);
|
|
} else {
|
|
loadedProviders33.delete('anthropic');
|
|
}
|
|
resetProviders33();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
});
|
|
|
|
// ── F8_fallback: fallback-hop cache-hit → X-OLP-Cache: hit ───────────────
|
|
|
|
describe('F8_fallback — fallback-hop cache-hit reports X-OLP-Cache: hit', () => {
|
|
|
|
it('F8_fallback: 2-hop chain, primary fails, secondary cache-hit → X-OLP-Cache: hit', async () => {
|
|
// Two providers: primary (alpha) always fails with SPAWN_FAILED (hard trigger); secondary
|
|
// (beta) succeeds. On second request, secondary serves from cache.
|
|
// The X-OLP-Cache header must be 'hit' on the second request.
|
|
//
|
|
// Uses an explicit chain config: [alpha → beta]. alpha is enabled but fails;
|
|
// beta is enabled and succeeds. Secondary's cache key is different from primary's
|
|
// (different provider name), so the second request hits beta's cache.
|
|
|
|
setProviders33({ anthropic: true, mistral: true });
|
|
clearCache33();
|
|
|
|
// Override anthropic spawn to always throw SPAWN_FAILED (hard trigger)
|
|
const savedAnthropic = loadedProviders33.get('anthropic');
|
|
const failingPrimary = {
|
|
...savedAnthropic,
|
|
spawn: async function* () {
|
|
throw new ProviderError('spawn failed (test)', 'SPAWN_FAILED');
|
|
},
|
|
};
|
|
loadedProviders33.set('anthropic', failingPrimary);
|
|
|
|
// Override mistral spawn to yield a valid response
|
|
const savedMistral = loadedProviders33.get('mistral');
|
|
let secondarySpawnCount = 0;
|
|
const succeedingSecondary = {
|
|
...savedMistral,
|
|
spawn: async function* () {
|
|
secondarySpawnCount++;
|
|
yield { type: 'delta', role: 'assistant', content: 'fallback-response' };
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
},
|
|
};
|
|
loadedProviders33.set('mistral', succeedingSecondary);
|
|
|
|
// Wire a 2-hop chain: anthropic (claude-sonnet-4-6) → mistral (devstral-2-25-12)
|
|
setFallbackConfig33({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'mistral', model: 'devstral-2-25-12' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
providersEnabled: { anthropic: true, mistral: true },
|
|
});
|
|
|
|
const s = createServer33();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
|
|
const makeRequest = () => fetch({
|
|
port: p,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'f8-fallback-cache-test' }],
|
|
stream: false,
|
|
},
|
|
});
|
|
|
|
try {
|
|
// First request: primary fails → secondary spawns → result cached under secondary key
|
|
const r1 = await makeRequest();
|
|
assert.equal(r1.status, 200, `r1 must be 200; got ${r1.status}`);
|
|
assert.equal(r1.headers['x-olp-cache'], 'miss',
|
|
'r1 must be cache miss (first time secondary serves)');
|
|
assert.equal(secondarySpawnCount, 1, 'secondary must spawn once for r1');
|
|
|
|
// Second request: primary still fails → secondary serves from its own cache
|
|
const r2 = await makeRequest();
|
|
assert.equal(r2.status, 200, `r2 must be 200; got ${r2.status}`);
|
|
assert.equal(r2.headers['x-olp-cache'], 'hit',
|
|
'r2 must be cache hit — fallback hop served from secondary cache');
|
|
assert.equal(secondarySpawnCount, 1,
|
|
'secondary must NOT spawn again for r2 (served from cache)');
|
|
} finally {
|
|
if (savedAnthropic !== undefined) {
|
|
loadedProviders33.set('anthropic', savedAnthropic);
|
|
} else {
|
|
loadedProviders33.delete('anthropic');
|
|
}
|
|
if (savedMistral !== undefined) {
|
|
loadedProviders33.set('mistral', savedMistral);
|
|
} else {
|
|
loadedProviders33.delete('mistral');
|
|
}
|
|
resetFallbackConfig33();
|
|
resetProviders33();
|
|
clearCache33();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
});
|
|
|
|
// ── F12: /v1/models stable `created` timestamps ────���──────────────────────
|
|
|
|
describe('F12 — /v1/models stable per-model created timestamp', () => {
|
|
|
|
it('F12a: getModelCreated returns a stable number for known model IDs', () => {
|
|
// These models have explicit created fields in models-registry.json
|
|
const claudeSonnet = getModelCreated('claude-sonnet-4-6');
|
|
const claudeSonnet2 = getModelCreated('claude-sonnet-4-6');
|
|
assert.equal(claudeSonnet, claudeSonnet2,
|
|
'getModelCreated must return the same value on repeated calls');
|
|
assert.ok(typeof claudeSonnet === 'number' && claudeSonnet > 0,
|
|
'getModelCreated must return a positive number');
|
|
// claude-sonnet-4-6 has created=1775001600 (2026-04-01)
|
|
assert.equal(claudeSonnet, 1775001600,
|
|
'claude-sonnet-4-6 created must match models-registry.json entry');
|
|
});
|
|
|
|
it('F12b: getModelCreated falls back to REGISTRY_BOOTSTRAP_CREATED for unknown IDs', () => {
|
|
const unknown = getModelCreated('non-existent-model-xyz');
|
|
assert.equal(unknown, REGISTRY_BOOTSTRAP_CREATED,
|
|
'unknown model must fall back to REGISTRY_BOOTSTRAP_CREATED');
|
|
});
|
|
|
|
it('F12c: REGISTRY_BOOTSTRAP_CREATED matches models-registry.json bootstrapCreated', () => {
|
|
assert.equal(REGISTRY_BOOTSTRAP_CREATED, modelsRegistry.bootstrapCreated,
|
|
'REGISTRY_BOOTSTRAP_CREATED must equal models-registry.json bootstrapCreated');
|
|
});
|
|
|
|
it('F12d: /v1/models returns stable created for canonical models (not Date.now())', async () => {
|
|
setProviders33({ anthropic: true });
|
|
const s = createServer33();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r1 = await fetch({ port: p, method: 'GET', path: '/v1/models' });
|
|
const r2 = await fetch({ port: p, method: 'GET', path: '/v1/models' });
|
|
assert.equal(r1.status, 200);
|
|
assert.equal(r2.status, 200);
|
|
const body1 = JSON.parse(r1.body);
|
|
const body2 = JSON.parse(r2.body);
|
|
|
|
// Both responses must have same created values for all entries
|
|
const entries1 = Object.fromEntries(body1.data.map(e => [e.id, e.created]));
|
|
const entries2 = Object.fromEntries(body2.data.map(e => [e.id, e.created]));
|
|
for (const [id, ts] of Object.entries(entries1)) {
|
|
assert.equal(entries2[id], ts,
|
|
`created for '${id}' must be stable across requests; got ${ts} vs ${entries2[id]}`);
|
|
}
|
|
|
|
// claude-sonnet-4-6 must use the registry value, not Date.now()
|
|
const sonnetEntry = body1.data.find(e => e.id === 'claude-sonnet-4-6');
|
|
assert.ok(sonnetEntry, 'claude-sonnet-4-6 must appear in /v1/models');
|
|
assert.equal(sonnetEntry.created, 1775001600,
|
|
'claude-sonnet-4-6 created must be 1775001600 (2026-04-01), not Date.now()');
|
|
} finally {
|
|
resetProviders33();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
it('F12e: alias entries in /v1/models use the canonical model\'s created timestamp', async () => {
|
|
setProviders33({ anthropic: true });
|
|
const s = createServer33();
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(0, '127.0.0.1', resolve);
|
|
s.once('error', reject);
|
|
});
|
|
const p = s.address().port;
|
|
try {
|
|
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
// 'sonnet' alias → canonical 'claude-sonnet-4-6' → created 1775001600
|
|
const sonnetAlias = body.data.find(e => e.id === 'sonnet');
|
|
const sonnetCanon = body.data.find(e => e.id === 'claude-sonnet-4-6');
|
|
assert.ok(sonnetAlias, '"sonnet" alias must appear in /v1/models');
|
|
assert.ok(sonnetCanon, 'claude-sonnet-4-6 canonical must appear in /v1/models');
|
|
assert.equal(sonnetAlias.created, sonnetCanon.created,
|
|
'alias "sonnet" must have same created as canonical "claude-sonnet-4-6"');
|
|
} finally {
|
|
resetProviders33();
|
|
await new Promise(r => s.close(r));
|
|
}
|
|
});
|
|
|
|
});
|
|
|
|
});
|
|
|
|
});
|
|
|
|
// ── Suite 18: D38 — maxConcurrent runtime enforcement (issue #1) ──────────
|
|
//
|
|
// Authority:
|
|
// - ADR 0002 Amendment 6 (D38): maxConcurrent runtime enforcement landed.
|
|
// - ADR 0004 Amendment 4 (D38): CONCURRENCY_LIMIT added to hard-trigger taxonomy.
|
|
// - GitHub issue #1 (maxConcurrent: declarative-only at v0.1).
|
|
//
|
|
// Tests:
|
|
// 18a: CONCURRENCY_LIMIT is in PROVIDER_ERROR_CODES.
|
|
// 18b: CONCURRENCY_LIMIT is a hard trigger (evaluateHardTriggers true).
|
|
// 18c: classifyTrigger returns 'hard' for CONCURRENCY_LIMIT.
|
|
// 18d: tryAcquireSpawn / releaseSpawn / getActiveSpawnCount unit behaviour.
|
|
// 18e: tryAcquireSpawn returns false at the limit.
|
|
// 18f: releaseSpawn throws when called without a matching acquire.
|
|
// 18g: DEFAULT_MAX_CONCURRENT_SPAWNS applies when maxConcurrent omitted.
|
|
// 18h: __resetSpawnCounters clears all in-flight counts.
|
|
// 18i: HTTP — 5 concurrent buffered requests against a maxConcurrent:2 mock →
|
|
// peak in-flight == 2; requests 3-5 hit chain-exhausted error.
|
|
// 18j: HTTP — counter releases after request completes (single-hop, sequential).
|
|
// 18k: Fallback — saturated primary (maxConcurrent:1) → secondary serves the 2nd.
|
|
// 18l: HTTP streaming — counter releases at end of stream, not at start.
|
|
|
|
describe('D38 — maxConcurrent runtime enforcement (Suite 18)', () => {
|
|
|
|
// ── 18a: PROVIDER_ERROR_CODES contains CONCURRENCY_LIMIT ──────────────────
|
|
|
|
it('18a: CONCURRENCY_LIMIT is in PROVIDER_ERROR_CODES', () => {
|
|
assert.ok(
|
|
PROVIDER_ERROR_CODES.includes('CONCURRENCY_LIMIT'),
|
|
'CONCURRENCY_LIMIT must be in PROVIDER_ERROR_CODES per ADR 0004 Amendment 4',
|
|
);
|
|
});
|
|
|
|
// ── 18b: CONCURRENCY_LIMIT triggers hard fallback ─────────────────────────
|
|
|
|
it('18b: evaluateHardTriggers(ProviderError CONCURRENCY_LIMIT) → true', () => {
|
|
const err = new ProviderError('provider stub at maxConcurrent (2)', 'CONCURRENCY_LIMIT');
|
|
assert.equal(
|
|
evaluateHardTriggers(err), true,
|
|
'CONCURRENCY_LIMIT must be classified as a hard trigger',
|
|
);
|
|
});
|
|
|
|
// ── 18c: AUTH_MISSING regression guard ───────────────────────────────────
|
|
// CONCURRENCY_LIMIT hard-trigger classification is covered by 18b;
|
|
// 18c guards that D38 did NOT accidentally flip AUTH_MISSING (a soft signal
|
|
// for v0.1 — see ADR 0004 Amendment 2) to hard alongside CONCURRENCY_LIMIT.
|
|
|
|
it('18c: evaluateHardTriggers still false for AUTH_MISSING (regression guard)', () => {
|
|
// Regression guard: D38 didn't accidentally flip AUTH_MISSING to a hard trigger.
|
|
const err = new ProviderError('auth missing', 'AUTH_MISSING');
|
|
assert.equal(evaluateHardTriggers(err), false);
|
|
});
|
|
|
|
// ── 18d: semaphore unit tests ─────────────────────────────────────────────
|
|
|
|
describe('18d — tryAcquireSpawn / releaseSpawn / getActiveSpawnCount unit', () => {
|
|
|
|
// Use a unique provider name to avoid colliding with other tests' state.
|
|
// __resetSpawnCounters() runs after each unit test to keep them isolated.
|
|
|
|
it('18d.1: tryAcquireSpawn increments count from 0 → 1', () => {
|
|
__resetSpawnCounters();
|
|
assert.equal(getActiveSpawnCount('unit-d1'), 0);
|
|
const ok = tryAcquireSpawn('unit-d1', 4);
|
|
assert.equal(ok, true, 'first acquire should succeed');
|
|
assert.equal(getActiveSpawnCount('unit-d1'), 1);
|
|
__resetSpawnCounters();
|
|
});
|
|
|
|
it('18d.2: releaseSpawn decrements count back to 0', () => {
|
|
__resetSpawnCounters();
|
|
tryAcquireSpawn('unit-d2', 4);
|
|
assert.equal(getActiveSpawnCount('unit-d2'), 1);
|
|
releaseSpawn('unit-d2');
|
|
assert.equal(getActiveSpawnCount('unit-d2'), 0);
|
|
__resetSpawnCounters();
|
|
});
|
|
|
|
it('18d.3: each provider has independent counters', () => {
|
|
__resetSpawnCounters();
|
|
tryAcquireSpawn('unit-d3-a', 4);
|
|
tryAcquireSpawn('unit-d3-a', 4);
|
|
tryAcquireSpawn('unit-d3-b', 4);
|
|
assert.equal(getActiveSpawnCount('unit-d3-a'), 2);
|
|
assert.equal(getActiveSpawnCount('unit-d3-b'), 1);
|
|
__resetSpawnCounters();
|
|
});
|
|
|
|
it('18d.4: counter is removed from map when it reaches 0', () => {
|
|
__resetSpawnCounters();
|
|
tryAcquireSpawn('unit-d4', 4);
|
|
releaseSpawn('unit-d4');
|
|
// After release-to-zero, getActiveSpawnCount still returns 0 (Map.delete'd).
|
|
assert.equal(getActiveSpawnCount('unit-d4'), 0);
|
|
__resetSpawnCounters();
|
|
});
|
|
});
|
|
|
|
// ── 18e: tryAcquireSpawn returns false at the limit ───────────────────────
|
|
|
|
it('18e: tryAcquireSpawn returns false when count == maxConcurrent', () => {
|
|
__resetSpawnCounters();
|
|
assert.equal(tryAcquireSpawn('unit-e', 2), true);
|
|
assert.equal(tryAcquireSpawn('unit-e', 2), true);
|
|
// At the limit now — third acquire must fail without incrementing.
|
|
assert.equal(tryAcquireSpawn('unit-e', 2), false,
|
|
'acquire must fail when current count equals limit');
|
|
// Counter must not have incremented on the failed acquire.
|
|
assert.equal(getActiveSpawnCount('unit-e'), 2,
|
|
'failed acquire must not increment the counter');
|
|
__resetSpawnCounters();
|
|
});
|
|
|
|
// ── 18f: releaseSpawn without acquire throws ──────────────────────────────
|
|
|
|
it('18f: releaseSpawn without matching acquire throws', () => {
|
|
__resetSpawnCounters();
|
|
assert.throws(
|
|
() => releaseSpawn('unit-f-never-acquired'),
|
|
/releaseSpawn.*counter would go negative/,
|
|
'release without acquire must throw to surface the bug loudly',
|
|
);
|
|
__resetSpawnCounters();
|
|
});
|
|
|
|
// ── 18g: DEFAULT_MAX_CONCURRENT_SPAWNS fallback ───────────────────────────
|
|
|
|
it('18g: DEFAULT_MAX_CONCURRENT_SPAWNS applies when maxConcurrent argument omitted', () => {
|
|
__resetSpawnCounters();
|
|
assert.ok(Number.isInteger(DEFAULT_MAX_CONCURRENT_SPAWNS) && DEFAULT_MAX_CONCURRENT_SPAWNS > 0,
|
|
'DEFAULT_MAX_CONCURRENT_SPAWNS must be a positive integer');
|
|
// Acquire up to the default limit without arg — all should succeed.
|
|
for (let i = 0; i < DEFAULT_MAX_CONCURRENT_SPAWNS; i++) {
|
|
assert.equal(tryAcquireSpawn('unit-g'), true, `acquire #${i + 1} should succeed`);
|
|
}
|
|
// Next acquire (still no arg) should fail at the default boundary.
|
|
assert.equal(tryAcquireSpawn('unit-g'), false,
|
|
`acquire #${DEFAULT_MAX_CONCURRENT_SPAWNS + 1} should fail at default limit`);
|
|
__resetSpawnCounters();
|
|
});
|
|
|
|
it('18g.2: non-integer maxConcurrent coerces to DEFAULT_MAX_CONCURRENT_SPAWNS', () => {
|
|
__resetSpawnCounters();
|
|
// Defensive coercion: a plugin path that bypasses validateProvider could
|
|
// produce undefined/null/NaN. The gate must not blow up; it must use the default.
|
|
assert.equal(tryAcquireSpawn('unit-g2', undefined), true);
|
|
assert.equal(tryAcquireSpawn('unit-g2', null), true);
|
|
assert.equal(tryAcquireSpawn('unit-g2', NaN), true);
|
|
assert.equal(tryAcquireSpawn('unit-g2', 'four'), true);
|
|
// After 4 acquires (matching default), 5th should fail.
|
|
assert.equal(getActiveSpawnCount('unit-g2'), 4);
|
|
assert.equal(tryAcquireSpawn('unit-g2', undefined), false);
|
|
__resetSpawnCounters();
|
|
});
|
|
|
|
// ── 18h: __resetSpawnCounters clears state ────────────────────────────────
|
|
|
|
it('18h: __resetSpawnCounters clears all in-flight counts', () => {
|
|
tryAcquireSpawn('unit-h-1', 4);
|
|
tryAcquireSpawn('unit-h-2', 4);
|
|
assert.equal(getActiveSpawnCount('unit-h-1'), 1);
|
|
assert.equal(getActiveSpawnCount('unit-h-2'), 1);
|
|
__resetSpawnCounters();
|
|
assert.equal(getActiveSpawnCount('unit-h-1'), 0);
|
|
assert.equal(getActiveSpawnCount('unit-h-2'), 0);
|
|
});
|
|
|
|
// ────────────────────────────────────────────────────────────────────────────
|
|
// HTTP integration tests
|
|
// ────────────────────────────────────────────────────────────────────────────
|
|
//
|
|
// Strategy: install a controllable mock spawn on the anthropic plugin that
|
|
// exposes a Promise-based "release gate" — the mock yields no data until
|
|
// the test explicitly resolves the gate. This lets us hold multiple spawns
|
|
// in-flight at the same time deterministically (no timing races).
|
|
//
|
|
// For the maxConcurrent:N tests we override anthropic.hints.maxConcurrent
|
|
// for the duration of the test, then restore. ProviderError synthesised by
|
|
// the gate carries CONCURRENCY_LIMIT and the fallback engine treats it as
|
|
// a hard trigger. With a single-hop chain the chain exhausts and the
|
|
// client receives a 502/non-200.
|
|
|
|
describe('18 HTTP integration', () => {
|
|
let server18;
|
|
let port18;
|
|
let savedToken18;
|
|
let savedMaxConcurrent;
|
|
|
|
before(async () => {
|
|
savedToken18 = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-suite-18';
|
|
__setProvidersEnabled({ anthropic: true });
|
|
__resetSpawnCounters();
|
|
|
|
const { createOlpServer: s18, __clearCache: cc18 } = await import('./server.mjs');
|
|
cc18();
|
|
server18 = s18();
|
|
await new Promise((resolve, reject) => {
|
|
server18.listen(0, '127.0.0.1', resolve);
|
|
server18.once('error', reject);
|
|
});
|
|
port18 = server18.address().port;
|
|
});
|
|
|
|
after(async () => {
|
|
__resetProvidersEnabled();
|
|
__resetSpawnImpl();
|
|
__resetSpawnCounters();
|
|
if (savedToken18 !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken18;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
if (!server18) return;
|
|
return new Promise(r => server18.close(r));
|
|
});
|
|
|
|
/**
|
|
* Makes a controllable mock spawn that completes only when the given
|
|
* gate Promise resolves. Records peak in-flight via `peakRef`.
|
|
*/
|
|
function makeGatedMockSpawn(gatePromise, content, peakRef, activeRef) {
|
|
return function gatedSpawn(_bin, _args, _opts) {
|
|
const proc = new EventEmitter();
|
|
proc.stdout = new EventEmitter();
|
|
proc.stderr = new EventEmitter();
|
|
proc.stdin = {
|
|
write: () => {},
|
|
end: async () => {
|
|
// Track this spawn as in-flight from the moment stdin closes.
|
|
activeRef.count++;
|
|
if (activeRef.count > peakRef.peak) peakRef.peak = activeRef.count;
|
|
// Wait for the gate to resolve before emitting any data.
|
|
await gatePromise;
|
|
// Decrement on exit.
|
|
setImmediate(() => {
|
|
proc.stdout.emit('data', Buffer.from(content));
|
|
proc.stdout.emit('end');
|
|
proc.stderr.emit('end');
|
|
proc.emit('close', 0, null);
|
|
activeRef.count--;
|
|
});
|
|
},
|
|
};
|
|
proc.killed = false;
|
|
proc.kill = () => {};
|
|
return proc;
|
|
};
|
|
}
|
|
|
|
// ── 18i: 5 concurrent requests, maxConcurrent:2 ────────────────────────
|
|
|
|
it('18i: 5 concurrent buffered requests with maxConcurrent:2 → peak in-flight == 2', async () => {
|
|
// Override anthropic's maxConcurrent for this test.
|
|
savedMaxConcurrent = anthropic.hints.maxConcurrent;
|
|
anthropic.hints.maxConcurrent = 2;
|
|
|
|
const { __clearCache: cc18i } = await import('./server.mjs');
|
|
cc18i();
|
|
|
|
// Gate that we'll resolve to let mock spawns complete.
|
|
let releaseGate;
|
|
const gatePromise = new Promise(resolve => { releaseGate = resolve; });
|
|
const peakRef = { peak: 0 };
|
|
const activeRef = { count: 0 };
|
|
__setSpawnImpl(makeGatedMockSpawn(gatePromise, 'concurrency-test-response', peakRef, activeRef));
|
|
|
|
try {
|
|
// Fire 5 concurrent buffered requests with DIFFERENT message content
|
|
// so cache lookups don't collapse them.
|
|
const reqs = [];
|
|
for (let i = 0; i < 5; i++) {
|
|
reqs.push(fetch({
|
|
port: port18,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: `concurrency-test-msg-${i}-${Date.now()}` }],
|
|
},
|
|
}));
|
|
}
|
|
|
|
// Give the server a moment to receive all requests and attempt acquires
|
|
// (the requests that succeed acquire will wait on gatePromise; the
|
|
// ones that fail acquire will return immediately with an error).
|
|
// The wait must be long enough for all 5 to either enter the spawn or
|
|
// be rejected by the gate, but short enough to keep the test fast.
|
|
await new Promise(r => setTimeout(r, 100));
|
|
|
|
// Peak in-flight count must equal exactly 2 (== maxConcurrent).
|
|
assert.equal(peakRef.peak, 2,
|
|
`peak in-flight must be exactly 2 (== maxConcurrent), observed ${peakRef.peak}`);
|
|
// Also assert via the exported getActiveSpawnCount helper (covers a
|
|
// different path — the module-level _activeSpawns map vs the mock-side
|
|
// counter). Both must agree.
|
|
assert.equal(getActiveSpawnCount('anthropic'), 2,
|
|
'module-level active-spawn counter must report 2 in-flight');
|
|
|
|
// Release the gate so the 2 in-flight spawns can complete.
|
|
releaseGate();
|
|
|
|
// Wait for all 5 requests to settle.
|
|
const results = await Promise.all(reqs);
|
|
|
|
// 2 successes (requests that acquired), 3 errors (CONCURRENCY_LIMIT).
|
|
const successes = results.filter(r => r.status === 200);
|
|
const failures = results.filter(r => r.status !== 200);
|
|
assert.equal(successes.length, 2,
|
|
`expected exactly 2 successful responses, got ${successes.length} (statuses: ${results.map(r => r.status).join(', ')})`);
|
|
assert.equal(failures.length, 3,
|
|
`expected exactly 3 failed responses (chain-exhausted on CONCURRENCY_LIMIT), got ${failures.length}`);
|
|
} finally {
|
|
// Restore original maxConcurrent.
|
|
anthropic.hints.maxConcurrent = savedMaxConcurrent;
|
|
__resetSpawnImpl();
|
|
__resetSpawnCounters();
|
|
}
|
|
});
|
|
|
|
// ── 18j: counter releases after request completes ─────────────────────
|
|
|
|
it('18j: counter releases to 0 after buffered request completes', async () => {
|
|
savedMaxConcurrent = anthropic.hints.maxConcurrent;
|
|
anthropic.hints.maxConcurrent = 2;
|
|
const { __clearCache: cc18j } = await import('./server.mjs');
|
|
cc18j();
|
|
|
|
// Plain mock that completes immediately.
|
|
__setSpawnImpl(makeMockSpawn(['release-test-response']));
|
|
|
|
try {
|
|
// Fire request 1 and wait for completion.
|
|
const r1 = await fetch({
|
|
port: port18,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: `release-test-msg-1-${Date.now()}` }],
|
|
},
|
|
});
|
|
assert.equal(r1.status, 200);
|
|
// Counter must be back to 0 after the request completes.
|
|
assert.equal(getActiveSpawnCount('anthropic'), 0,
|
|
'counter must release to 0 after request completes');
|
|
|
|
// Fire request 2 (different content for cache miss) — must also succeed,
|
|
// proving the slot is reusable.
|
|
const r2 = await fetch({
|
|
port: port18,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: `release-test-msg-2-${Date.now()}` }],
|
|
},
|
|
});
|
|
assert.equal(r2.status, 200);
|
|
assert.equal(getActiveSpawnCount('anthropic'), 0,
|
|
'counter must release after second request too');
|
|
} finally {
|
|
anthropic.hints.maxConcurrent = savedMaxConcurrent;
|
|
__resetSpawnImpl();
|
|
__resetSpawnCounters();
|
|
}
|
|
});
|
|
|
|
// ── 18l: streaming counter release ────────────────────────────────────
|
|
|
|
it('18l: streaming request — counter releases at end of stream', async () => {
|
|
savedMaxConcurrent = anthropic.hints.maxConcurrent;
|
|
anthropic.hints.maxConcurrent = 2;
|
|
const { __clearCache: cc18l } = await import('./server.mjs');
|
|
cc18l();
|
|
|
|
__setSpawnImpl(makeMockSpawn(['stream-release-content']));
|
|
|
|
try {
|
|
// Make a streaming request and consume the full stream.
|
|
const port = port18;
|
|
const body = await new Promise((resolve, reject) => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
}, res => {
|
|
let data = '';
|
|
res.on('data', c => { data += c; });
|
|
res.on('end', () => resolve(data));
|
|
res.on('error', reject);
|
|
});
|
|
req.on('error', reject);
|
|
req.write(JSON.stringify({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: `stream-release-${Date.now()}` }],
|
|
stream: true,
|
|
}));
|
|
req.end();
|
|
});
|
|
assert.ok(body.includes('data:'), 'streaming body must contain SSE data lines');
|
|
|
|
// After the stream completes (res.on('end') resolved), the counter must
|
|
// be back to 0. Tests the streaming-branch finally{} fired.
|
|
assert.equal(getActiveSpawnCount('anthropic'), 0,
|
|
'counter must release after streaming response completes');
|
|
} finally {
|
|
anthropic.hints.maxConcurrent = savedMaxConcurrent;
|
|
__resetSpawnImpl();
|
|
__resetSpawnCounters();
|
|
}
|
|
});
|
|
});
|
|
|
|
// ── 18k: fallback test — saturated primary advances to secondary ──────────
|
|
//
|
|
// Uses a 2-hop chain executed via executeWithFallback (unit-level, not HTTP).
|
|
// Primary has maxConcurrent:1 and is already saturated by a synthetic
|
|
// pre-acquire; second request must advance to the fallback hop.
|
|
|
|
describe('18k — fallback advances on saturation', () => {
|
|
after(() => __resetSpawnCounters());
|
|
|
|
it('18k: saturated primary (maxConcurrent:1) → fallback hop serves the request', async () => {
|
|
// Pre-saturate the 'anthropic' counter (1/1) so the next acquire fails.
|
|
__resetSpawnCounters();
|
|
tryAcquireSpawn('anthropic', 1);
|
|
assert.equal(getActiveSpawnCount('anthropic'), 1);
|
|
|
|
// Build a 2-hop chain: anthropic (saturated) → openai (will serve).
|
|
// executeHopFn for this test calls tryAcquireSpawn itself, mirroring
|
|
// the production gate semantics. On failure: throws ProviderError
|
|
// CONCURRENCY_LIMIT; fallback engine advances to next hop.
|
|
const chain = [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
];
|
|
const ir = makeIR({ model: 'claude-sonnet-4-6', stream: false });
|
|
|
|
async function testHopFn(hopProvider, hopModel, _ir) {
|
|
const max = hopProvider === 'anthropic' ? 1 : 4;
|
|
if (!tryAcquireSpawn(hopProvider, max)) {
|
|
const err = new ProviderError(
|
|
`provider ${hopProvider} at maxConcurrent (${max})`,
|
|
'CONCURRENCY_LIMIT',
|
|
);
|
|
err.providerName = hopProvider;
|
|
err.maxConcurrent = max;
|
|
err.activeSpawns = max;
|
|
throw err;
|
|
}
|
|
try {
|
|
// Yield a single stop chunk so the hop "succeeds".
|
|
return [{ type: 'delta', content: `served-by-${hopProvider}` }, { type: 'stop', finish_reason: 'stop' }];
|
|
} finally {
|
|
releaseSpawn(hopProvider);
|
|
}
|
|
}
|
|
|
|
const result = await executeWithFallback(chain, ir, testHopFn, { logEvent: () => {} });
|
|
assert.ok(result.chunks !== null, 'fallback should have produced chunks');
|
|
assert.equal(result.fallbackHops, 1, 'must have advanced to second hop');
|
|
assert.equal(result.providerUsed, 'openai',
|
|
'secondary (openai) must serve when primary (anthropic) is saturated');
|
|
const content = result.chunks.filter(c => c.type === 'delta').map(c => c.content).join('');
|
|
assert.ok(content.includes('served-by-openai'), 'content must come from fallback hop');
|
|
|
|
// Cleanup: release the synthetic pre-saturation slot.
|
|
releaseSpawn('anthropic');
|
|
assert.equal(getActiveSpawnCount('anthropic'), 0);
|
|
});
|
|
});
|
|
|
|
});
|
|
|
|
// ── Suite 19: lib/keys.mjs — multi-key auth (ADR 0007, D44) ───────────────
|
|
|
|
import { mkdtempSync, rmSync, statSync } from 'node:fs';
|
|
import { tmpdir } from 'node:os';
|
|
import { join as pathJoin } from 'node:path';
|
|
import {
|
|
createKey, listKeys, revokeKey, validateKey, touchLastUsed,
|
|
generateToken, hashToken, validateManifest,
|
|
readManifest, writeManifestAtomic,
|
|
SCHEMA_VERSION, ENV_OWNER_KEY_ID, ANONYMOUS_KEY_ID, ENV_OWNER_VAR,
|
|
__setTouchInterleaveHook, __resetWriteLocks, __writeLockSize,
|
|
} from './lib/keys.mjs';
|
|
|
|
describe('Suite 19 — lib/keys.mjs multi-key auth (ADR 0007, D44)', () => {
|
|
|
|
describe('19a-d — Token generation (§ 5)', () => {
|
|
it('19a: generateToken produces olp_<43-char base64url> (47 chars total)', () => {
|
|
const t = generateToken();
|
|
assert.match(t, /^olp_[A-Za-z0-9_-]{43}$/);
|
|
assert.equal(t.length, 47);
|
|
});
|
|
|
|
it('19b: consecutive generateToken calls produce different tokens', () => {
|
|
assert.notEqual(generateToken(), generateToken());
|
|
});
|
|
|
|
it('19c: hashToken returns 64-char lowercase hex sha256', () => {
|
|
assert.match(hashToken('olp_test'), /^[a-f0-9]{64}$/);
|
|
});
|
|
|
|
it('19d: hashToken is deterministic', () => {
|
|
assert.equal(hashToken('hello'), hashToken('hello'));
|
|
});
|
|
});
|
|
|
|
describe('19e-j — Manifest write + read (§ 4, § 6.1)', () => {
|
|
let TMP;
|
|
before(() => { TMP = mkdtempSync(pathJoin(tmpdir(), 'olp-keys-19e-')); });
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); __resetWriteLocks(); });
|
|
|
|
it('19e: createKey writes a valid manifest readable by readManifest', () => {
|
|
const { id, plaintext_token, manifest } = createKey({
|
|
name: 'test-key',
|
|
owner_tier: 'owner',
|
|
providers_enabled: ['anthropic'],
|
|
olpHome: TMP,
|
|
});
|
|
assert.match(plaintext_token, /^olp_[A-Za-z0-9_-]{43}$/);
|
|
assert.equal(manifest.schema_version, SCHEMA_VERSION);
|
|
assert.equal(manifest.id, id);
|
|
assert.equal(manifest.owner_tier, 'owner');
|
|
assert.deepEqual(manifest.providers_enabled, ['anthropic']);
|
|
assert.equal(manifest.token_hash_algo, 'sha256');
|
|
assert.equal(manifest.token_hash, hashToken(plaintext_token));
|
|
assert.equal(manifest.revoked_at, null);
|
|
assert.equal(manifest.last_used_at, null);
|
|
|
|
const reread = readManifest(id, { olpHome: TMP });
|
|
assert.deepEqual(reread, manifest);
|
|
});
|
|
|
|
it('19f: manifest file mode 0600 + key dir mode 0700 enforced', () => {
|
|
const { id } = createKey({ name: 'mode-test', olpHome: TMP });
|
|
const fileMode = statSync(pathJoin(TMP, 'keys', id, 'manifest.json')).mode & 0o777;
|
|
const dirMode = statSync(pathJoin(TMP, 'keys', id)).mode & 0o777;
|
|
assert.equal(fileMode, 0o600, 'manifest must be 0600');
|
|
assert.equal(dirMode, 0o700, 'key dir must be 0700');
|
|
});
|
|
|
|
it('19g: readManifest returns null for non-existent key', () => {
|
|
assert.equal(readManifest('does-not-exist', { olpHome: TMP }), null);
|
|
});
|
|
|
|
it('19h: validateManifest rejects unrecognized schema_version', () => {
|
|
assert.throws(
|
|
() => validateManifest({
|
|
schema_version: 99, id: 'x', name: 'x', token_hash: 'x',
|
|
token_hash_algo: 'sha256', owner_tier: 'owner',
|
|
providers_enabled: '*', created_at: 'x',
|
|
}),
|
|
/manifest_invalid: unrecognized schema_version 99/,
|
|
);
|
|
});
|
|
|
|
it('19i: validateManifest rejects bad owner_tier', () => {
|
|
assert.throws(
|
|
() => validateManifest({
|
|
schema_version: 1, id: 'x', name: 'x', token_hash: 'x',
|
|
token_hash_algo: 'sha256', owner_tier: 'admin',
|
|
providers_enabled: '*', created_at: 'x',
|
|
}),
|
|
/owner_tier must be "owner" or "guest"/,
|
|
);
|
|
});
|
|
|
|
it('19j: createKey rejects empty name + bad owner_tier + bad providers_enabled', () => {
|
|
assert.throws(() => createKey({ name: '', olpHome: TMP }), /name is required/);
|
|
assert.throws(() => createKey({ name: 'x', owner_tier: 'admin', olpHome: TMP }), /owner_tier must be "owner" or "guest"/);
|
|
assert.throws(() => createKey({ name: 'x', providers_enabled: 'invalid', olpHome: TMP }), /providers_enabled must be/);
|
|
});
|
|
});
|
|
|
|
describe('19k-p — validateKey (§ 5, § 6.3.5, § 9.4)', () => {
|
|
let TMP;
|
|
before(() => { TMP = mkdtempSync(pathJoin(tmpdir(), 'olp-keys-19k-')); });
|
|
after(() => {
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
delete process.env[ENV_OWNER_VAR];
|
|
__resetWriteLocks();
|
|
});
|
|
|
|
it('19k: valid plaintext returns key identity from filesystem', () => {
|
|
const { id, plaintext_token } = createKey({
|
|
name: 'valid-test',
|
|
owner_tier: 'guest',
|
|
providers_enabled: ['anthropic', 'openai'],
|
|
olpHome: TMP,
|
|
});
|
|
const identity = validateKey(plaintext_token, { olpHome: TMP });
|
|
assert.ok(identity);
|
|
assert.equal(identity.id, id);
|
|
assert.equal(identity.owner_tier, 'guest');
|
|
assert.deepEqual(identity.providers_enabled, ['anthropic', 'openai']);
|
|
assert.equal(identity.source, 'filesystem');
|
|
});
|
|
|
|
it('19l: wrong plaintext returns null', () => {
|
|
createKey({ name: 'wrong-test', olpHome: TMP });
|
|
const wrong = 'olp_' + 'a'.repeat(43);
|
|
assert.equal(validateKey(wrong, { olpHome: TMP }), null);
|
|
});
|
|
|
|
it('19m: missing plaintext with allowAnonymous:false → null', () => {
|
|
assert.equal(validateKey(null, { olpHome: TMP, allowAnonymous: false }), null);
|
|
assert.equal(validateKey('', { olpHome: TMP, allowAnonymous: false }), null);
|
|
assert.equal(validateKey(undefined, { olpHome: TMP, allowAnonymous: false }), null);
|
|
// Default allowAnonymous is false (§ 7.2)
|
|
assert.equal(validateKey(null, { olpHome: TMP }), null);
|
|
});
|
|
|
|
it('19m-extra: non-string truthy plaintext (number / object / array) → null (no throw)', () => {
|
|
// D44 fold-in P2 #2: prior version threw TypeError on these inputs.
|
|
assert.equal(validateKey(42, { olpHome: TMP }), null);
|
|
assert.equal(validateKey({}, { olpHome: TMP }), null);
|
|
assert.equal(validateKey([], { olpHome: TMP }), null);
|
|
assert.equal(validateKey({ token: 'olp_xxx' }, { olpHome: TMP }), null);
|
|
// Same guard for allowAnonymous: true (env path doesn't change behaviour)
|
|
assert.equal(validateKey(42, { olpHome: TMP, allowAnonymous: true }), null);
|
|
});
|
|
|
|
it('19n: missing plaintext with allowAnonymous:true → anonymous identity', () => {
|
|
const id = validateKey(null, { olpHome: TMP, allowAnonymous: true });
|
|
assert.ok(id);
|
|
assert.equal(id.id, ANONYMOUS_KEY_ID);
|
|
assert.equal(id.owner_tier, 'anonymous');
|
|
assert.equal(id.source, 'anonymous');
|
|
});
|
|
|
|
it('19o: revoked key returns null on validation (criterion #6 validation-side)', async () => {
|
|
const { id, plaintext_token } = createKey({ name: 'revoke-test', olpHome: TMP });
|
|
assert.ok(validateKey(plaintext_token, { olpHome: TMP }), 'valid before revoke');
|
|
await revokeKey({ id, olpHome: TMP });
|
|
assert.equal(validateKey(plaintext_token, { olpHome: TMP }), null, '401 path after revoke');
|
|
});
|
|
|
|
it('19p: OLP_OWNER_TOKEN env override returns __env_owner__ identity (§ 9.4)', () => {
|
|
const envToken = 'olp_' + 'e'.repeat(43);
|
|
process.env[ENV_OWNER_VAR] = envToken;
|
|
try {
|
|
const id = validateKey(envToken, { olpHome: TMP });
|
|
assert.ok(id);
|
|
assert.equal(id.id, ENV_OWNER_KEY_ID);
|
|
assert.equal(id.owner_tier, 'owner');
|
|
assert.equal(id.providers_enabled, '*');
|
|
assert.equal(id.source, 'env');
|
|
} finally {
|
|
delete process.env[ENV_OWNER_VAR];
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('19q-r — revokeKey (§ 6.1)', () => {
|
|
let TMP;
|
|
before(() => { TMP = mkdtempSync(pathJoin(tmpdir(), 'olp-keys-19q-')); });
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); __resetWriteLocks(); });
|
|
|
|
it('19q: revokeKey marks revoked_at and is idempotent', async () => {
|
|
const { id } = createKey({ name: 'rev-idem', olpHome: TMP });
|
|
assert.equal(await revokeKey({ id, olpHome: TMP }), true);
|
|
const m1 = readManifest(id, { olpHome: TMP });
|
|
assert.ok(m1.revoked_at !== null);
|
|
assert.equal(await revokeKey({ id, olpHome: TMP }), true);
|
|
const m2 = readManifest(id, { olpHome: TMP });
|
|
assert.equal(m1.revoked_at, m2.revoked_at, 'second revoke must not rewrite timestamp');
|
|
});
|
|
|
|
it('19r: revokeKey returns false for non-existent id', async () => {
|
|
assert.equal(await revokeKey({ id: 'no-such-key-zzz', olpHome: TMP }), false);
|
|
});
|
|
});
|
|
|
|
describe('19s-t — listKeys', () => {
|
|
let TMP;
|
|
before(() => { TMP = mkdtempSync(pathJoin(tmpdir(), 'olp-keys-19s-')); });
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); __resetWriteLocks(); });
|
|
|
|
it('19s: empty dir returns []', () => {
|
|
assert.deepEqual(listKeys({ olpHome: TMP }), []);
|
|
});
|
|
|
|
it('19t: lists all keys with token_hash redacted', () => {
|
|
createKey({ name: 'k1', owner_tier: 'owner', olpHome: TMP });
|
|
createKey({ name: 'k2', owner_tier: 'guest', olpHome: TMP });
|
|
const list = listKeys({ olpHome: TMP });
|
|
assert.equal(list.length, 2);
|
|
for (const m of list) {
|
|
assert.ok(m.token_hash === undefined, 'token_hash must be redacted from listKeys output');
|
|
assert.ok(['k1', 'k2'].includes(m.name));
|
|
assert.ok(['owner', 'guest'].includes(m.owner_tier));
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('19u-x — touchLastUsed read-modify-write (§ 6.3)', () => {
|
|
let TMP;
|
|
before(() => { TMP = mkdtempSync(pathJoin(tmpdir(), 'olp-keys-19u-')); });
|
|
after(() => {
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
__resetWriteLocks();
|
|
__setTouchInterleaveHook(null);
|
|
});
|
|
|
|
it('19u: updates last_used_at on non-revoked key (preserves all other fields)', async () => {
|
|
const { id, manifest: m_before } = createKey({
|
|
name: 'touch-test', owner_tier: 'owner', providers_enabled: ['mistral'], olpHome: TMP,
|
|
});
|
|
assert.equal(m_before.last_used_at, null);
|
|
await touchLastUsed(id, { olpHome: TMP });
|
|
const m_after = readManifest(id, { olpHome: TMP });
|
|
assert.ok(m_after.last_used_at !== null);
|
|
assert.equal(m_after.revoked_at, null);
|
|
assert.equal(m_after.owner_tier, 'owner');
|
|
assert.deepEqual(m_after.providers_enabled, ['mistral']);
|
|
assert.equal(m_after.token_hash, m_before.token_hash);
|
|
});
|
|
|
|
it('19v: NO-OPs on already-revoked key (preserves revoked_at, does not set last_used_at)', async () => {
|
|
const { id } = createKey({ name: 'touch-revoked', olpHome: TMP });
|
|
await revokeKey({ id, olpHome: TMP });
|
|
const revokedTs = readManifest(id, { olpHome: TMP }).revoked_at;
|
|
assert.ok(revokedTs !== null);
|
|
|
|
await touchLastUsed(id, { olpHome: TMP });
|
|
const m_after_touch = readManifest(id, { olpHome: TMP });
|
|
assert.equal(m_after_touch.revoked_at, revokedTs, 'revoked_at must be preserved (revoke dominates touch)');
|
|
assert.equal(m_after_touch.last_used_at, null, 'NO-OP must not write last_used_at on revoked key');
|
|
});
|
|
|
|
it('19w: NO-OPs silently on anonymous + env-owner identities (no manifest exists)', async () => {
|
|
await touchLastUsed(ANONYMOUS_KEY_ID, { olpHome: TMP });
|
|
await touchLastUsed(ENV_OWNER_KEY_ID, { olpHome: TMP });
|
|
});
|
|
|
|
it('19x: failure is best-effort (no throw on missing key)', async () => {
|
|
await touchLastUsed('phantom-key-xyz', { olpHome: TMP });
|
|
});
|
|
|
|
it('19x-extra: per-key write-lock Map is cleaned up after sequential calls (D44 fold-in P2 #1 regression)', async () => {
|
|
__resetWriteLocks();
|
|
assert.equal(__writeLockSize(), 0, 'lock map starts empty');
|
|
const { id } = createKey({ name: 'lock-cleanup-test', olpHome: TMP });
|
|
// Sequential touch calls
|
|
for (let i = 0; i < 5; i++) {
|
|
await touchLastUsed(id, { olpHome: TMP });
|
|
}
|
|
assert.equal(__writeLockSize(), 0,
|
|
'lock map MUST be empty after sequential awaited calls (prior bug: stored derived promise never matched cleanup identity check, leaving stale entry per unique key-id)');
|
|
});
|
|
|
|
it('19x-extra-2: per-key write-lock Map cleans up after concurrent contention resolves', async () => {
|
|
__resetWriteLocks();
|
|
const ids = [];
|
|
for (let i = 0; i < 3; i++) {
|
|
const { id } = createKey({ name: `concurrent-${i}`, olpHome: TMP });
|
|
ids.push(id);
|
|
}
|
|
// Race: 3 keys, each with 3 concurrent touches
|
|
const calls = [];
|
|
for (const id of ids) {
|
|
for (let j = 0; j < 3; j++) {
|
|
calls.push(touchLastUsed(id, { olpHome: TMP }));
|
|
}
|
|
}
|
|
await Promise.all(calls);
|
|
assert.equal(__writeLockSize(), 0,
|
|
'lock map MUST drain to 0 after all queued callers across all keys finish');
|
|
});
|
|
});
|
|
|
|
describe('19y-1 to 19y-4 — Acceptance criterion #7: concurrent revoke + touch (§ 6.3, § 6.4)', () => {
|
|
let TMP;
|
|
before(() => { TMP = mkdtempSync(pathJoin(tmpdir(), 'olp-keys-19y-')); });
|
|
after(() => {
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
__resetWriteLocks();
|
|
__setTouchInterleaveHook(null);
|
|
});
|
|
|
|
it('19y-1: revoke-then-touch — revoked_at preserved', async () => {
|
|
const { id } = createKey({ name: 'race-1', olpHome: TMP });
|
|
await revokeKey({ id, olpHome: TMP });
|
|
const revokedTs = readManifest(id, { olpHome: TMP }).revoked_at;
|
|
await touchLastUsed(id, { olpHome: TMP });
|
|
const m = readManifest(id, { olpHome: TMP });
|
|
assert.equal(m.revoked_at, revokedTs, 'revoked_at must survive subsequent touch');
|
|
assert.equal(m.last_used_at, null, 'touch must NO-OP on revoked key');
|
|
});
|
|
|
|
it('19y-2: touch-then-revoke — revoked_at present + last_used_at preserved from prior touch', async () => {
|
|
const { id } = createKey({ name: 'race-2', olpHome: TMP });
|
|
await touchLastUsed(id, { olpHome: TMP });
|
|
const lastUsedAtBeforeRevoke = readManifest(id, { olpHome: TMP }).last_used_at;
|
|
assert.ok(lastUsedAtBeforeRevoke !== null);
|
|
await revokeKey({ id, olpHome: TMP });
|
|
const m = readManifest(id, { olpHome: TMP });
|
|
assert.ok(m.revoked_at !== null, 'revoked_at must be set');
|
|
assert.equal(m.last_used_at, lastUsedAtBeforeRevoke, 'last_used_at from prior touch must persist through revoke');
|
|
});
|
|
|
|
it('19y-3: interleaved external revoke fires inside touch lock before read — revoked_at preserved', async () => {
|
|
// SCENARIO TESTED: external revoke lands after touch acquires its lock
|
|
// but before touch's read. touch's subsequent read sees the revoke and
|
|
// NO-OPs per § 6.3 step 2.
|
|
//
|
|
// SCENARIO NOT TESTED (and currently UNREACHABLE in implementation):
|
|
// external revoke between touch's read and touch's write. The current
|
|
// touchLastUsed has SYNCHRONOUS read→write (no await between
|
|
// readManifest and writeManifestAtomic), so no external write can
|
|
// interleave between them at the JS event-loop level. If a future
|
|
// refactor introduces an await between read and write, this guarantee
|
|
// breaks and a post-read hook + matching test would be required to
|
|
// catch the regression. ADR § 10 criterion #7 scenario 3 is satisfied
|
|
// by the synchronous-read-write property of the current implementation.
|
|
const { id } = createKey({ name: 'race-3', olpHome: TMP });
|
|
let hookFired = false;
|
|
__setTouchInterleaveHook(async () => {
|
|
// Simulates an external (out-of-process / CLI) revoke landing
|
|
// between touch's lock acquisition and touch's read. Bypasses
|
|
// _withKeyLock because external writers do not see in-process
|
|
// locks at Phase 2 (§ 6.4).
|
|
const fresh = readManifest(id, { olpHome: TMP });
|
|
fresh.revoked_at = new Date().toISOString();
|
|
writeManifestAtomic(id, fresh, { olpHome: TMP });
|
|
hookFired = true;
|
|
});
|
|
try {
|
|
await touchLastUsed(id, { olpHome: TMP });
|
|
} finally {
|
|
__setTouchInterleaveHook(null);
|
|
}
|
|
assert.ok(hookFired, 'hook must have fired');
|
|
const m = readManifest(id, { olpHome: TMP });
|
|
assert.ok(m.revoked_at !== null, 'revoked_at must NOT be cleared by an interleaved touch — § 6.3 contract');
|
|
assert.equal(m.last_used_at, null, 'touch must NO-OP after observing the interleaved revoke');
|
|
});
|
|
|
|
it('19y-4: stress — 30 iterations of concurrent revoke + touch never lose revoked_at', async () => {
|
|
for (let i = 0; i < 30; i++) {
|
|
const { id } = createKey({ name: `stress-${i}`, olpHome: TMP });
|
|
await Promise.all([
|
|
revokeKey({ id, olpHome: TMP }),
|
|
touchLastUsed(id, { olpHome: TMP }),
|
|
]);
|
|
const m = readManifest(id, { olpHome: TMP });
|
|
assert.ok(m.revoked_at !== null, `iter ${i}: revoked_at lost — race regression`);
|
|
}
|
|
});
|
|
});
|
|
|
|
});
|
|
|
|
// ── Suite 20: server.mjs auth integration (D45, ADR 0007 §§ 5/6.2/7/9.4) ──
|
|
//
|
|
// HTTP-level tests for the Phase 2 D45 server-side wire-up:
|
|
// - auth.allow_anonymous false-by-default 401
|
|
// - Authorization Bearer / x-api-key header acceptance
|
|
// - revoked-key 401 (acceptance criterion #6 — full coverage with D45)
|
|
// - OLP_OWNER_TOKEN env override (acceptance criterion #10 — full coverage)
|
|
// - providers_enabled 403 scope enforcement (acceptance criterion #11)
|
|
// - per-key cache namespace isolation (acceptance criterion #1)
|
|
// - audit ndjson written with correct fields (acceptance criterion #8)
|
|
// - touchLastUsed wire updates last_used_at after a request
|
|
// - /v1/models gates auth the same way as /v1/chat/completions
|
|
|
|
import { readFileSync as fsReadFileSync, existsSync as fsExistsSync } from 'node:fs';
|
|
import { appendAuditEvent, __resetAuditDropCount } from './lib/audit.mjs';
|
|
|
|
describe('Suite 20 — server.mjs auth integration (D45, ADR 0007)', () => {
|
|
// Each describe block gets its own tmp OLP_HOME so audit + key writes are
|
|
// isolated and verifiable. Restore the global test default in after().
|
|
const GLOBAL_OLP_HOME = process.env.OLP_HOME;
|
|
|
|
// Each Suite 20 server requires the anthropic auth-check to pass before the
|
|
// mock spawn runs. lib/providers/anthropic.mjs _spawnAndStream checks for an
|
|
// OAuth token BEFORE calling the (mock) spawn — without a token the AUTH_MISSING
|
|
// pre-check fires and the request 502s before the mock can return chunks. Same
|
|
// pattern as Suite 9 cache HTTP tests (line ~2154 "test-fake-oauth-token-for-
|
|
// cache-tests"). CI Node 24 has no OAuth env; local dev machines may; CI was
|
|
// the trigger that caught this.
|
|
let _suite20SavedOAuth;
|
|
function ensureSuite20FakeOAuth() {
|
|
_suite20SavedOAuth = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'suite20-fake-oauth-token';
|
|
}
|
|
function restoreSuite20OAuth() {
|
|
if (_suite20SavedOAuth !== undefined) process.env.CLAUDE_CODE_OAUTH_TOKEN = _suite20SavedOAuth;
|
|
else delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
|
|
function makeSuite20Server() {
|
|
__setProvidersEnabled({ anthropic: true });
|
|
__setSpawnImpl(makeMockSpawn(['suite20-mock-response']));
|
|
ensureSuite20FakeOAuth();
|
|
const server = createOlpServer();
|
|
return new Promise(resolve => {
|
|
server.listen(0, '127.0.0.1', () => resolve({ server, port: server.address().port }));
|
|
});
|
|
}
|
|
|
|
function teardownSuite20(server) {
|
|
return new Promise(resolve => {
|
|
__resetSpawnImpl();
|
|
__setProvidersEnabled({});
|
|
__clearCache();
|
|
restoreSuite20OAuth();
|
|
if (server) server.close(() => resolve());
|
|
else resolve();
|
|
});
|
|
}
|
|
|
|
// ── 20a-d: header parsing + happy paths ────────────────────────────────
|
|
|
|
describe('20a-d — header parsing + valid key paths', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-20ad-'));
|
|
process.env.OLP_HOME = TMP;
|
|
__setAuthConfig({ allow_anonymous: false });
|
|
({ server, port } = await makeSuite20Server());
|
|
});
|
|
after(async () => {
|
|
await teardownSuite20(server);
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('20a: allow_anonymous=false + no Authorization header → 401 auth_required', async () => {
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20a' }] },
|
|
});
|
|
assert.equal(r.status, 401);
|
|
const err = JSON.parse(r.body);
|
|
assert.equal(err.error.type, 'auth_required');
|
|
});
|
|
|
|
it('20b: valid Authorization: Bearer <token> → 200 (filesystem identity)', async () => {
|
|
const { plaintext_token } = createKey({ name: '20b-bearer', owner_tier: 'guest', providers_enabled: ['anthropic'], olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20b' }] },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
});
|
|
|
|
it('20c: x-api-key header alternative → 200 (filesystem identity)', async () => {
|
|
const { plaintext_token } = createKey({ name: '20c-xapikey', owner_tier: 'guest', providers_enabled: ['anthropic'], olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { 'x-api-key': plaintext_token },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20c' }] },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
});
|
|
|
|
it('20d: invalid token → 401 invalid_or_revoked_key', async () => {
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: 'Bearer olp_not-a-real-key-aaaaaaaaaaaaaaaaaaaaaaaaaaaa' },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20d' }] },
|
|
});
|
|
assert.equal(r.status, 401);
|
|
const err = JSON.parse(r.body);
|
|
assert.equal(err.error.type, 'invalid_or_revoked_key');
|
|
});
|
|
});
|
|
|
|
// ── 20e-g: revocation + env owner + allow_anonymous true ────────────────
|
|
|
|
describe('20e-g — revocation + env owner override + anonymous-mode dev', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-20eg-'));
|
|
process.env.OLP_HOME = TMP;
|
|
__setAuthConfig({ allow_anonymous: false });
|
|
({ server, port } = await makeSuite20Server());
|
|
});
|
|
after(async () => {
|
|
await teardownSuite20(server);
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true });
|
|
delete process.env.ENV_OWNER_VAR;
|
|
delete process.env[ENV_OWNER_VAR];
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('20e: revoked key → 401 invalid_or_revoked_key (criterion #6 full coverage)', async () => {
|
|
const { id, plaintext_token } = createKey({ name: '20e-rev', owner_tier: 'guest', providers_enabled: ['anthropic'], olpHome: TMP });
|
|
// First request — should succeed
|
|
const r1 = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20e-1' }] },
|
|
});
|
|
assert.equal(r1.status, 200);
|
|
// Revoke
|
|
await revokeKey({ id, olpHome: TMP });
|
|
// Second request — must 401
|
|
const r2 = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20e-2' }] },
|
|
});
|
|
assert.equal(r2.status, 401);
|
|
const err = JSON.parse(r2.body);
|
|
assert.equal(err.error.type, 'invalid_or_revoked_key');
|
|
});
|
|
|
|
it('20f: OLP_OWNER_TOKEN env override → 200 (env identity, criterion #10 full coverage)', async () => {
|
|
const envToken = 'olp_' + 'f'.repeat(43);
|
|
process.env[ENV_OWNER_VAR] = envToken;
|
|
try {
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${envToken}` },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20f' }] },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
} finally {
|
|
delete process.env[ENV_OWNER_VAR];
|
|
}
|
|
});
|
|
|
|
it('20g: allow_anonymous=true + no header → 200 (anonymous identity — dev escape hatch)', async () => {
|
|
__setAuthConfig({ allow_anonymous: true });
|
|
try {
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20g' }] },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
} finally {
|
|
__setAuthConfig({ allow_anonymous: false });
|
|
}
|
|
});
|
|
});
|
|
|
|
// ── 20h: providers_enabled scope enforcement (criterion #11) ─────────────
|
|
|
|
describe('20h — providers_enabled scope enforcement (criterion #11)', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-20h-'));
|
|
process.env.OLP_HOME = TMP;
|
|
__setAuthConfig({ allow_anonymous: false });
|
|
({ server, port } = await makeSuite20Server());
|
|
});
|
|
after(async () => {
|
|
await teardownSuite20(server);
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('20h: providers_enabled: ["mistral"] + anthropic-routed model → 403 key_no_provider_access', async () => {
|
|
const { plaintext_token } = createKey({ name: '20h-scoped', owner_tier: 'guest', providers_enabled: ['mistral'], olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20h' }] },
|
|
});
|
|
assert.equal(r.status, 403);
|
|
const err = JSON.parse(r.body);
|
|
assert.equal(err.error.type, 'key_no_provider_access');
|
|
});
|
|
|
|
it('20h-extra: providers_enabled: "*" + anthropic-routed model → 200 (sanity baseline)', async () => {
|
|
const { plaintext_token } = createKey({ name: '20h-wildcard', owner_tier: 'guest', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20h-star' }] },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
});
|
|
|
|
it('20h-extra-audit: D53 — key_no_provider_access 403 audit row has tried_providers = []', async () => {
|
|
// Per ADR 0007 § 8 tried_providers semantics clarification: the field
|
|
// captures providers the server actually dispatched. On 403 (filter
|
|
// rejected the whole chain) the server dispatched zero providers, so
|
|
// tried_providers MUST be []. The configured-but-blocked chain only
|
|
// appears in the human-readable error message, not in the audit.
|
|
// D45 P2 deferral fix — was previously stamping the original chain
|
|
// which distorted "which providers did key X actually call" queries.
|
|
const { id, plaintext_token } = createKey({ name: '20h-extra-audit', owner_tier: 'guest', providers_enabled: ['mistral'], olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20h-extra-audit' }] },
|
|
});
|
|
assert.equal(r.status, 403);
|
|
assert.equal(JSON.parse(r.body).error.type, 'key_no_provider_access');
|
|
// Wait for the res.on('finish') audit append
|
|
await new Promise(resolve => setTimeout(resolve, 25));
|
|
const auditPath = _pathJoinForSetup(TMP, 'logs', 'audit.ndjson');
|
|
const lines = fsReadFileSync(auditPath, 'utf-8').trim().split('\n').filter(Boolean);
|
|
const row = lines.map(l => JSON.parse(l)).find(r =>
|
|
r.path === '/v1/chat/completions' && r.status_code === 403 && r.key_id === id,
|
|
);
|
|
assert.ok(row, '403 audit row must be present');
|
|
assert.equal(row.error_code, 'key_no_provider_access');
|
|
assert.deepEqual(row.tried_providers, [],
|
|
'tried_providers MUST be [] on key_no_provider_access — D53 semantic fix per ADR 0007 § 8 clarification');
|
|
});
|
|
});
|
|
|
|
// ── 20i: per-key cache namespace isolation (criterion #1) ────────────────
|
|
|
|
describe('20i — per-key cache namespace isolation (criterion #1)', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-20i-'));
|
|
process.env.OLP_HOME = TMP;
|
|
__setAuthConfig({ allow_anonymous: false });
|
|
({ server, port } = await makeSuite20Server());
|
|
});
|
|
after(async () => {
|
|
await teardownSuite20(server);
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('20i: keys A + B sending identical payload do NOT share cache (per-keyId namespace from ADR 0005 D1)', async () => {
|
|
const { plaintext_token: tokenA } = createKey({ name: '20i-A', owner_tier: 'guest', providers_enabled: ['anthropic'], olpHome: TMP });
|
|
const { plaintext_token: tokenB } = createKey({ name: '20i-B', owner_tier: 'guest', providers_enabled: ['anthropic'], olpHome: TMP });
|
|
const sharedPayload = {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: '20i-cache-shared-prompt' }],
|
|
};
|
|
|
|
// Key A: first request → miss
|
|
const rA1 = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${tokenA}` },
|
|
body: sharedPayload,
|
|
});
|
|
assert.equal(rA1.status, 200);
|
|
assert.equal(rA1.headers['x-olp-cache'], 'miss');
|
|
|
|
// Key A: second identical request → hit
|
|
const rA2 = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${tokenA}` },
|
|
body: sharedPayload,
|
|
});
|
|
assert.equal(rA2.status, 200);
|
|
assert.equal(rA2.headers['x-olp-cache'], 'hit', 'Key A second request must hit cache within A namespace');
|
|
|
|
// Key B: identical payload but different key → MUST be miss (isolated namespace)
|
|
const rB1 = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${tokenB}` },
|
|
body: sharedPayload,
|
|
});
|
|
assert.equal(rB1.status, 200);
|
|
assert.equal(rB1.headers['x-olp-cache'], 'miss',
|
|
'Key B first request MUST be miss — per-key cache isolation (ADR 0005 D1 + ADR 0007 § 7)');
|
|
});
|
|
});
|
|
|
|
// ── 20j: audit ndjson written with § 8 schema fields ─────────────────────
|
|
|
|
describe('20j — audit ndjson written per § 8 schema', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-20j-'));
|
|
process.env.OLP_HOME = TMP;
|
|
__setAuthConfig({ allow_anonymous: false });
|
|
__resetAuditDropCount();
|
|
({ server, port } = await makeSuite20Server());
|
|
});
|
|
after(async () => {
|
|
await teardownSuite20(server);
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('20j: successful request appends an audit row with key_id, model, status_code, latency_ms (criterion #8)', async () => {
|
|
const { id, plaintext_token } = createKey({ name: '20j-aud', owner_tier: 'guest', providers_enabled: ['anthropic'], olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20j' }] },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
// Allow the res.on('finish') hook to flush the audit write
|
|
await new Promise(resolve => setTimeout(resolve, 25));
|
|
|
|
const auditPath = _pathJoinForSetup(TMP, 'logs', 'audit.ndjson');
|
|
assert.ok(fsExistsSync(auditPath), 'audit.ndjson must be created on first request');
|
|
const lines = fsReadFileSync(auditPath, 'utf-8').trim().split('\n').filter(Boolean);
|
|
assert.ok(lines.length >= 1, 'at least one audit line expected');
|
|
const lastRow = JSON.parse(lines[lines.length - 1]);
|
|
|
|
assert.equal(lastRow.key_id, id);
|
|
assert.equal(lastRow.owner_tier, 'guest');
|
|
assert.equal(lastRow.method, 'POST');
|
|
assert.equal(lastRow.path, '/v1/chat/completions');
|
|
assert.equal(lastRow.model, 'claude-sonnet-4-6');
|
|
assert.equal(lastRow.status_code, 200);
|
|
assert.ok(typeof lastRow.latency_ms === 'number' && lastRow.latency_ms >= 0);
|
|
assert.ok(typeof lastRow.ts === 'string');
|
|
assert.equal(lastRow.provider, 'anthropic');
|
|
|
|
// PII guard: no message / response content in the audit row
|
|
assert.ok(!('content' in lastRow), 'no message content in audit row (§ 8 no PII)');
|
|
assert.ok(JSON.stringify(lastRow).indexOf('20j') === -1,
|
|
'request payload text must NOT appear in audit row (§ 8 no PII)');
|
|
});
|
|
|
|
it('20j-stream: streaming-path audit row populates provider + cache_status (D45 fold-in P1 regression)', async () => {
|
|
const { plaintext_token } = createKey({ name: '20j-stream', owner_tier: 'guest', providers_enabled: ['anthropic'], olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${plaintext_token}`, Accept: 'text/event-stream' },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20j-stream' }], stream: true },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
await new Promise(resolve => setTimeout(resolve, 50));
|
|
|
|
const auditPath = _pathJoinForSetup(TMP, 'logs', 'audit.ndjson');
|
|
const lines = fsReadFileSync(auditPath, 'utf-8').trim().split('\n').filter(Boolean);
|
|
const streamingRows = lines.map(l => JSON.parse(l)).filter(row =>
|
|
row.path === '/v1/chat/completions' && row.status_code === 200 && row.key_id !== ANONYMOUS_KEY_ID,
|
|
);
|
|
assert.ok(streamingRows.length >= 1, 'at least one successful authed row expected');
|
|
const lastStreamingRow = streamingRows[streamingRows.length - 1];
|
|
// Prior to D45 fold-in P1: provider was null on real-streaming success path.
|
|
assert.equal(lastStreamingRow.provider, 'anthropic',
|
|
'streaming-path audit row MUST populate provider (D45 fold-in P1 regression)');
|
|
assert.equal(lastStreamingRow.cache_status, 'miss',
|
|
'streaming-path audit row MUST populate cache_status');
|
|
assert.deepEqual(lastStreamingRow.tried_providers, ['anthropic']);
|
|
});
|
|
|
|
it('20j-401: 401 unauth request still appends audit row with error_code', async () => {
|
|
const before = fsExistsSync(_pathJoinForSetup(TMP, 'logs', 'audit.ndjson'))
|
|
? fsReadFileSync(_pathJoinForSetup(TMP, 'logs', 'audit.ndjson'), 'utf-8').split('\n').filter(Boolean).length
|
|
: 0;
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20j-401' }] },
|
|
});
|
|
assert.equal(r.status, 401);
|
|
await new Promise(resolve => setTimeout(resolve, 25));
|
|
const after = fsReadFileSync(_pathJoinForSetup(TMP, 'logs', 'audit.ndjson'), 'utf-8').split('\n').filter(Boolean);
|
|
assert.ok(after.length > before, '401 must also append a row');
|
|
const lastRow = JSON.parse(after[after.length - 1]);
|
|
assert.equal(lastRow.status_code, 401);
|
|
assert.equal(lastRow.error_code, 'auth_required');
|
|
});
|
|
});
|
|
|
|
// ── 20k: touchLastUsed wire after successful request ────────────────────
|
|
|
|
describe('20k — touchLastUsed wire updates last_used_at post-request', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-20k-'));
|
|
process.env.OLP_HOME = TMP;
|
|
__setAuthConfig({ allow_anonymous: false });
|
|
({ server, port } = await makeSuite20Server());
|
|
});
|
|
after(async () => {
|
|
await teardownSuite20(server);
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('20k: filesystem key last_used_at populated after first successful request', async () => {
|
|
const { id, plaintext_token, manifest: m0 } = createKey({
|
|
name: '20k-touch', owner_tier: 'guest', providers_enabled: ['anthropic'], olpHome: TMP,
|
|
});
|
|
assert.equal(m0.last_used_at, null);
|
|
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '20k' }] },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
// Allow the async touchLastUsed (fired in res.on('finish')) to settle
|
|
await new Promise(resolve => setTimeout(resolve, 50));
|
|
|
|
const m1 = readManifest(id, { olpHome: TMP });
|
|
assert.ok(m1.last_used_at !== null, 'last_used_at must be populated after first request');
|
|
assert.equal(m1.revoked_at, null, 'revoked_at unchanged');
|
|
});
|
|
});
|
|
|
|
// ── 20l: /v1/models also enforces auth ───────────────────────────────────
|
|
|
|
describe('20m — /health with no auth + allow_anonymous=false → 401 (D46 consistent gating)', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-20m-'));
|
|
process.env.OLP_HOME = TMP;
|
|
__setAuthConfig({ allow_anonymous: false });
|
|
({ server, port } = await makeSuite20Server());
|
|
});
|
|
after(async () => {
|
|
await teardownSuite20(server);
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('20m: GET /health with no auth and allow_anonymous=false → 401', async () => {
|
|
const r = await fetch({ port, method: 'GET', path: '/health' });
|
|
assert.equal(r.status, 401);
|
|
});
|
|
});
|
|
|
|
describe('20l — /v1/models also enforces auth', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-20l-'));
|
|
process.env.OLP_HOME = TMP;
|
|
__setAuthConfig({ allow_anonymous: false });
|
|
({ server, port } = await makeSuite20Server());
|
|
});
|
|
after(async () => {
|
|
await teardownSuite20(server);
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('20l: /v1/models without auth → 401', async () => {
|
|
const r = await fetch({ port, method: 'GET', path: '/v1/models' });
|
|
assert.equal(r.status, 401);
|
|
});
|
|
|
|
it('20l-200: /v1/models with valid Bearer → 200 + data array', async () => {
|
|
const { plaintext_token } = createKey({ name: '20l-ok', owner_tier: 'guest', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/v1/models',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.object, 'list');
|
|
assert.ok(Array.isArray(body.data));
|
|
});
|
|
});
|
|
});
|
|
|
|
// ── Suite 21: D46 owner-vs-guest gating (ADR 0007 § 7.1 + § 7.2) ──────────
|
|
//
|
|
// HTTP-level tests for:
|
|
// - /health payload trimming (criterion #4): owner sees full per-provider
|
|
// statuses; guest + anonymous see trimmed { ok, version }
|
|
// - X-OLP-Fallback-Detail emission gating (criterion #5): policy
|
|
// 'owner_only' (default) emits only to owner; 'all' emits to everyone;
|
|
// 'none' suppresses entirely
|
|
|
|
describe('Suite 21 — D46 owner-vs-guest gating (ADR 0007 §§ 7.1, 7.2)', () => {
|
|
const GLOBAL_OLP_HOME = process.env.OLP_HOME;
|
|
|
|
let _suite21SavedOAuth;
|
|
function ensureSuite21FakeOAuth() {
|
|
_suite21SavedOAuth = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'suite21-fake-oauth-token';
|
|
}
|
|
function restoreSuite21OAuth() {
|
|
if (_suite21SavedOAuth !== undefined) process.env.CLAUDE_CODE_OAUTH_TOKEN = _suite21SavedOAuth;
|
|
else delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
|
|
function makeSuite21Server() {
|
|
__setProvidersEnabled({ anthropic: true });
|
|
__setSpawnImpl(makeMockSpawn(['suite21-response']));
|
|
ensureSuite21FakeOAuth();
|
|
const server = createOlpServer();
|
|
return new Promise(resolve => {
|
|
server.listen(0, '127.0.0.1', () => resolve({ server, port: server.address().port }));
|
|
});
|
|
}
|
|
|
|
function teardownSuite21(server) {
|
|
return new Promise(resolve => {
|
|
__resetSpawnImpl();
|
|
__setProvidersEnabled({});
|
|
__clearCache();
|
|
restoreSuite21OAuth();
|
|
if (server) server.close(() => resolve());
|
|
else resolve();
|
|
});
|
|
}
|
|
|
|
// ── 21a-d: /health payload trimming (criterion #4) ──────────────────────
|
|
|
|
describe('21a-d — /health payload trimming (criterion #4)', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-21ad-'));
|
|
process.env.OLP_HOME = TMP;
|
|
// Default-tight gating: /health is owner-only-endpoint
|
|
__setAuthConfig({
|
|
allow_anonymous: true,
|
|
owner_only_endpoints: ['/health'],
|
|
fallback_detail_header_policy: 'owner_only',
|
|
});
|
|
({ server, port } = await makeSuite21Server());
|
|
});
|
|
after(async () => {
|
|
await teardownSuite21(server);
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('21a: anonymous /health (allow_anonymous=true) → trimmed { ok, version }', async () => {
|
|
const r = await fetch({ port, method: 'GET', path: '/health' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.ok, true);
|
|
assert.equal(typeof body.version, 'string');
|
|
assert.ok(!('providers' in body), 'anonymous /health MUST NOT include providers field');
|
|
});
|
|
|
|
it('21b: guest /health → trimmed { ok, version }', async () => {
|
|
const { plaintext_token } = createKey({ name: '21b-guest', owner_tier: 'guest', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/health',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.ok, true);
|
|
assert.ok(!('providers' in body), 'guest /health MUST NOT include providers field');
|
|
});
|
|
|
|
it('21c: owner /health → full payload with providers', async () => {
|
|
const { plaintext_token } = createKey({ name: '21c-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/health',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.ok, true);
|
|
assert.ok('providers' in body, 'owner /health MUST include providers field');
|
|
assert.equal(typeof body.providers, 'object');
|
|
assert.equal(typeof body.providers.enabled, 'number');
|
|
assert.ok('status' in body.providers, 'owner /health providers.status must be present');
|
|
});
|
|
|
|
it('21c-extra: owner /health each provider status carries activeSpawns field (D56, v1.x #4 / ADR 0002 Amendment 6)', async () => {
|
|
// ADR 0002 Amendment 6 forward note: when surfaced on /health, the
|
|
// per-provider concurrency counter lives at providers.status.<name>.
|
|
// activeSpawns. D56 wires it. With no requests in flight, the value
|
|
// is 0; under saturation it equals hints.maxConcurrent.
|
|
const { plaintext_token } = createKey({ name: '21c-extra', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/health',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
// At least one provider must be enabled in the fixture
|
|
const statusEntries = Object.entries(body.providers.status);
|
|
assert.ok(statusEntries.length >= 1, 'fixture has at least one enabled provider');
|
|
for (const [name, status] of statusEntries) {
|
|
assert.ok('activeSpawns' in status,
|
|
`providers.status.${name}.activeSpawns MUST be present (ADR 0002 Amendment 6)`);
|
|
assert.equal(typeof status.activeSpawns, 'number');
|
|
assert.ok(status.activeSpawns >= 0, 'activeSpawns >= 0');
|
|
}
|
|
});
|
|
|
|
it('21d: owner_only_endpoints config opt-out — empty list → guest gets full payload', async () => {
|
|
__setAuthConfig({
|
|
allow_anonymous: true,
|
|
owner_only_endpoints: [],
|
|
fallback_detail_header_policy: 'owner_only',
|
|
});
|
|
try {
|
|
const { plaintext_token } = createKey({ name: '21d-guest-optout', owner_tier: 'guest', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/health',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.ok('providers' in body, 'with owner_only_endpoints: [], guest /health MUST get full payload');
|
|
} finally {
|
|
__setAuthConfig({
|
|
allow_anonymous: true,
|
|
owner_only_endpoints: ['/health'],
|
|
fallback_detail_header_policy: 'owner_only',
|
|
});
|
|
}
|
|
});
|
|
});
|
|
|
|
// ── 21e-h: X-OLP-Fallback-Detail emission gating (criterion #5) ─────────
|
|
|
|
describe('21e-h — X-OLP-Fallback-Detail emission gating (criterion #5)', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-21eh-'));
|
|
process.env.OLP_HOME = TMP;
|
|
// We will swap fallback_detail_header_policy per-case via __setAuthConfig.
|
|
__setAuthConfig({
|
|
allow_anonymous: false,
|
|
owner_only_endpoints: [],
|
|
fallback_detail_header_policy: 'owner_only',
|
|
});
|
|
// Wire a 2-hop chain with anthropic primary failing + openai secondary
|
|
// succeeding so X-OLP-Fallback-Detail has content to emit.
|
|
__setProvidersEnabled({ anthropic: true, openai: true });
|
|
__setFallbackConfig({
|
|
chains: {
|
|
'claude-sonnet-4-6': [
|
|
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
|
|
{ provider: 'openai', model: 'gpt-5.5' },
|
|
],
|
|
},
|
|
soft_triggers: {},
|
|
providersEnabled: { anthropic: true, openai: true },
|
|
});
|
|
// Mock: anthropic spawn fails (exit code 1); openai succeeds.
|
|
// Use a custom spawn impl that branches on the bin name to distinguish.
|
|
// Simpler: spy on both providers — for simplicity here we use the global
|
|
// anthropic spawn mock that returns exit 1, and a real-looking codex one.
|
|
// The easier path: use the mistral-codex paired spawn — but our existing
|
|
// makeMockSpawn doesn't distinguish providers. Pattern used in Suite 18k:
|
|
// an executeHopFn that itself fails primary. Here we re-use the global
|
|
// anthropic mock that fails so the chain advances and openai serves.
|
|
__setSpawnImpl(makeMockSpawn([], 1)); // anthropic spawn fails
|
|
// Codex provider has its own __setSpawnImpl pattern; mock it separately.
|
|
ensureSuite21FakeOAuth();
|
|
// Codex auth needs CODEX env or its own auth artifact. For tests:
|
|
process.env.OPENAI_CODEX_AUTH_PATH = '/dev/null'; // bypass to no-auth
|
|
const codexMod = await import('./lib/providers/codex.mjs');
|
|
codexMod.__setSpawnImpl?.(makeMockSpawn(['suite21-codex-served']));
|
|
|
|
const serverInst = createOlpServer();
|
|
await new Promise(resolve => {
|
|
serverInst.listen(0, '127.0.0.1', () => resolve());
|
|
});
|
|
server = serverInst;
|
|
port = serverInst.address().port;
|
|
});
|
|
after(async () => {
|
|
__resetSpawnImpl();
|
|
__setProvidersEnabled({});
|
|
__resetFallbackConfig();
|
|
__clearCache();
|
|
restoreSuite21OAuth();
|
|
delete process.env.OPENAI_CODEX_AUTH_PATH;
|
|
const codexMod = await import('./lib/providers/codex.mjs');
|
|
codexMod.__resetSpawnImpl?.();
|
|
await new Promise(resolve => server.close(() => resolve()));
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
// NOTE: 21e-h tests use a contract-level audit on the header alone.
|
|
// The header surfaces ONLY for non-empty fallbackDetail (D40) AND when
|
|
// the identity is permitted per fallback_detail_header_policy (D46).
|
|
// The 2-hop chain with primary-fail provides the non-empty trail.
|
|
|
|
it('21e: policy=owner_only + guest → X-OLP-Fallback-Detail header ABSENT', async () => {
|
|
__setAuthConfig({
|
|
allow_anonymous: false,
|
|
owner_only_endpoints: [],
|
|
fallback_detail_header_policy: 'owner_only',
|
|
});
|
|
const { plaintext_token } = createKey({ name: '21e-guest', owner_tier: 'guest', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '21e' }] },
|
|
});
|
|
// success or 502 both valid here — what matters is whether the
|
|
// detail header is present. We just need to invoke fallback.
|
|
assert.ok(r.headers['x-olp-fallback-detail'] === undefined,
|
|
'guest identity MUST NOT receive X-OLP-Fallback-Detail under owner_only policy');
|
|
});
|
|
|
|
it('21f: policy=owner_only + owner → X-OLP-Fallback-Detail header PRESENT', async () => {
|
|
__setAuthConfig({
|
|
allow_anonymous: false,
|
|
owner_only_endpoints: [],
|
|
fallback_detail_header_policy: 'owner_only',
|
|
});
|
|
const { plaintext_token } = createKey({ name: '21f-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '21f' }] },
|
|
});
|
|
assert.ok(r.headers['x-olp-fallback-detail'] !== undefined,
|
|
'owner identity MUST receive X-OLP-Fallback-Detail when fallback chain has failures');
|
|
// Validate header is JSON-parseable per D40 contract
|
|
const parsed = JSON.parse(r.headers['x-olp-fallback-detail']);
|
|
assert.ok(Array.isArray(parsed));
|
|
});
|
|
|
|
it('21g: policy=all + guest → X-OLP-Fallback-Detail header PRESENT (v0.1.1 opt-back-in)', async () => {
|
|
__setAuthConfig({
|
|
allow_anonymous: false,
|
|
owner_only_endpoints: [],
|
|
fallback_detail_header_policy: 'all',
|
|
});
|
|
const { plaintext_token } = createKey({ name: '21g-guest-all', owner_tier: 'guest', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '21g' }] },
|
|
});
|
|
assert.ok(r.headers['x-olp-fallback-detail'] !== undefined,
|
|
'policy=all MUST emit X-OLP-Fallback-Detail to guest identity (D46 opt-back-in to v0.1.1)');
|
|
});
|
|
|
|
it('21h: policy=none + owner → X-OLP-Fallback-Detail header ABSENT (full suppression)', async () => {
|
|
__setAuthConfig({
|
|
allow_anonymous: false,
|
|
owner_only_endpoints: [],
|
|
fallback_detail_header_policy: 'none',
|
|
});
|
|
const { plaintext_token } = createKey({ name: '21h-owner-none', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: '21h' }] },
|
|
});
|
|
assert.ok(r.headers['x-olp-fallback-detail'] === undefined,
|
|
'policy=none MUST suppress X-OLP-Fallback-Detail even for owner identity');
|
|
});
|
|
});
|
|
});
|
|
|
|
// ── Suite 22: D47 keygen CLI (bin/olp-keys.mjs, ADR 0007 § 9.1) ───────────
|
|
//
|
|
// Tests for the minimal keygen command surface that satisfies ADR § 10
|
|
// acceptance criterion #9 (bootstrap workflow must be reproducible without
|
|
// manual file editing).
|
|
|
|
import { runCli as runOlpKeysCli, parseArgv as parseOlpKeysArgv } from './bin/olp-keys.mjs';
|
|
|
|
describe('Suite 22 — D47 keygen CLI (bin/olp-keys.mjs, ADR 0007 § 9.1)', () => {
|
|
|
|
describe('22a — parseArgv unit tests', () => {
|
|
it('22a-1: empty argv → empty positional + empty flags', () => {
|
|
const r = parseOlpKeysArgv([]);
|
|
assert.deepEqual(r.positional, []);
|
|
assert.deepEqual(r.flags, {});
|
|
});
|
|
|
|
it('22a-2: --flag=value form', () => {
|
|
const r = parseOlpKeysArgv(['--name=keyA', '--tier=owner']);
|
|
assert.equal(r.flags.name, 'keyA');
|
|
assert.equal(r.flags.tier, 'owner');
|
|
});
|
|
|
|
it('22a-3: --flag value form (space-separated)', () => {
|
|
const r = parseOlpKeysArgv(['--name', 'keyB', '--tier', 'guest']);
|
|
assert.equal(r.flags.name, 'keyB');
|
|
assert.equal(r.flags.tier, 'guest');
|
|
});
|
|
|
|
it('22a-4: --flag (boolean, no value) when followed by another --flag', () => {
|
|
const r = parseOlpKeysArgv(['--owner', '--force']);
|
|
assert.equal(r.flags.owner, true);
|
|
assert.equal(r.flags.force, true);
|
|
});
|
|
|
|
it('22a-5: positional args + flags mixed', () => {
|
|
const r = parseOlpKeysArgv(['keygen', '--owner']);
|
|
assert.deepEqual(r.positional, ['keygen']);
|
|
assert.equal(r.flags.owner, true);
|
|
});
|
|
});
|
|
|
|
describe('22b — keygen subcommand', () => {
|
|
let TMP;
|
|
before(() => { TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-22b-')); });
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); __resetWriteLocks(); });
|
|
|
|
it('22b-1: keygen --owner creates owner key + prints plaintext token to stdout', async () => {
|
|
let out = '';
|
|
const code = await runOlpKeysCli(['keygen', '--owner', '--olp-home', TMP], {
|
|
out: s => { out += s; },
|
|
err: () => {},
|
|
});
|
|
assert.equal(code, 0);
|
|
assert.match(out, /token \(plaintext\):\s+olp_[A-Za-z0-9_-]{43}/);
|
|
assert.match(out, /owner_tier:\s+owner/);
|
|
// Verify manifest was actually written
|
|
const list = listKeys({ olpHome: TMP });
|
|
assert.equal(list.length, 1);
|
|
assert.equal(list[0].owner_tier, 'owner');
|
|
});
|
|
|
|
it('22b-2: keygen --name=test --providers=anthropic,openai → guest key with explicit providers', async () => {
|
|
let out = '';
|
|
const code = await runOlpKeysCli(
|
|
['keygen', '--name=test-guest', '--providers=anthropic,openai', '--olp-home', TMP],
|
|
{ out: s => { out += s; }, err: () => {} },
|
|
);
|
|
assert.equal(code, 0);
|
|
assert.match(out, /owner_tier:\s+guest/);
|
|
assert.match(out, /providers_enabled:\s+\[anthropic, openai\]/);
|
|
});
|
|
|
|
it('22b-3: keygen without --name AND without --owner → error exit 1', async () => {
|
|
let err = '';
|
|
const code = await runOlpKeysCli(['keygen', '--olp-home', TMP], {
|
|
out: () => {},
|
|
err: s => { err += s; },
|
|
});
|
|
assert.equal(code, 1);
|
|
assert.match(err, /--name is required/);
|
|
});
|
|
|
|
it('22b-4: keygen --tier=admin → error exit 1 (invalid tier)', async () => {
|
|
let err = '';
|
|
const code = await runOlpKeysCli(
|
|
['keygen', '--name=bad', '--tier=admin', '--olp-home', TMP],
|
|
{ out: () => {}, err: s => { err += s; } },
|
|
);
|
|
assert.equal(code, 1);
|
|
assert.match(err, /--tier must be "owner" or "guest"/);
|
|
});
|
|
|
|
it('22b-5: keygen --owner --force revokes existing owner + creates new', async () => {
|
|
// Use a fresh tmpdir for this isolation
|
|
const TMP2 = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-22b5-'));
|
|
try {
|
|
// Create initial owner
|
|
await runOlpKeysCli(['keygen', '--owner', '--olp-home', TMP2], { out: () => {}, err: () => {} });
|
|
const list1 = listKeys({ olpHome: TMP2 });
|
|
assert.equal(list1.length, 1);
|
|
const originalId = list1[0].id;
|
|
|
|
// --force: revoke existing + create new
|
|
let err = '';
|
|
const code = await runOlpKeysCli(['keygen', '--owner', '--force', '--olp-home', TMP2], {
|
|
out: () => {},
|
|
err: s => { err += s; },
|
|
});
|
|
assert.equal(code, 0);
|
|
assert.match(err, new RegExp(`Revoked existing owner key id=${originalId}`));
|
|
|
|
// Listing including revoked must show 2 entries; active-only must show 1
|
|
const listAll = listKeys({ olpHome: TMP2 });
|
|
assert.equal(listAll.length, 2);
|
|
const activeOwners = listAll.filter(k => k.owner_tier === 'owner' && k.revoked_at === null);
|
|
assert.equal(activeOwners.length, 1, 'exactly one active owner after --force');
|
|
const revokedOwners = listAll.filter(k => k.owner_tier === 'owner' && k.revoked_at !== null);
|
|
assert.equal(revokedOwners.length, 1, 'exactly one revoked owner after --force');
|
|
assert.equal(revokedOwners[0].id, originalId);
|
|
} finally {
|
|
rmSync(TMP2, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('22c — list subcommand', () => {
|
|
let TMP;
|
|
before(() => { TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-22c-')); });
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); __resetWriteLocks(); });
|
|
|
|
it('22c-1: list with no keys → "No keys."', async () => {
|
|
let out = '';
|
|
const code = await runOlpKeysCli(['list', '--olp-home', TMP], {
|
|
out: s => { out += s; }, err: () => {},
|
|
});
|
|
assert.equal(code, 0);
|
|
assert.match(out, /No keys\./);
|
|
});
|
|
|
|
it('22c-2: list after 2 keys → both visible with redacted token_hash', async () => {
|
|
await runOlpKeysCli(['keygen', '--name=alpha', '--olp-home', TMP], { out: () => {}, err: () => {} });
|
|
await runOlpKeysCli(['keygen', '--name=beta', '--owner', '--olp-home', TMP], { out: () => {}, err: () => {} });
|
|
let out = '';
|
|
const code = await runOlpKeysCli(['list', '--olp-home', TMP], { out: s => { out += s; }, err: () => {} });
|
|
assert.equal(code, 0);
|
|
assert.match(out, /name:\s+alpha/);
|
|
assert.match(out, /name:\s+beta/);
|
|
assert.match(out, /owner_tier:\s+owner/);
|
|
assert.match(out, /owner_tier:\s+guest/);
|
|
// token_hash MUST NOT appear in list output (lib/keys.mjs listKeys redacts it)
|
|
assert.ok(!out.includes('token_hash'), 'list output must not include token_hash field');
|
|
});
|
|
|
|
it('22c-3: list --owner-only filters to owner_tier=owner', async () => {
|
|
let out = '';
|
|
const code = await runOlpKeysCli(['list', '--owner-only', '--olp-home', TMP], {
|
|
out: s => { out += s; }, err: () => {},
|
|
});
|
|
assert.equal(code, 0);
|
|
assert.ok(!out.includes('name: alpha'), 'guest "alpha" must be filtered out with --owner-only');
|
|
assert.match(out, /name:\s+beta/, 'owner "beta" must remain');
|
|
});
|
|
});
|
|
|
|
describe('22d — revoke subcommand', () => {
|
|
let TMP, keyId;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-22d-'));
|
|
const r = createKey({ name: '22d-target', owner_tier: 'guest', olpHome: TMP });
|
|
keyId = r.id;
|
|
});
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); __resetWriteLocks(); });
|
|
|
|
it('22d-1: revoke --id=<valid> → exit 0, manifest revoked_at set', async () => {
|
|
let out = '';
|
|
const code = await runOlpKeysCli(['revoke', `--id=${keyId}`, '--olp-home', TMP], {
|
|
out: s => { out += s; }, err: () => {},
|
|
});
|
|
assert.equal(code, 0);
|
|
assert.match(out, new RegExp(`Revoked key id=${keyId}`));
|
|
const m = readManifest(keyId, { olpHome: TMP });
|
|
assert.ok(m.revoked_at !== null);
|
|
});
|
|
|
|
it('22d-2: revoke --id=<already-revoked> → exit 0 + "already revoked" message (idempotent)', async () => {
|
|
let out = '';
|
|
const code = await runOlpKeysCli(['revoke', `--id=${keyId}`, '--olp-home', TMP], {
|
|
out: s => { out += s; }, err: () => {},
|
|
});
|
|
assert.equal(code, 0);
|
|
assert.match(out, /already revoked/);
|
|
});
|
|
|
|
it('22d-3: revoke without --id → exit 1', async () => {
|
|
let err = '';
|
|
const code = await runOlpKeysCli(['revoke', '--olp-home', TMP], {
|
|
out: () => {}, err: s => { err += s; },
|
|
});
|
|
assert.equal(code, 1);
|
|
assert.match(err, /--id=<key-id> is required/);
|
|
});
|
|
|
|
it('22d-4: revoke --id=<nonexistent> → exit 2', async () => {
|
|
let err = '';
|
|
const code = await runOlpKeysCli(['revoke', '--id=nonexistent-key', '--olp-home', TMP], {
|
|
out: () => {}, err: s => { err += s; },
|
|
});
|
|
assert.equal(code, 2);
|
|
assert.match(err, /no key with id="nonexistent-key"/);
|
|
});
|
|
});
|
|
|
|
describe('22e — top-level CLI behaviour', () => {
|
|
it('22e-1: --help → exit 0 with usage text', async () => {
|
|
let out = '';
|
|
const code = await runOlpKeysCli(['--help'], { out: s => { out += s; }, err: () => {} });
|
|
assert.equal(code, 0);
|
|
assert.match(out, /OLP key management CLI/);
|
|
assert.match(out, /keygen/);
|
|
assert.match(out, /list/);
|
|
assert.match(out, /revoke/);
|
|
});
|
|
|
|
it('22e-2: no args → exit 1 with usage text', async () => {
|
|
let out = '';
|
|
const code = await runOlpKeysCli([], { out: s => { out += s; }, err: () => {} });
|
|
assert.equal(code, 1);
|
|
assert.match(out, /OLP key management CLI/);
|
|
});
|
|
|
|
it('22e-3: unknown subcommand → exit 1 with error', async () => {
|
|
let err = '';
|
|
const code = await runOlpKeysCli(['frobnicate'], { out: () => {}, err: s => { err += s; } });
|
|
assert.equal(code, 1);
|
|
assert.match(err, /unknown subcommand "frobnicate"/);
|
|
});
|
|
});
|
|
});
|
|
|
|
// ── Suite 23: D49 lib/audit-query.mjs (Phase 3 audit aggregate query layer) ──
|
|
//
|
|
// Unit tests for the in-memory audit query layer per ADR 0008 § 4.
|
|
// - discoverAuditFiles: filesystem scan + date-suffix recognition
|
|
// - readAuditWindow: window filtering + cross-file walk + malformed skip
|
|
// - aggregateRequests: count + median/p95 latency + by_provider + by_owner_tier + by_path
|
|
// - topFallbackChains: sort + tied-count tiebreak + tried_providers shape
|
|
// - spendTrendDaily: sparse-fill + UTC day buckets
|
|
// - cacheHitRateWindow: per-provider + bypass-not-in-denominator
|
|
// - PII guard: aggregate shapes never include message content
|
|
|
|
import {
|
|
discoverAuditFiles,
|
|
readAuditWindow,
|
|
aggregateRequests,
|
|
topFallbackChains,
|
|
spendTrendDaily,
|
|
cacheHitRateWindow,
|
|
} from './lib/audit-query.mjs';
|
|
import { mkdirSync, writeFileSync as fsWriteFileSyncForS23 } from 'node:fs';
|
|
|
|
describe('Suite 23 — D49 lib/audit-query.mjs (Phase 3 audit aggregate query layer, ADR 0008 § 4)', () => {
|
|
|
|
// Helper: write synthetic audit ndjson files to a tmpdir's logs/ subdir.
|
|
// entries is { 'live': [...events] | 'YYYY-MM-DD': [...events] }
|
|
function setupAuditFiles(tmp, entries) {
|
|
const logsDir = _pathJoinForSetup(tmp, 'logs');
|
|
mkdirSync(logsDir, { recursive: true, mode: 0o700 });
|
|
for (const [date, events] of Object.entries(entries)) {
|
|
const filename = date === 'live' ? 'audit.ndjson' : `audit-${date}.ndjson`;
|
|
const path = _pathJoinForSetup(logsDir, filename);
|
|
const lines = events.map(ev => JSON.stringify(ev)).join('\n') + '\n';
|
|
fsWriteFileSyncForS23(path, lines, { mode: 0o600 });
|
|
}
|
|
}
|
|
|
|
function makeEvent(overrides = {}) {
|
|
return {
|
|
ts: new Date().toISOString(),
|
|
key_id: 'k1',
|
|
owner_tier: 'owner',
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
provider: 'anthropic',
|
|
model: 'claude-sonnet-4-6',
|
|
status_code: 200,
|
|
latency_ms: 100,
|
|
cache_status: 'miss',
|
|
fallback_hops: 0,
|
|
tried_providers: ['anthropic'],
|
|
error_code: null,
|
|
ir_request_hash: 'abc123',
|
|
chain_id: 'c1',
|
|
...overrides,
|
|
};
|
|
}
|
|
|
|
describe('23a — discoverAuditFiles', () => {
|
|
let TMP;
|
|
before(() => { TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23a-')); });
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
|
|
|
it('23a-1: empty logs dir → empty Map', () => {
|
|
const r = discoverAuditFiles({ olpHome: TMP });
|
|
assert.equal(r.size, 0);
|
|
});
|
|
|
|
it('23a-2: only audit.ndjson → Map with "live" key', () => {
|
|
setupAuditFiles(TMP, { 'live': [makeEvent()] });
|
|
const r = discoverAuditFiles({ olpHome: TMP });
|
|
assert.equal(r.size, 1);
|
|
assert.ok(r.has('live'));
|
|
});
|
|
|
|
it('23a-3: rotated files + live → all recognized', () => {
|
|
setupAuditFiles(TMP, {
|
|
'live': [makeEvent()],
|
|
'2026-05-24': [makeEvent({ ts: '2026-05-24T12:00:00Z' })],
|
|
'2026-05-23': [makeEvent({ ts: '2026-05-23T12:00:00Z' })],
|
|
});
|
|
const r = discoverAuditFiles({ olpHome: TMP });
|
|
assert.equal(r.size, 3);
|
|
assert.ok(r.has('live'));
|
|
assert.ok(r.has('2026-05-24'));
|
|
assert.ok(r.has('2026-05-23'));
|
|
});
|
|
|
|
it('23a-4: non-audit files ignored', () => {
|
|
setupAuditFiles(TMP, { 'live': [makeEvent()] });
|
|
fsWriteFileSyncForS23(_pathJoinForSetup(TMP, 'logs', 'some-random.log'), 'noise\n', { mode: 0o600 });
|
|
fsWriteFileSyncForS23(_pathJoinForSetup(TMP, 'logs', 'audit-invalid-date.ndjson'), 'noise\n', { mode: 0o600 });
|
|
const r = discoverAuditFiles({ olpHome: TMP });
|
|
// live + 3 prior rotated from 23a-3 (state carries within describe block); ignore random files
|
|
assert.ok(r.size >= 1);
|
|
for (const key of r.keys()) {
|
|
assert.ok(key === 'live' || /^\d{4}-\d{2}-\d{2}$/.test(key));
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('23b — readAuditWindow', () => {
|
|
let TMP;
|
|
before(() => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23b-'));
|
|
setupAuditFiles(TMP, {
|
|
'2026-05-23': [
|
|
makeEvent({ ts: '2026-05-23T10:00:00Z', provider: 'anthropic' }),
|
|
makeEvent({ ts: '2026-05-23T20:00:00Z', provider: 'openai' }),
|
|
],
|
|
'2026-05-24': [
|
|
makeEvent({ ts: '2026-05-24T10:00:00Z', provider: 'mistral' }),
|
|
],
|
|
'live': [
|
|
makeEvent({ ts: '2026-05-25T10:00:00Z', provider: 'anthropic' }),
|
|
],
|
|
});
|
|
});
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
|
|
|
it('23b-1: window covering all → all 4 events', () => {
|
|
const start = Date.parse('2026-05-23T00:00:00Z');
|
|
const end = Date.parse('2026-05-26T00:00:00Z');
|
|
const events = [...readAuditWindow({ startMs: start, endMs: end, olpHome: TMP })];
|
|
assert.equal(events.length, 4);
|
|
});
|
|
|
|
it('23b-2: window covering only 2026-05-24 → 1 event (mistral)', () => {
|
|
const start = Date.parse('2026-05-24T00:00:00Z');
|
|
const end = Date.parse('2026-05-25T00:00:00Z');
|
|
const events = [...readAuditWindow({ startMs: start, endMs: end, olpHome: TMP })];
|
|
assert.equal(events.length, 1);
|
|
assert.equal(events[0].provider, 'mistral');
|
|
});
|
|
|
|
it('23b-3: half-open semantics — endMs exclusive', () => {
|
|
// Event at 2026-05-23T20:00:00Z. Window endMs = exactly that ts → exclude.
|
|
const start = Date.parse('2026-05-23T00:00:00Z');
|
|
const end = Date.parse('2026-05-23T20:00:00Z');
|
|
const events = [...readAuditWindow({ startMs: start, endMs: end, olpHome: TMP })];
|
|
assert.equal(events.length, 1); // only the 10:00 event
|
|
assert.equal(events[0].ts, '2026-05-23T10:00:00Z');
|
|
});
|
|
|
|
it('23b-4: empty window (end <= start) → empty', () => {
|
|
const events = [...readAuditWindow({ startMs: 1000, endMs: 1000, olpHome: TMP })];
|
|
assert.equal(events.length, 0);
|
|
const events2 = [...readAuditWindow({ startMs: 2000, endMs: 1000, olpHome: TMP })];
|
|
assert.equal(events2.length, 0);
|
|
});
|
|
|
|
it('23b-5: missing files → empty iteration (not an error)', () => {
|
|
const TMP2 = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23b5-'));
|
|
try {
|
|
const events = [...readAuditWindow({
|
|
startMs: Date.parse('2026-05-01T00:00:00Z'),
|
|
endMs: Date.parse('2026-05-02T00:00:00Z'),
|
|
olpHome: TMP2,
|
|
})];
|
|
assert.equal(events.length, 0);
|
|
} finally {
|
|
rmSync(TMP2, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('23b-6: malformed lines skipped + warn fired', () => {
|
|
const TMP3 = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23b6-'));
|
|
try {
|
|
mkdirSync(_pathJoinForSetup(TMP3, 'logs'), { recursive: true, mode: 0o700 });
|
|
const path = _pathJoinForSetup(TMP3, 'logs', 'audit.ndjson');
|
|
const lines = [
|
|
JSON.stringify(makeEvent({ ts: '2026-05-25T10:00:00Z' })),
|
|
'{ this is not valid json',
|
|
JSON.stringify(makeEvent({ ts: '2026-05-25T11:00:00Z' })),
|
|
'',
|
|
'definitely not json',
|
|
JSON.stringify(makeEvent({ ts: '2026-05-25T12:00:00Z' })),
|
|
].join('\n');
|
|
fsWriteFileSyncForS23(path, lines, { mode: 0o600 });
|
|
|
|
let warnFired = false;
|
|
const events = [...readAuditWindow({
|
|
startMs: Date.parse('2026-05-25T00:00:00Z'),
|
|
endMs: Date.parse('2026-05-26T00:00:00Z'),
|
|
olpHome: TMP3,
|
|
logEvent: (level, ev) => { if (level === 'warn' && ev === 'audit_query_skip_malformed') warnFired = true; },
|
|
})];
|
|
assert.equal(events.length, 3, 'malformed lines skipped, 3 valid events kept');
|
|
assert.ok(warnFired, 'warn must fire for malformed lines');
|
|
} finally {
|
|
rmSync(TMP3, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('23c — aggregateRequests', () => {
|
|
let TMP;
|
|
before(() => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23c-'));
|
|
const now = Date.now();
|
|
const t = (offsetMs) => new Date(now - offsetMs).toISOString();
|
|
setupAuditFiles(TMP, {
|
|
'live': [
|
|
makeEvent({ ts: t(60000), provider: 'anthropic', cache_status: 'hit', owner_tier: 'owner', status_code: 200, latency_ms: 50 }),
|
|
makeEvent({ ts: t(50000), provider: 'anthropic', cache_status: 'miss', owner_tier: 'owner', status_code: 200, latency_ms: 150 }),
|
|
makeEvent({ ts: t(40000), provider: 'openai', cache_status: 'miss', owner_tier: 'guest', status_code: 200, latency_ms: 200 }),
|
|
makeEvent({ ts: t(30000), provider: 'mistral', cache_status: 'bypass', owner_tier: 'owner', status_code: 401, latency_ms: 10 }),
|
|
makeEvent({ ts: t(20000), provider: 'anthropic', cache_status: 'miss', owner_tier: 'owner', status_code: 503, latency_ms: 5000, fallback_hops: 1, tried_providers: ['anthropic', 'openai'] }),
|
|
],
|
|
});
|
|
});
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
|
|
|
it('23c-1: aggregateRequests counts requests + status buckets + by_provider', () => {
|
|
const r = aggregateRequests({ windowMs: 86400 * 1000, olpHome: TMP });
|
|
assert.equal(r.request_count, 5);
|
|
assert.equal(r.status_2xx, 3); // 200x3
|
|
assert.equal(r.status_4xx, 1); // 401
|
|
assert.equal(r.status_5xx, 1); // 503
|
|
assert.equal(r.by_provider.anthropic.count, 3);
|
|
assert.equal(r.by_provider.anthropic.cache_hit, 1);
|
|
assert.equal(r.by_provider.anthropic.cache_miss, 2);
|
|
assert.equal(r.by_provider.anthropic.fallback_count, 1);
|
|
assert.equal(r.by_provider.openai.count, 1);
|
|
assert.equal(r.by_provider.mistral.cache_bypass, 1);
|
|
});
|
|
|
|
it('23c-2: by_owner_tier breakdown', () => {
|
|
const r = aggregateRequests({ windowMs: 86400 * 1000, olpHome: TMP });
|
|
assert.equal(r.by_owner_tier.owner, 4);
|
|
assert.equal(r.by_owner_tier.guest, 1);
|
|
assert.equal(r.by_owner_tier.anonymous, 0);
|
|
});
|
|
|
|
it('23c-3: median + p95 latency over [5, 50, 150, 200, 5000]', () => {
|
|
const r = aggregateRequests({ windowMs: 86400 * 1000, olpHome: TMP });
|
|
// sorted: 5, 10, 50, 150, 200; we have 5 events with latencies [50, 150, 200, 10, 5000]
|
|
// sorted: [10, 50, 150, 200, 5000]; median (index 2) = 150
|
|
assert.equal(r.median_latency_ms, 150);
|
|
// p95 index = floor(5 * 0.95) = 4 → 5000
|
|
assert.equal(r.p95_latency_ms, 5000);
|
|
});
|
|
|
|
it('23c-4: throws on invalid windowMs', () => {
|
|
assert.throws(() => aggregateRequests({ olpHome: TMP }), /windowMs.*required/);
|
|
assert.throws(() => aggregateRequests({ windowMs: 0, olpHome: TMP }), /windowMs.*required/);
|
|
assert.throws(() => aggregateRequests({ windowMs: -1, olpHome: TMP }), /windowMs.*required/);
|
|
});
|
|
});
|
|
|
|
describe('23d — topFallbackChains', () => {
|
|
let TMP;
|
|
before(() => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23d-'));
|
|
const now = Date.now();
|
|
const t = (offsetMs) => new Date(now - offsetMs).toISOString();
|
|
setupAuditFiles(TMP, {
|
|
'live': [
|
|
// 3x anthropic→openai
|
|
makeEvent({ ts: t(5000), fallback_hops: 1, tried_providers: ['anthropic', 'openai'] }),
|
|
makeEvent({ ts: t(4000), fallback_hops: 1, tried_providers: ['anthropic', 'openai'] }),
|
|
makeEvent({ ts: t(3000), fallback_hops: 1, tried_providers: ['anthropic', 'openai'] }),
|
|
// 2x anthropic→mistral
|
|
makeEvent({ ts: t(2500), fallback_hops: 1, tried_providers: ['anthropic', 'mistral'] }),
|
|
makeEvent({ ts: t(2000), fallback_hops: 1, tried_providers: ['anthropic', 'mistral'] }),
|
|
// 1x openai→mistral (single chain)
|
|
makeEvent({ ts: t(1000), fallback_hops: 1, tried_providers: ['openai', 'mistral'] }),
|
|
// events with fallback_hops:0 are excluded
|
|
makeEvent({ ts: t(500), fallback_hops: 0, tried_providers: ['anthropic'] }),
|
|
],
|
|
});
|
|
});
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
|
|
|
it('23d-1: sorted desc by count', () => {
|
|
const r = topFallbackChains({ windowMs: 86400 * 1000, olpHome: TMP });
|
|
assert.equal(r.length, 3);
|
|
assert.equal(r[0].count, 3);
|
|
assert.deepEqual(r[0].chain, ['anthropic', 'openai']);
|
|
assert.equal(r[1].count, 2);
|
|
assert.deepEqual(r[1].chain, ['anthropic', 'mistral']);
|
|
assert.equal(r[2].count, 1);
|
|
assert.deepEqual(r[2].chain, ['openai', 'mistral']);
|
|
});
|
|
|
|
it('23d-2: limit argument truncates result', () => {
|
|
const r = topFallbackChains({ windowMs: 86400 * 1000, limit: 2, olpHome: TMP });
|
|
assert.equal(r.length, 2);
|
|
});
|
|
|
|
it('23d-3: events with fallback_hops=0 excluded from chains', () => {
|
|
const r = topFallbackChains({ windowMs: 86400 * 1000, olpHome: TMP });
|
|
const allCounts = r.reduce((s, c) => s + c.count, 0);
|
|
assert.equal(allCounts, 6); // 3+2+1 — fallback_hops:0 event not counted
|
|
});
|
|
|
|
it('23d-4: chain entries carry first_seen + last_seen', () => {
|
|
const r = topFallbackChains({ windowMs: 86400 * 1000, olpHome: TMP });
|
|
for (const c of r) {
|
|
assert.ok(typeof c.first_seen === 'string');
|
|
assert.ok(typeof c.last_seen === 'string');
|
|
assert.ok(c.first_seen <= c.last_seen);
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('23e — spendTrendDaily', () => {
|
|
let TMP;
|
|
before(() => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23e-'));
|
|
// 3 events on day-2, 1 event on day-0; day-1 has none (sparse fill test)
|
|
const today = new Date().toISOString().slice(0, 10);
|
|
const day1 = new Date(Date.now() - 86400 * 1000).toISOString().slice(0, 10);
|
|
const day2 = new Date(Date.now() - 2 * 86400 * 1000).toISOString().slice(0, 10);
|
|
setupAuditFiles(TMP, {
|
|
[day2]: [
|
|
makeEvent({ ts: `${day2}T10:00:00Z`, provider: 'anthropic', latency_ms: 100 }),
|
|
makeEvent({ ts: `${day2}T11:00:00Z`, provider: 'openai', latency_ms: 200 }),
|
|
makeEvent({ ts: `${day2}T12:00:00Z`, provider: 'anthropic', latency_ms: 50 }),
|
|
],
|
|
'live': [
|
|
makeEvent({ ts: new Date().toISOString(), provider: 'mistral', latency_ms: 75 }),
|
|
],
|
|
});
|
|
});
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
|
|
|
it('23e-1: 3-day window returns 3 entries (sparse-fills empty days)', () => {
|
|
const r = spendTrendDaily({ days: 3, olpHome: TMP });
|
|
assert.equal(r.length, 3);
|
|
// Each entry has a date + request_count (>=0)
|
|
for (const e of r) {
|
|
assert.ok(typeof e.date === 'string');
|
|
assert.equal(e.date.length, 10);
|
|
assert.ok(typeof e.request_count === 'number');
|
|
assert.ok(typeof e.median_latency_ms === 'number');
|
|
assert.ok(typeof e.by_provider === 'object');
|
|
}
|
|
});
|
|
|
|
it('23e-2: day with data has correct breakdown', () => {
|
|
const r = spendTrendDaily({ days: 5, olpHome: TMP });
|
|
const day2Date = new Date(Date.now() - 2 * 86400 * 1000).toISOString().slice(0, 10);
|
|
const day2 = r.find(e => e.date === day2Date);
|
|
assert.ok(day2);
|
|
assert.equal(day2.request_count, 3);
|
|
assert.equal(day2.by_provider.anthropic, 2);
|
|
assert.equal(day2.by_provider.openai, 1);
|
|
// median of [50, 100, 200] = 100
|
|
assert.equal(day2.median_latency_ms, 100);
|
|
});
|
|
|
|
it('23e-3: empty day has request_count=0 and empty by_provider', () => {
|
|
const r = spendTrendDaily({ days: 3, olpHome: TMP });
|
|
const day1Date = new Date(Date.now() - 86400 * 1000).toISOString().slice(0, 10);
|
|
const day1 = r.find(e => e.date === day1Date);
|
|
assert.ok(day1);
|
|
assert.equal(day1.request_count, 0);
|
|
assert.deepEqual(day1.by_provider, {});
|
|
assert.equal(day1.median_latency_ms, 0);
|
|
});
|
|
});
|
|
|
|
describe('23f — cacheHitRateWindow', () => {
|
|
let TMP;
|
|
before(() => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23f-'));
|
|
const now = Date.now();
|
|
const t = (offsetMs) => new Date(now - offsetMs).toISOString();
|
|
setupAuditFiles(TMP, {
|
|
'live': [
|
|
makeEvent({ ts: t(5000), provider: 'anthropic', cache_status: 'hit' }),
|
|
makeEvent({ ts: t(4000), provider: 'anthropic', cache_status: 'hit' }),
|
|
makeEvent({ ts: t(3000), provider: 'anthropic', cache_status: 'miss' }),
|
|
makeEvent({ ts: t(2000), provider: 'openai', cache_status: 'miss' }),
|
|
makeEvent({ ts: t(1500), provider: 'openai', cache_status: 'bypass' }),
|
|
// events with cache_status null (e.g., 401 paths) excluded
|
|
makeEvent({ ts: t(1000), provider: null, cache_status: null }),
|
|
],
|
|
});
|
|
});
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
|
|
|
it('23f-1: hit_rate excludes bypass from denominator', () => {
|
|
const r = cacheHitRateWindow({ windowMs: 86400 * 1000, olpHome: TMP });
|
|
// overall: hit=2, miss=2, bypass=1 → hit_rate = 2/(2+2) = 0.5
|
|
assert.equal(r.hit, 2);
|
|
assert.equal(r.miss, 2);
|
|
assert.equal(r.bypass, 1);
|
|
assert.equal(r.hit_rate, 0.5);
|
|
});
|
|
|
|
it('23f-2: per-provider hit_rate', () => {
|
|
const r = cacheHitRateWindow({ windowMs: 86400 * 1000, olpHome: TMP });
|
|
// anthropic: 2 hit + 1 miss → rate 2/3
|
|
assert.equal(r.by_provider.anthropic.hit, 2);
|
|
assert.equal(r.by_provider.anthropic.miss, 1);
|
|
assert.equal(r.by_provider.anthropic.hit_rate, 2 / 3);
|
|
// openai: 1 miss + 1 bypass → rate 0/1 = 0
|
|
assert.equal(r.by_provider.openai.miss, 1);
|
|
assert.equal(r.by_provider.openai.bypass, 1);
|
|
assert.equal(r.by_provider.openai.hit_rate, 0);
|
|
});
|
|
|
|
it('23f-3: events with cache_status null excluded from total', () => {
|
|
const r = cacheHitRateWindow({ windowMs: 86400 * 1000, olpHome: TMP });
|
|
assert.equal(r.total, 5); // 6 events but 1 has cache_status:null
|
|
});
|
|
});
|
|
|
|
describe('23g — PII guard (ADR 0008 § 4.3)', () => {
|
|
let TMP;
|
|
before(() => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23g-'));
|
|
setupAuditFiles(TMP, {
|
|
'live': [makeEvent({ provider: 'anthropic', latency_ms: 100 })],
|
|
});
|
|
});
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
|
|
|
it('23g-1: aggregateRequests returns NO message-content fields', () => {
|
|
const r = aggregateRequests({ windowMs: 86400 * 1000, olpHome: TMP });
|
|
const json = JSON.stringify(r);
|
|
for (const piiKey of ['content', 'message', 'messages', 'prompt', 'response', 'body']) {
|
|
assert.ok(!json.includes(`"${piiKey}"`),
|
|
`aggregateRequests result MUST NOT contain field "${piiKey}" (PII guard)`);
|
|
}
|
|
});
|
|
|
|
it('23g-2: spendTrendDaily returns NO message-content fields', () => {
|
|
const r = spendTrendDaily({ days: 1, olpHome: TMP });
|
|
const json = JSON.stringify(r);
|
|
for (const piiKey of ['content', 'message', 'messages', 'prompt', 'response', 'body']) {
|
|
assert.ok(!json.includes(`"${piiKey}"`));
|
|
}
|
|
});
|
|
|
|
it('23g-3: topFallbackChains + cacheHitRateWindow contain no message fields', () => {
|
|
const j1 = JSON.stringify(topFallbackChains({ windowMs: 86400 * 1000, olpHome: TMP }));
|
|
const j2 = JSON.stringify(cacheHitRateWindow({ windowMs: 86400 * 1000, olpHome: TMP }));
|
|
for (const piiKey of ['content', 'message', 'messages', 'prompt', 'response', 'body']) {
|
|
assert.ok(!j1.includes(`"${piiKey}"`));
|
|
assert.ok(!j2.includes(`"${piiKey}"`));
|
|
}
|
|
});
|
|
});
|
|
});
|
|
|
|
// ── Suite 24: D50 management endpoints (Phase 3, ADR 0008 §§ 7-8) ─────────
|
|
//
|
|
// HTTP-level tests for the 4 owner_only_block endpoints:
|
|
// /dashboard, /v0/management/dashboard-data, /v0/management/quota, /cache/stats
|
|
// Each must 401 non-owner identities (including anonymous-when-allow_anonymous=true
|
|
// per ADR 0008 § 8) + serve owner identities with proper Content-Type and shape.
|
|
|
|
describe('Suite 24 — D50 management endpoints (Phase 3, ADR 0008 §§ 7-8)', () => {
|
|
const GLOBAL_OLP_HOME = process.env.OLP_HOME;
|
|
|
|
let _suite24SavedOAuth;
|
|
function ensureSuite24FakeOAuth() {
|
|
_suite24SavedOAuth = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'suite24-fake-oauth-token';
|
|
}
|
|
function restoreSuite24OAuth() {
|
|
if (_suite24SavedOAuth !== undefined) process.env.CLAUDE_CODE_OAUTH_TOKEN = _suite24SavedOAuth;
|
|
else delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
|
|
function makeSuite24Server() {
|
|
__setProvidersEnabled({ anthropic: true });
|
|
__setSpawnImpl(makeMockSpawn(['suite24-response']));
|
|
ensureSuite24FakeOAuth();
|
|
const server = createOlpServer();
|
|
return new Promise(resolve => {
|
|
server.listen(0, '127.0.0.1', () => resolve({ server, port: server.address().port }));
|
|
});
|
|
}
|
|
function teardownSuite24(server) {
|
|
return new Promise(resolve => {
|
|
__resetSpawnImpl();
|
|
__setProvidersEnabled({});
|
|
__clearCache();
|
|
restoreSuite24OAuth();
|
|
if (server) server.close(() => resolve());
|
|
else resolve();
|
|
});
|
|
}
|
|
|
|
describe('24a-d — /dashboard owner_only_block', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-24ad-'));
|
|
process.env.OLP_HOME = TMP;
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
({ server, port } = await makeSuite24Server());
|
|
});
|
|
after(async () => {
|
|
await teardownSuite24(server);
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('24a: owner → 200 text/html with "OLP Dashboard"', async () => {
|
|
const { plaintext_token } = createKey({ name: '24a-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/dashboard',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
assert.ok(r.headers['content-type'].startsWith('text/html'));
|
|
assert.match(r.body, /OLP Dashboard/i);
|
|
});
|
|
|
|
it('24b: guest → 401 owner_required', async () => {
|
|
const { plaintext_token } = createKey({ name: '24b-guest', owner_tier: 'guest', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/dashboard',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 401);
|
|
const err = JSON.parse(r.body);
|
|
assert.equal(err.error.type, 'owner_required');
|
|
});
|
|
|
|
it('24c: anonymous (allow_anonymous: true) → 401 owner_required (owner_only_block per ADR 0008 § 8)', async () => {
|
|
// allow_anonymous: true → no header → anonymous identity → STILL 401
|
|
// because management endpoints are owner_only_block (not trim).
|
|
const r = await fetch({ port, method: 'GET', path: '/dashboard' });
|
|
assert.equal(r.status, 401);
|
|
const err = JSON.parse(r.body);
|
|
assert.equal(err.error.type, 'owner_required');
|
|
});
|
|
|
|
it('24d: allow_anonymous: false + no header → 401 auth_required (middleware path)', async () => {
|
|
__setAuthConfig({ allow_anonymous: false, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
try {
|
|
const r = await fetch({ port, method: 'GET', path: '/dashboard' });
|
|
assert.equal(r.status, 401);
|
|
const err = JSON.parse(r.body);
|
|
assert.equal(err.error.type, 'auth_required');
|
|
} finally {
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('24e-g — /v0/management/dashboard-data + /v0/management/quota', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-24eg-'));
|
|
process.env.OLP_HOME = TMP;
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
({ server, port } = await makeSuite24Server());
|
|
});
|
|
after(async () => {
|
|
await teardownSuite24(server);
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('24e: owner GET /v0/management/dashboard-data → 200 JSON with required ADR 0008 § 7.2 fields', async () => {
|
|
const { plaintext_token } = createKey({ name: '24e-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/v0/management/dashboard-data',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.ok(typeof body.generated_at === 'string');
|
|
assert.ok(typeof body.window_24h === 'object');
|
|
assert.ok(typeof body.window_24h.request_count === 'number');
|
|
assert.ok(typeof body.cache_hit_24h === 'object');
|
|
assert.ok(Array.isArray(body.quota));
|
|
assert.ok(Array.isArray(body.spend_trend_30d));
|
|
assert.equal(body.spend_trend_30d.length, 30, 'spend_trend_30d must have 30 entries');
|
|
assert.ok(Array.isArray(body.top_fallback_chains_24h));
|
|
assert.ok(typeof body.cache_stats === 'object');
|
|
});
|
|
|
|
it('24f: guest GET /v0/management/dashboard-data → 401 owner_required', async () => {
|
|
const { plaintext_token } = createKey({ name: '24f-guest', owner_tier: 'guest', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/v0/management/dashboard-data',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 401);
|
|
assert.equal(JSON.parse(r.body).error.type, 'owner_required');
|
|
});
|
|
|
|
it('24g: owner GET /v0/management/quota → 200 JSON with quota array', async () => {
|
|
const { plaintext_token } = createKey({ name: '24g-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/v0/management/quota',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.ok(typeof body.generated_at === 'string');
|
|
assert.ok(Array.isArray(body.quota));
|
|
// Each quota entry has at least a provider key (and possibly more fields).
|
|
for (const q of body.quota) {
|
|
assert.ok(typeof q.provider === 'string');
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('24h — /cache/stats', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-24h-'));
|
|
process.env.OLP_HOME = TMP;
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
({ server, port } = await makeSuite24Server());
|
|
});
|
|
after(async () => {
|
|
await teardownSuite24(server);
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('24h: owner GET /cache/stats → 200 JSON with hits/misses/size/inflightCount', async () => {
|
|
const { plaintext_token } = createKey({ name: '24h-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/cache/stats',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.ok(typeof body.generated_at === 'string');
|
|
assert.ok(typeof body.hits === 'number');
|
|
assert.ok(typeof body.misses === 'number');
|
|
assert.ok(typeof body.size === 'number');
|
|
assert.ok(typeof body.inflightCount === 'number');
|
|
});
|
|
|
|
it('24h-401: guest GET /cache/stats → 401', async () => {
|
|
const { plaintext_token } = createKey({ name: '24h-guest', owner_tier: 'guest', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/cache/stats',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 401);
|
|
});
|
|
});
|
|
|
|
describe('24i-j — audit rows on management endpoints', () => {
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-24ij-'));
|
|
process.env.OLP_HOME = TMP;
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
({ server, port } = await makeSuite24Server());
|
|
});
|
|
after(async () => {
|
|
await teardownSuite24(server);
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('24i: successful /v0/management/dashboard-data appends audit row with path + status_code 200', async () => {
|
|
const { id, plaintext_token } = createKey({ name: '24i-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/v0/management/dashboard-data',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
await new Promise(resolve => setTimeout(resolve, 25));
|
|
|
|
const auditPath = _pathJoinForSetup(TMP, 'logs', 'audit.ndjson');
|
|
assert.ok(fsExistsSync(auditPath));
|
|
const lines = fsReadFileSync(auditPath, 'utf-8').trim().split('\n').filter(Boolean);
|
|
const mgmtRow = lines.map(l => JSON.parse(l)).find(row =>
|
|
row.path === '/v0/management/dashboard-data' && row.status_code === 200,
|
|
);
|
|
assert.ok(mgmtRow, 'management dashboard-data audit row must be present');
|
|
assert.equal(mgmtRow.key_id, id);
|
|
assert.equal(mgmtRow.owner_tier, 'owner');
|
|
assert.equal(mgmtRow.method, 'GET');
|
|
});
|
|
|
|
it('24j: 401 (guest blocked) /v0/management/dashboard-data appends audit row with error_code', async () => {
|
|
const { id, plaintext_token } = createKey({ name: '24j-guest', owner_tier: 'guest', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/v0/management/dashboard-data',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 401);
|
|
await new Promise(resolve => setTimeout(resolve, 25));
|
|
|
|
const auditPath = _pathJoinForSetup(TMP, 'logs', 'audit.ndjson');
|
|
const lines = fsReadFileSync(auditPath, 'utf-8').trim().split('\n').filter(Boolean);
|
|
const blockedRow = lines.map(l => JSON.parse(l)).find(row =>
|
|
row.path === '/v0/management/dashboard-data' && row.status_code === 401 && row.key_id === id,
|
|
);
|
|
assert.ok(blockedRow, 'management 401 audit row must be present');
|
|
assert.equal(blockedRow.error_code, 'owner_required');
|
|
assert.equal(blockedRow.owner_tier, 'guest');
|
|
});
|
|
});
|
|
});
|
|
|
|
// ── Suite 25: D51 dashboard.html UI smoke (Phase 3, ADR 0008 § 6) ─────────
|
|
//
|
|
// HTML-level smoke tests for the D51 dashboard. Per ADR 0008 § 10 criterion
|
|
// #12 the "no JS console errors in a real browser" sub-claim is manual or
|
|
// playwright; this suite covers the server-observable claims:
|
|
// - The owner-served HTML contains the 4 panel container IDs.
|
|
// - The polling script references the 30s interval + visibilitychange
|
|
// pause hook.
|
|
// - No external script/style src (no build step, no framework, no CDN).
|
|
|
|
describe('Suite 25 — D51 dashboard.html UI smoke (Phase 3, ADR 0008 § 6)', () => {
|
|
const GLOBAL_OLP_HOME = process.env.OLP_HOME;
|
|
|
|
let _suite25SavedOAuth;
|
|
function ensureSuite25FakeOAuth() {
|
|
_suite25SavedOAuth = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'suite25-fake-oauth-token';
|
|
}
|
|
function restoreSuite25OAuth() {
|
|
if (_suite25SavedOAuth !== undefined) process.env.CLAUDE_CODE_OAUTH_TOKEN = _suite25SavedOAuth;
|
|
else delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
|
|
let TMP, server, port;
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-25-'));
|
|
process.env.OLP_HOME = TMP;
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
__setProvidersEnabled({ anthropic: true });
|
|
__setSpawnImpl(makeMockSpawn(['suite25-response']));
|
|
ensureSuite25FakeOAuth();
|
|
server = createOlpServer();
|
|
await new Promise(resolve => {
|
|
server.listen(0, '127.0.0.1', () => resolve());
|
|
});
|
|
port = server.address().port;
|
|
});
|
|
after(async () => {
|
|
__resetSpawnImpl();
|
|
__setProvidersEnabled({});
|
|
__clearCache();
|
|
restoreSuite25OAuth();
|
|
if (server) await new Promise(resolve => server.close(() => resolve()));
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME;
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
it('25a: owner /dashboard response contains all 4 panel container IDs', async () => {
|
|
const { plaintext_token } = createKey({ name: '25a-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/dashboard',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
assert.match(r.body, /id="panel-quota"/);
|
|
assert.match(r.body, /id="panel-24h"/);
|
|
assert.match(r.body, /id="panel-trend"/);
|
|
assert.match(r.body, /id="panel-chains"/);
|
|
});
|
|
|
|
it('25b: dashboard JS references 30s polling interval + setInterval/clearInterval', async () => {
|
|
const { plaintext_token } = createKey({ name: '25b-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/dashboard',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
assert.match(r.body, /POLL_INTERVAL_MS\s*=\s*30000/, 'must declare 30s POLL_INTERVAL_MS constant');
|
|
assert.match(r.body, /setInterval\(/, 'must call setInterval');
|
|
assert.match(r.body, /clearInterval\(/, 'must call clearInterval (for visibilitychange pause)');
|
|
});
|
|
|
|
it('25c: dashboard JS wires visibilitychange listener for tab-hidden pause', async () => {
|
|
const { plaintext_token } = createKey({ name: '25c-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/dashboard',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
assert.match(r.body, /addEventListener\(\s*['"]visibilitychange['"]/, 'must register visibilitychange listener');
|
|
assert.match(r.body, /document\.visibilityState\s*===\s*['"]hidden['"]/, 'must check hidden state');
|
|
});
|
|
|
|
it('25d: dashboard has NO external script/style src (no build step, no framework, no CDN)', async () => {
|
|
const { plaintext_token } = createKey({ name: '25d-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/dashboard',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
// ADR 0008 Lane 1 = A: static HTML + vanilla JS + fetch; no framework.
|
|
// Strict: no <script src=> and no <link rel="stylesheet" href=>.
|
|
assert.ok(!/<script\s+[^>]*src\s*=/i.test(r.body),
|
|
'dashboard must NOT include any <script src="..."> (no external JS — ADR 0008 Lane 1 A)');
|
|
assert.ok(!/<link\s+[^>]*rel\s*=\s*['"]stylesheet['"][^>]*href\s*=/i.test(r.body),
|
|
'dashboard must NOT include any external <link rel="stylesheet" href="..."> (no external CSS)');
|
|
});
|
|
|
|
it('25e: dashboard fetches /v0/management/dashboard-data (the only backing endpoint hit by JS)', async () => {
|
|
const { plaintext_token } = createKey({ name: '25e-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/dashboard',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
assert.match(r.body, /fetch\(\s*['"]\/v0\/management\/dashboard-data['"]/,
|
|
'dashboard JS must fetch /v0/management/dashboard-data (consolidated D50 endpoint)');
|
|
});
|
|
|
|
it('25f: dashboard 401 surface for non-owner shows actionable error banner instructions', async () => {
|
|
// The dashboard HTML itself is served owner-only_block at handleDashboard
|
|
// (Suite 24 covers that). This test verifies the IN-PAGE error banner
|
|
// string mentions owner-token guidance so a future maintainer who lands
|
|
// on a 401 from the in-page JS knows what to do.
|
|
const { plaintext_token } = createKey({ name: '25f-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP });
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/dashboard',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
assert.match(r.body, /401.*owner-tier/i, 'dashboard error banner must mention owner-tier in 401 case');
|
|
});
|
|
});
|
|
|
|
// ── Suite 26: D52 audit rotation (Phase 3, ADR 0008 § 5) ──────────────────
|
|
//
|
|
// Tests for daily audit rotation:
|
|
// - First append after UTC date change triggers rename
|
|
// - Concurrent appends during rotation produce exactly one rename
|
|
// - External cron tool (bin/olp-audit-rotate.mjs) coexists with in-server
|
|
// - Rotated files readable by lib/audit-query.mjs after rotation
|
|
|
|
import {
|
|
appendAuditEvent as appendAuditEventS26,
|
|
_maybeRotateAudit,
|
|
getAuditRotateCount,
|
|
__resetAuditRotateState,
|
|
__setLastSeenUtcDateForTesting,
|
|
} from './lib/audit.mjs';
|
|
import { runCli as runAuditRotateCli } from './bin/olp-audit-rotate.mjs';
|
|
|
|
describe('Suite 26 — D52 audit rotation (Phase 3, ADR 0008 § 5)', () => {
|
|
|
|
function writeAuditFile(tmp, filename, events) {
|
|
mkdirSync(_pathJoinForSetup(tmp, 'logs'), { recursive: true, mode: 0o700 });
|
|
const path = _pathJoinForSetup(tmp, 'logs', filename);
|
|
const lines = events.map(ev => JSON.stringify(ev)).join('\n') + '\n';
|
|
fsWriteFileSyncForS23(path, lines, { mode: 0o600 });
|
|
}
|
|
|
|
function makeEvent(ts) {
|
|
return {
|
|
ts,
|
|
key_id: 'k1',
|
|
owner_tier: 'owner',
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
provider: 'anthropic',
|
|
model: 'claude-sonnet-4-6',
|
|
status_code: 200,
|
|
latency_ms: 100,
|
|
cache_status: 'miss',
|
|
fallback_hops: 0,
|
|
tried_providers: ['anthropic'],
|
|
};
|
|
}
|
|
|
|
describe('26a — _maybeRotateAudit', () => {
|
|
let TMP;
|
|
before(() => { TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-26a-')); });
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); __resetAuditRotateState(); });
|
|
|
|
it('26a-1: no live file → no rotation, returns { rotated: false }', () => {
|
|
__resetAuditRotateState();
|
|
const r = _maybeRotateAudit({ olpHome: TMP });
|
|
assert.equal(r.rotated, false);
|
|
});
|
|
|
|
it('26a-2: live file already on today → no rotation', () => {
|
|
__resetAuditRotateState();
|
|
const today = new Date().toISOString().slice(0, 10);
|
|
writeAuditFile(TMP, 'audit.ndjson', [makeEvent(`${today}T10:00:00Z`)]);
|
|
const r = _maybeRotateAudit({ olpHome: TMP });
|
|
assert.equal(r.rotated, false);
|
|
});
|
|
|
|
it('26a-3: live file holds yesterday\'s events → rotated to audit-YYYY-MM-DD.ndjson', () => {
|
|
__resetAuditRotateState();
|
|
const livePath = _pathJoinForSetup(TMP, 'logs', 'audit.ndjson');
|
|
rmSync(livePath, { force: true });
|
|
const yesterday = new Date(Date.now() - 86400 * 1000).toISOString().slice(0, 10);
|
|
writeAuditFile(TMP, 'audit.ndjson', [makeEvent(`${yesterday}T15:00:00Z`)]);
|
|
const r = _maybeRotateAudit({ olpHome: TMP });
|
|
assert.equal(r.rotated, true);
|
|
assert.equal(r.dateUsed, yesterday);
|
|
const rotatedPath = _pathJoinForSetup(TMP, 'logs', `audit-${yesterday}.ndjson`);
|
|
assert.ok(fsExistsSync(rotatedPath));
|
|
assert.ok(!fsExistsSync(livePath));
|
|
assert.equal(getAuditRotateCount(), 1);
|
|
});
|
|
|
|
it('26a-4: idempotent — second call after rotation is a no-op', () => {
|
|
const r = _maybeRotateAudit({ olpHome: TMP });
|
|
assert.equal(r.rotated, false);
|
|
assert.equal(getAuditRotateCount(), 1);
|
|
});
|
|
|
|
it('26a-5: rotation target already exists → skip + warn (cron-race safety)', () => {
|
|
__resetAuditRotateState();
|
|
const livePath = _pathJoinForSetup(TMP, 'logs', 'audit.ndjson');
|
|
rmSync(livePath, { force: true });
|
|
const yesterday = new Date(Date.now() - 86400 * 1000).toISOString().slice(0, 10);
|
|
writeAuditFile(TMP, `audit-${yesterday}.ndjson`, [makeEvent(`${yesterday}T08:00:00Z`)]);
|
|
writeAuditFile(TMP, 'audit.ndjson', [makeEvent(`${yesterday}T20:00:00Z`)]);
|
|
|
|
let warned = false;
|
|
const r = _maybeRotateAudit({
|
|
olpHome: TMP,
|
|
logEvent: (level, ev) => { if (level === 'warn' && ev === 'audit_rotate_target_exists') warned = true; },
|
|
});
|
|
assert.equal(r.rotated, false);
|
|
assert.ok(warned, 'must warn when rotation target already exists');
|
|
assert.ok(fsExistsSync(livePath));
|
|
});
|
|
});
|
|
|
|
describe('26b — appendAuditEvent triggers rotation on date change', () => {
|
|
let TMP;
|
|
before(() => { TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-26b-')); });
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); __resetAuditRotateState(); });
|
|
|
|
it('26b-1: appending past a UTC date change triggers sync rotation + append lands in new file', () => {
|
|
__resetAuditRotateState();
|
|
const livePath = _pathJoinForSetup(TMP, 'logs', 'audit.ndjson');
|
|
rmSync(livePath, { force: true });
|
|
const yesterday = new Date(Date.now() - 86400 * 1000).toISOString().slice(0, 10);
|
|
writeAuditFile(TMP, 'audit.ndjson', [makeEvent(`${yesterday}T10:00:00Z`)]);
|
|
__setLastSeenUtcDateForTesting(yesterday);
|
|
|
|
const todayTs = new Date().toISOString();
|
|
appendAuditEventS26(makeEvent(todayTs), { olpHome: TMP });
|
|
|
|
const rotatedPath = _pathJoinForSetup(TMP, 'logs', `audit-${yesterday}.ndjson`);
|
|
assert.ok(fsExistsSync(rotatedPath), `yesterday's rotated file must exist (${rotatedPath})`);
|
|
assert.ok(fsExistsSync(livePath), 'live file must exist (with today\'s new event)');
|
|
const liveContent = fsReadFileSync(livePath, 'utf-8');
|
|
assert.ok(liveContent.includes(todayTs), 'live file must contain today\'s appended event');
|
|
assert.ok(!liveContent.includes(`${yesterday}T10:00:00Z`), 'live file must NOT contain yesterday\'s content (it rotated)');
|
|
assert.equal(getAuditRotateCount(), 1);
|
|
});
|
|
});
|
|
|
|
describe('26c — concurrent appends during rotation', () => {
|
|
let TMP;
|
|
before(() => { TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-26c-')); });
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); __resetAuditRotateState(); });
|
|
|
|
it('26c-1: N sequential appendAuditEvent during date change → exactly 1 rotation + all events land in live', () => {
|
|
__resetAuditRotateState();
|
|
const livePath = _pathJoinForSetup(TMP, 'logs', 'audit.ndjson');
|
|
rmSync(livePath, { force: true });
|
|
const yesterday = new Date(Date.now() - 86400 * 1000).toISOString().slice(0, 10);
|
|
writeAuditFile(TMP, 'audit.ndjson', [makeEvent(`${yesterday}T10:00:00Z`)]);
|
|
__setLastSeenUtcDateForTesting(yesterday);
|
|
|
|
// 10 sync appends in quick succession. The first triggers rotation
|
|
// (date change detected); subsequent appends short-circuit the date
|
|
// check because _lastSeenUtcDate was updated inside the first call's
|
|
// _maybeRotateAudit. Result: exactly 1 rotation + all 10 events in
|
|
// the new live file.
|
|
const todayTs = new Date().toISOString();
|
|
for (let i = 0; i < 10; i++) {
|
|
appendAuditEventS26(makeEvent(todayTs), { olpHome: TMP });
|
|
}
|
|
|
|
assert.equal(getAuditRotateCount(), 1, 'exactly 1 rotation despite 10 sequential appends past date change');
|
|
const rotatedPath = _pathJoinForSetup(TMP, 'logs', `audit-${yesterday}.ndjson`);
|
|
assert.ok(fsExistsSync(rotatedPath));
|
|
const liveContent = fsReadFileSync(livePath, 'utf-8');
|
|
const liveLines = liveContent.split('\n').filter(Boolean);
|
|
assert.equal(liveLines.length, 10, 'live file has all 10 appended events');
|
|
});
|
|
});
|
|
|
|
describe('26d — bin/olp-audit-rotate.mjs CLI', () => {
|
|
let TMP;
|
|
before(() => { TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-26d-')); });
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); __resetAuditRotateState(); });
|
|
|
|
it('26d-1: --help → exit 0 with usage', async () => {
|
|
let out = '';
|
|
const code = await runAuditRotateCli(['--help'], { out: s => { out += s; }, err: () => {} });
|
|
assert.equal(code, 0);
|
|
assert.match(out, /OLP audit rotation cron tool/);
|
|
});
|
|
|
|
it('26d-2: no live file → exit 0 "no rotation needed"', async () => {
|
|
__resetAuditRotateState();
|
|
let out = '';
|
|
const code = await runAuditRotateCli([`--olp-home=${TMP}`], { out: s => { out += s; }, err: () => {} });
|
|
assert.equal(code, 0);
|
|
assert.match(out, /No rotation needed/);
|
|
});
|
|
|
|
it('26d-3: live file with yesterday events → CLI rotates + exit 0', async () => {
|
|
__resetAuditRotateState();
|
|
const yesterday = new Date(Date.now() - 86400 * 1000).toISOString().slice(0, 10);
|
|
writeAuditFile(TMP, 'audit.ndjson', [makeEvent(`${yesterday}T10:00:00Z`)]);
|
|
|
|
let out = '';
|
|
const code = await runAuditRotateCli([`--olp-home=${TMP}`], { out: s => { out += s; }, err: () => {} });
|
|
assert.equal(code, 0);
|
|
assert.match(out, /Rotated.*dateUsed=/);
|
|
const rotatedPath = _pathJoinForSetup(TMP, 'logs', `audit-${yesterday}.ndjson`);
|
|
assert.ok(fsExistsSync(rotatedPath));
|
|
});
|
|
|
|
it('26d-4: unknown flag → exit 1', async () => {
|
|
let err = '';
|
|
const code = await runAuditRotateCli([`--olp-home=${TMP}`, '--unknown'], {
|
|
out: () => {}, err: s => { err += s; },
|
|
});
|
|
assert.equal(code, 1);
|
|
assert.match(err, /unknown flag --unknown/);
|
|
});
|
|
});
|
|
|
|
describe('26e — rotated files queryable via lib/audit-query.mjs', () => {
|
|
let TMP;
|
|
before(() => { TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-26e-')); });
|
|
after(() => { rmSync(TMP, { recursive: true, force: true }); __resetAuditRotateState(); });
|
|
|
|
it('26e-1: after rotation, discoverAuditFiles + readAuditWindow see both live + rotated', () => {
|
|
__resetAuditRotateState();
|
|
const yesterday = new Date(Date.now() - 86400 * 1000).toISOString().slice(0, 10);
|
|
const today = new Date().toISOString().slice(0, 10);
|
|
writeAuditFile(TMP, 'audit.ndjson', [
|
|
makeEvent(`${yesterday}T10:00:00Z`),
|
|
makeEvent(`${yesterday}T11:00:00Z`),
|
|
]);
|
|
__setLastSeenUtcDateForTesting(yesterday);
|
|
|
|
// Sync rotation: completes before append returns
|
|
appendAuditEventS26(makeEvent(`${today}T05:00:00Z`), { olpHome: TMP });
|
|
|
|
const files = discoverAuditFiles({ olpHome: TMP });
|
|
assert.ok(files.has('live'), 'live file present');
|
|
assert.ok(files.has(yesterday), `rotated file for ${yesterday} present`);
|
|
|
|
const startMs = Date.parse(`${yesterday}T00:00:00Z`);
|
|
const endMs = Date.parse(`${today}T23:59:59Z`);
|
|
const events = [...readAuditWindow({ startMs, endMs, olpHome: TMP })];
|
|
assert.equal(events.length, 3, 'cross-file read yields yesterday(2) + today(1) = 3 events');
|
|
});
|
|
});
|
|
});
|
|
|
|
// ── Suite 27: D57 streaming singleflight (cache layer) ─────────────────────
|
|
//
|
|
// Authority: ADR 0005 Amendment 8 §§1-11, 14 (issue #16, D57).
|
|
//
|
|
// Cache-layer unit tests for the streaming-singleflight tee fan-out. Each
|
|
// test constructs a fake sourceFactory that returns an async generator
|
|
// producing a fixed chunk sequence; the cache store coordinates dedup +
|
|
// late-joiner replay + per-client backpressure. No real provider CLIs are
|
|
// spawned; no HTTP requests issued. D58 will wire this into server.mjs in a
|
|
// separate PR (Iron Rule 11).
|
|
//
|
|
// Test count: 12 (one per Amendment 8 §§1-11 + §14 fixture + composite-key
|
|
// isolation). Aim for +12 tests minimum per D57 prompt.
|
|
|
|
describe('Suite 27 — D57 streaming singleflight (cache layer)', () => {
|
|
// Helper: build a deterministic async source. Each yielded chunk waits a
|
|
// microtask so the tee/queue dynamics are observable across concurrent
|
|
// attached clients.
|
|
function makeChunkSequence(chunks, opts = {}) {
|
|
let returnedCount = 0;
|
|
const gen = (async function* fakeStream() {
|
|
try {
|
|
for (const c of chunks) {
|
|
if (opts.signal && opts.signal.aborted) return;
|
|
yield c;
|
|
await new Promise(r => setImmediate(r));
|
|
}
|
|
} finally {
|
|
returnedCount++;
|
|
if (opts.onReturn) opts.onReturn(returnedCount);
|
|
}
|
|
})();
|
|
return gen;
|
|
}
|
|
|
|
// D57 — ADR 0005 Amendment 8 §1: cache-hit + single client + source-mode
|
|
it('27a — single client streaming: behaviour identical to today (source role, full sequence)', async () => {
|
|
const store = new CacheStore({ _warnFn: () => {} });
|
|
let spawns = 0;
|
|
const factory = () => {
|
|
spawns++;
|
|
return makeChunkSequence(['a', 'b', 'c']);
|
|
};
|
|
const r = await store.getOrComputeStreaming('k1', 'ck-27a', factory);
|
|
assert.equal(r.isFirst, true);
|
|
assert.equal(r.role, 'source');
|
|
const out = [];
|
|
for await (const c of r.stream) out.push(c);
|
|
assert.deepEqual(out, ['a', 'b', 'c']);
|
|
assert.equal(spawns, 1);
|
|
// After completion, the cache should have an entry for replay.
|
|
const cached = await store.get('k1', 'ck-27a');
|
|
assert.ok(cached, 'cache populated post-completion');
|
|
assert.deepEqual(cached.value, ['a', 'b', 'c']);
|
|
});
|
|
|
|
// D57 — ADR 0005 Amendment 8 §1, §4: 2 concurrent identical streams
|
|
it('27b — 2 concurrent identical streams: only 1 sourceFactory call; identical chunks in order', async () => {
|
|
const store = new CacheStore({ _warnFn: () => {} });
|
|
let spawns = 0;
|
|
const factory = () => {
|
|
spawns++;
|
|
return makeChunkSequence(['x', 'y', 'z']);
|
|
};
|
|
const p1 = store.getOrComputeStreaming('k', 'ck-27b', factory, { clientId: 'A' });
|
|
const p2 = store.getOrComputeStreaming('k', 'ck-27b', factory, { clientId: 'B' });
|
|
const [r1, r2] = await Promise.all([p1, p2]);
|
|
assert.equal(r1.isFirst, true);
|
|
assert.equal(r1.role, 'source');
|
|
assert.equal(r2.isFirst, false);
|
|
assert.equal(r2.role, 'attached');
|
|
const out1 = [];
|
|
const out2 = [];
|
|
const c1 = (async () => { for await (const c of r1.stream) out1.push(c); })();
|
|
const c2 = (async () => { for await (const c of r2.stream) out2.push(c); })();
|
|
await Promise.all([c1, c2]);
|
|
assert.equal(spawns, 1, 'sourceFactory invoked exactly once');
|
|
assert.deepEqual(out1, ['x', 'y', 'z']);
|
|
assert.deepEqual(out2, ['x', 'y', 'z']);
|
|
});
|
|
|
|
// D57 — ADR 0005 Amendment 8 §5: mid-stream join (replay burst + live tail)
|
|
// + post-completion join (cache_hit)
|
|
it('27c — 3 concurrent, mid-stream join: A=source, B=attached (burst + tail), C=cache_hit', async () => {
|
|
const store = new CacheStore({ _warnFn: () => {} });
|
|
let spawns = 0;
|
|
// Six chunks. A iterates fast; B attaches after A has consumed ~3.
|
|
const factory = () => {
|
|
spawns++;
|
|
return makeChunkSequence(['1', '2', '3', '4', '5', '6']);
|
|
};
|
|
const r1 = await store.getOrComputeStreaming('k', 'ck-27c', factory, { clientId: 'A' });
|
|
assert.equal(r1.role, 'source');
|
|
|
|
// A iterates the first 3 chunks before B attaches.
|
|
const itA = r1.stream;
|
|
const outA = [];
|
|
const n1 = await itA.next(); outA.push(n1.value);
|
|
const n2 = await itA.next(); outA.push(n2.value);
|
|
const n3 = await itA.next(); outA.push(n3.value);
|
|
|
|
// Now B attaches mid-stream. Its replay drain picks up everything in
|
|
// accumulatedChunks at attach-time + the live tail thereafter.
|
|
const r2 = await store.getOrComputeStreaming('k', 'ck-27c', factory, { clientId: 'B' });
|
|
assert.equal(r2.role, 'attached');
|
|
|
|
// A finishes consuming.
|
|
const outA2 = [];
|
|
for await (const c of itA) outA2.push(c);
|
|
const outB = [];
|
|
for await (const c of r2.stream) outB.push(c);
|
|
|
|
assert.deepEqual([...outA, ...outA2], ['1', '2', '3', '4', '5', '6']);
|
|
// B receives the full sequence too (burst replays earlier chunks + live tail).
|
|
assert.deepEqual(outB, ['1', '2', '3', '4', '5', '6']);
|
|
assert.equal(spawns, 1, 'still only 1 spawn');
|
|
|
|
// C attaches AFTER source completion → cache_hit (not inflight, not respawn).
|
|
const r3 = await store.getOrComputeStreaming('k', 'ck-27c', factory, { clientId: 'C' });
|
|
assert.equal(r3.role, 'cache_hit');
|
|
assert.equal(r3.isFirst, false);
|
|
const outC = [];
|
|
for await (const c of r3.stream) outC.push(c);
|
|
assert.deepEqual(outC, ['1', '2', '3', '4', '5', '6']);
|
|
assert.equal(spawns, 1, 'no additional spawns');
|
|
});
|
|
|
|
// D57 — ADR 0005 Amendment 8 §9: first client early-return, others continue
|
|
it('27d — first client iterator early-return mid-stream: others continue; source NOT aborted; cache written', async () => {
|
|
const warnings = [];
|
|
const store = new CacheStore({ _warnFn: (msg, meta) => warnings.push({ msg, meta }) });
|
|
let sourceReturned = false;
|
|
const factory = () => (async function* () {
|
|
try {
|
|
for (let i = 0; i < 6; i++) {
|
|
yield `chunk-${i}`;
|
|
await new Promise(r => setImmediate(r));
|
|
}
|
|
} finally { sourceReturned = true; }
|
|
})();
|
|
const r1 = await store.getOrComputeStreaming('k', 'ck-27d', factory, { clientId: 'A' });
|
|
const r2 = await store.getOrComputeStreaming('k', 'ck-27d', factory, { clientId: 'B' });
|
|
// A early-returns after 2 chunks; B keeps consuming.
|
|
const itA = r1.stream;
|
|
const outA = [];
|
|
outA.push((await itA.next()).value);
|
|
outA.push((await itA.next()).value);
|
|
await itA.return(); // simulates HTTP client close
|
|
const outB = [];
|
|
for await (const c of r2.stream) outB.push(c);
|
|
// Source should have completed normally (not aborted) because B was still attached.
|
|
assert.equal(sourceReturned, true);
|
|
assert.deepEqual(outB, ['chunk-0', 'chunk-1', 'chunk-2', 'chunk-3', 'chunk-4', 'chunk-5']);
|
|
// Cache should be populated post-completion (B was a live client to the end).
|
|
const cached = await store.get('k', 'ck-27d');
|
|
assert.ok(cached, 'cache populated despite A\'s mid-stream return');
|
|
// No abort warning emitted.
|
|
assert.equal(warnings.filter(w => w.msg === 'streaming_inflight_abort').length, 0);
|
|
});
|
|
|
|
// D57 — ADR 0005 Amendment 8 §9: all clients disconnect → source aborted, no cache
|
|
it('27e — all clients disconnect mid-stream: source aborted via AbortController; no cache write; next call respawns', async () => {
|
|
const warnings = [];
|
|
const store = new CacheStore({ _warnFn: (msg, meta) => warnings.push({ msg, meta }) });
|
|
let spawns = 0;
|
|
let sourceReturned = false;
|
|
const factory = () => {
|
|
spawns++;
|
|
return (async function* () {
|
|
try {
|
|
for (let i = 0; i < 20; i++) {
|
|
yield `c-${i}`;
|
|
await new Promise(r => setImmediate(r));
|
|
}
|
|
} finally { sourceReturned = true; }
|
|
})();
|
|
};
|
|
const r1 = await store.getOrComputeStreaming('k', 'ck-27e', factory, { clientId: 'A' });
|
|
const itA = r1.stream;
|
|
await itA.next();
|
|
await itA.next();
|
|
await itA.return();
|
|
// Wait microtasks for the tee task to observe attachedClients.size === 0 and abort.
|
|
await new Promise(r => setTimeout(r, 30));
|
|
assert.equal(sourceReturned, true);
|
|
assert.equal(warnings.filter(w => w.msg === 'streaming_inflight_abort').length, 1);
|
|
// No cache entry — subsequent call respawns (no inflight, no cache hit).
|
|
const cached = await store.get('k', 'ck-27e');
|
|
assert.equal(cached, null, 'no cache write on abort');
|
|
const r2 = await store.getOrComputeStreaming('k', 'ck-27e', factory, { clientId: 'B' });
|
|
assert.equal(r2.role, 'source', 'subsequent call gets fresh source spawn');
|
|
assert.equal(spawns, 2);
|
|
// Drain r2 so it doesn't dangle.
|
|
for await (const _c of r2.stream) { /* drain */ }
|
|
});
|
|
|
|
// D57 — ADR 0005 Amendment 8 §4: source throws mid-stream
|
|
it('27f — source throws mid-stream: all attached clients receive the error; no cache write; entry removed', async () => {
|
|
const warnings = [];
|
|
const store = new CacheStore({ _warnFn: (msg, meta) => warnings.push({ msg, meta }) });
|
|
const err = new Error('synthetic source failure');
|
|
const factory = () => (async function* () {
|
|
yield 'a';
|
|
await new Promise(r => setImmediate(r));
|
|
yield 'b';
|
|
await new Promise(r => setImmediate(r));
|
|
throw err;
|
|
})();
|
|
const r1 = await store.getOrComputeStreaming('k', 'ck-27f', factory, { clientId: 'A' });
|
|
const r2 = await store.getOrComputeStreaming('k', 'ck-27f', factory, { clientId: 'B' });
|
|
// Both clients should reject when the source throws.
|
|
let e1, e2;
|
|
const c1 = (async () => { try { for await (const _c of r1.stream) {} } catch (e) { e1 = e; } })();
|
|
const c2 = (async () => { try { for await (const _c of r2.stream) {} } catch (e) { e2 = e; } })();
|
|
await Promise.all([c1, c2]);
|
|
assert.ok(e1, 'client A received error');
|
|
assert.ok(e2, 'client B received error');
|
|
assert.equal(e1.message, 'synthetic source failure');
|
|
assert.equal(e2.message, 'synthetic source failure');
|
|
// No cache write.
|
|
const cached = await store.get('k', 'ck-27f');
|
|
assert.equal(cached, null);
|
|
// Inflight entry removed (next call would respawn).
|
|
assert.equal(store.stats('k').inflightCount, 0);
|
|
});
|
|
|
|
// D57 — ADR 0005 Amendment 8 §8: backpressure — slow client overflows queue
|
|
it('27g — backpressure: slow client queue overflow → STREAM_BACKPRESSURE terminator; fast client unaffected', async () => {
|
|
const warnings = [];
|
|
const store = new CacheStore({ _warnFn: (msg, meta) => warnings.push({ msg, meta }) });
|
|
// Inject perClientQueueCap=1024 bytes; chunks ~256B each.
|
|
const bigText = 'x'.repeat(250);
|
|
const factory = () => makeChunkSequence(
|
|
[{ idx: 0, t: bigText }, { idx: 1, t: bigText }, { idx: 2, t: bigText }, { idx: 3, t: bigText }, { idx: 4, t: bigText }, { idx: 5, t: bigText }, { type: 'stop', finish_reason: 'stop' }]
|
|
);
|
|
const r1 = await store.getOrComputeStreaming('k', 'ck-27g', factory, { clientId: 'fast', perClientQueueCap: 1024 });
|
|
const r2 = await store.getOrComputeStreaming('k', 'ck-27g', factory, { clientId: 'slow', perClientQueueCap: 1024 });
|
|
// Fast drains immediately; slow defers consumption.
|
|
const fastOut = [];
|
|
const fast = (async () => { for await (const c of r1.stream) fastOut.push(c); })();
|
|
await fast;
|
|
// Now consume slow's stream; should hit backpressure terminator.
|
|
const slowOut = [];
|
|
for await (const c of r2.stream) slowOut.push(c);
|
|
// Fast client received the full sequence.
|
|
assert.equal(fastOut.length, 7);
|
|
assert.deepEqual(fastOut[6], { type: 'stop', finish_reason: 'stop' });
|
|
// Slow client received some prefix + the STREAM_BACKPRESSURE terminator.
|
|
assert.deepEqual(slowOut[slowOut.length - 2], { type: 'stop', finish_reason: 'length' });
|
|
assert.equal(slowOut[slowOut.length - 1], '[DONE]');
|
|
assert.ok(slowOut.length < 7, 'slow client cut short before reaching natural end');
|
|
// Backpressure warning emitted for slow client.
|
|
const bp = warnings.filter(w => w.msg === 'stream_backpressure_disconnect');
|
|
assert.ok(bp.length >= 1, 'at least one stream_backpressure_disconnect emitted');
|
|
assert.ok(bp.some(w => w.meta.client_id === 'slow'), 'slow client identified in warning');
|
|
});
|
|
|
|
// D57 — ADR 0005 Amendment 8 §10: replay buffer cap — cache write skipped
|
|
it('27h — replay cap exceeded: cache write skipped; first caller still receives full stream; late joiner past cap gets STREAM_BACKPRESSURE', async () => {
|
|
const warnings = [];
|
|
const store = new CacheStore({ _warnFn: (msg, meta) => warnings.push({ msg, meta }) });
|
|
const big = 'y'.repeat(400);
|
|
// Source emits 6 chunks at ~400B each → ~2400B total, exceeds replay cap 1024.
|
|
const factory = () => makeChunkSequence(
|
|
[{ t: big }, { t: big }, { t: big }, { t: big }, { t: big }, { t: big }]
|
|
);
|
|
const r1 = await store.getOrComputeStreaming('k', 'ck-27h', factory, {
|
|
clientId: 'first',
|
|
accumulatedReplayCap: 1024,
|
|
perClientQueueCap: 1024 * 1024, // huge per-client cap so first caller never overflows
|
|
});
|
|
// First caller iterates through.
|
|
const itA = r1.stream;
|
|
const outA = [];
|
|
// Pull 2 chunks then attempt late join while past cap.
|
|
outA.push((await itA.next()).value);
|
|
outA.push((await itA.next()).value);
|
|
outA.push((await itA.next()).value); // now well past 1024B accumulated
|
|
// Late joiner attaches past replay cap → gets STREAM_BACKPRESSURE.
|
|
const r2 = await store.getOrComputeStreaming('k', 'ck-27h', factory, {
|
|
clientId: 'late',
|
|
accumulatedReplayCap: 1024,
|
|
perClientQueueCap: 1024,
|
|
});
|
|
assert.equal(r2.role, 'attached');
|
|
const outLate = [];
|
|
for await (const c of r2.stream) outLate.push(c);
|
|
// Late joiner's stream is just the backpressure terminator (drain over cap).
|
|
assert.deepEqual(outLate, [{ type: 'stop', finish_reason: 'length' }, '[DONE]']);
|
|
// Drain first caller fully.
|
|
for await (const c of itA) outA.push(c);
|
|
assert.equal(outA.length, 6, 'first caller receives full source stream');
|
|
// Cache write skipped.
|
|
const cached = await store.get('k', 'ck-27h');
|
|
assert.equal(cached, null, 'cache NOT written when replay cap exceeded');
|
|
// Replay-cap-exceeded warning emitted.
|
|
assert.ok(
|
|
warnings.some(w => w.msg === 'streaming_inflight_replay_cap_exceeded'),
|
|
'replay-cap-exceeded warning fired'
|
|
);
|
|
});
|
|
|
|
// D57 — ADR 0005 Amendment 8 §6: cache TTL race — late joiner attaches via inflight
|
|
it('27i — cache TTL race: cached entry expires during inflight; late joiner attaches via inflight Map', async () => {
|
|
const store = new CacheStore({ _warnFn: () => {} });
|
|
// Pre-populate cache with TTL=10ms — will expire shortly.
|
|
await store.set('k', 'ck-27i', ['cached-a', 'cached-b'], 10);
|
|
// Wait so the cached entry expires.
|
|
await new Promise(r => setTimeout(r, 20));
|
|
// Now a streaming request comes in: cache is expired → entry is recomputed.
|
|
let spawns = 0;
|
|
const factory = () => {
|
|
spawns++;
|
|
return makeChunkSequence(['live-1', 'live-2', 'live-3']);
|
|
};
|
|
const r1 = await store.getOrComputeStreaming('k', 'ck-27i', factory, { clientId: 'A' });
|
|
assert.equal(r1.role, 'source', 'expired cache → fresh source spawn');
|
|
// While the source is running, a late joiner arrives — must attach via inflight Map.
|
|
const r2 = await store.getOrComputeStreaming('k', 'ck-27i', factory, { clientId: 'B' });
|
|
assert.equal(r2.role, 'attached', 'late joiner attaches via inflight Map even after cache expiry');
|
|
assert.equal(spawns, 1, 'no respawn');
|
|
// Drain both.
|
|
const o1 = []; for await (const c of r1.stream) o1.push(c);
|
|
const o2 = []; for await (const c of r2.stream) o2.push(c);
|
|
assert.deepEqual(o1, ['live-1', 'live-2', 'live-3']);
|
|
assert.deepEqual(o2, ['live-1', 'live-2', 'live-3']);
|
|
// Inflight completion overwrites the expired slot.
|
|
const cached = await store.get('k', 'ck-27i');
|
|
assert.ok(cached);
|
|
assert.deepEqual(cached.value, ['live-1', 'live-2', 'live-3']);
|
|
});
|
|
|
|
// D57 — ADR 0005 Amendment 8 §7: sourceFactory throws → first caller errors; no zombie state
|
|
it('27j — sourceFactory throws (e.g. CONCURRENCY_LIMIT): first caller errors; subsequent call retries; no zombie inflight', async () => {
|
|
const store = new CacheStore({ _warnFn: () => {} });
|
|
let attempts = 0;
|
|
const factory = () => {
|
|
attempts++;
|
|
if (attempts === 1) {
|
|
throw new Error('CONCURRENCY_LIMIT');
|
|
}
|
|
return makeChunkSequence(['a', 'b']);
|
|
};
|
|
let firstErr;
|
|
try {
|
|
await store.getOrComputeStreaming('k', 'ck-27j', factory, { clientId: 'A' });
|
|
} catch (e) {
|
|
firstErr = e;
|
|
}
|
|
assert.ok(firstErr, 'first call rejected with factory error');
|
|
assert.equal(firstErr.message, 'CONCURRENCY_LIMIT');
|
|
// No zombie inflight entry left dangling.
|
|
assert.equal(store.stats('k').inflightCount, 0, 'no stale streaming-inflight entry');
|
|
// Subsequent call uses the factory again (it returns a real iterator now).
|
|
const r2 = await store.getOrComputeStreaming('k', 'ck-27j', factory, { clientId: 'B' });
|
|
assert.equal(r2.role, 'source');
|
|
const out = [];
|
|
for await (const c of r2.stream) out.push(c);
|
|
assert.deepEqual(out, ['a', 'b']);
|
|
assert.equal(attempts, 2);
|
|
});
|
|
|
|
// D57 — ADR 0005 Amendment 8 §1: stats accounting (hits / misses / inflightCount)
|
|
it('27k — stats: hits incremented for cache_hit + attached; misses for source; inflightCount reflects active streaming entries', async () => {
|
|
const store = new CacheStore({ _warnFn: () => {} });
|
|
const factory = () => makeChunkSequence(['p', 'q', 'r']);
|
|
|
|
// First caller — miss.
|
|
const r1 = await store.getOrComputeStreaming('k', 'ck-27k', factory, { clientId: 'A' });
|
|
let stats = store.stats('k');
|
|
assert.equal(stats.misses, 1, 'first caller increments misses');
|
|
assert.equal(stats.hits, 0);
|
|
// Inflight entry alive during source phase.
|
|
assert.equal(stats.inflightCount, 1, 'streaming entry counted in inflightCount');
|
|
|
|
// Concurrent joiner — attached → hit.
|
|
const r2 = await store.getOrComputeStreaming('k', 'ck-27k', factory, { clientId: 'B' });
|
|
stats = store.stats('k');
|
|
assert.equal(stats.hits, 1, 'attached client increments hits');
|
|
// Drain both.
|
|
await Promise.all([
|
|
(async () => { for await (const _c of r1.stream) {} })(),
|
|
(async () => { for await (const _c of r2.stream) {} })(),
|
|
]);
|
|
// After completion, entry removed from inflight.
|
|
stats = store.stats('k');
|
|
assert.equal(stats.inflightCount, 0, 'streaming entry removed post-completion');
|
|
|
|
// Third call after completion — cache_hit (also a hit).
|
|
const r3 = await store.getOrComputeStreaming('k', 'ck-27k', factory, { clientId: 'C' });
|
|
assert.equal(r3.role, 'cache_hit');
|
|
for await (const _c of r3.stream) { /* drain */ }
|
|
stats = store.stats('k');
|
|
assert.equal(stats.hits, 2, 'cache_hit also increments hits');
|
|
});
|
|
|
|
// D57 — ADR 0005 Amendment 8 §2: composite key isolation (keyId\0cacheKey)
|
|
it('27l — composite key isolation: same cacheKey + different keyId → two independent inflight entries + two spawns', async () => {
|
|
const store = new CacheStore({ _warnFn: () => {} });
|
|
let spawns = 0;
|
|
const factory = () => {
|
|
spawns++;
|
|
return makeChunkSequence(['n1', 'n2']);
|
|
};
|
|
// Same cacheKey, different keyId.
|
|
const p1 = store.getOrComputeStreaming('key-A', 'shared-cache-key', factory, { clientId: 'A' });
|
|
const p2 = store.getOrComputeStreaming('key-B', 'shared-cache-key', factory, { clientId: 'B' });
|
|
const [r1, r2] = await Promise.all([p1, p2]);
|
|
// Both should be `source` — no cross-keyId sharing.
|
|
assert.equal(r1.role, 'source');
|
|
assert.equal(r2.role, 'source');
|
|
assert.equal(spawns, 2, 'two spawns because two distinct (keyId,cacheKey) composites');
|
|
// Drain.
|
|
const out1 = []; for await (const c of r1.stream) out1.push(c);
|
|
const out2 = []; for await (const c of r2.stream) out2.push(c);
|
|
assert.deepEqual(out1, ['n1', 'n2']);
|
|
assert.deepEqual(out2, ['n1', 'n2']);
|
|
// Stats: each keyId has its own miss counter.
|
|
assert.equal(store.stats('key-A').misses, 1);
|
|
assert.equal(store.stats('key-B').misses, 1);
|
|
});
|
|
});
|
|
|
|
// ── Suite 28: D58 streaming singleflight (server.mjs HTTP wiring) ──────────
|
|
//
|
|
// Authority: ADR 0005 Amendment 8 §§ 7, 8, 9, 11, 12 (issue #16, D58).
|
|
//
|
|
// HTTP-layer integration tests for the server.mjs streaming branch wired to
|
|
// cacheStore.getOrComputeStreaming(). D57 (cache layer) tested singleflight
|
|
// in unit form; D58 verifies the server actually plumbs it through and emits
|
|
// the new X-OLP-Streaming-Inflight header. Provider spawn is mocked at the
|
|
// plugin level via lp.set('anthropic', { ...real, spawn: fake }) — the
|
|
// pattern used by Suites 15d/15e — so no real CLI is invoked.
|
|
//
|
|
// Test count: 8 (28a single, 28b 2-concurrent join, 28c TOCTOU pre-cache,
|
|
// 28d mid-stream join behaviour-validated via parallel HTTP, 28e omitted in
|
|
// favour of Suite 27g unit coverage per D58 prompt, 28f one-of-N disconnect,
|
|
// 28g all-disconnect, 28h CONCURRENCY_LIMIT fallthrough).
|
|
|
|
describe('Suite 28 — D58 streaming singleflight (server.mjs HTTP wiring, ADR 0005 Amendment 8)', () => {
|
|
let server28;
|
|
let port28;
|
|
let savedToken28;
|
|
let lp28;
|
|
let savedAnthropic28;
|
|
|
|
before(async () => {
|
|
savedToken28 = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-suite-28';
|
|
__setProvidersEnabled({ anthropic: true });
|
|
|
|
const mod = await import('./server.mjs');
|
|
lp28 = mod.loadedProviders;
|
|
savedAnthropic28 = lp28.get('anthropic');
|
|
mod.__clearCache();
|
|
|
|
server28 = mod.createOlpServer();
|
|
await new Promise((resolve, reject) => {
|
|
server28.listen(0, '127.0.0.1', resolve);
|
|
server28.once('error', reject);
|
|
});
|
|
port28 = server28.address().port;
|
|
});
|
|
|
|
after(async () => {
|
|
// Restore original anthropic provider so other suites are unaffected.
|
|
if (savedAnthropic28 !== undefined) {
|
|
lp28.set('anthropic', savedAnthropic28);
|
|
} else {
|
|
lp28.delete('anthropic');
|
|
}
|
|
__resetProvidersEnabled();
|
|
__resetSpawnImpl();
|
|
if (savedToken28 !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken28;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
if (!server28) return;
|
|
return new Promise(r => server28.close(r));
|
|
});
|
|
|
|
/**
|
|
* Install a custom spawn async generator on the anthropic plugin for one
|
|
* test. `factory` is a function returning the async generator each spawn.
|
|
* Tracks invocation count via spawnCount.
|
|
*/
|
|
function installFakeStreamProvider(spawnImpl) {
|
|
const counter = { count: 0 };
|
|
const fake = {
|
|
...savedAnthropic28,
|
|
spawn: async function* (ir, authContext) {
|
|
counter.count++;
|
|
yield* spawnImpl(ir, authContext);
|
|
},
|
|
};
|
|
lp28.set('anthropic', fake);
|
|
return counter;
|
|
}
|
|
|
|
/**
|
|
* Fire an SSE request and collect the full body + headers. The promise
|
|
* resolves when the server ends the response (res 'end' event).
|
|
*/
|
|
function makeStreamRequest(extra = {}) {
|
|
return new Promise((resolve, reject) => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port: port28,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
}, res => {
|
|
let data = '';
|
|
res.on('data', d => { data += d.toString(); });
|
|
res.on('end', () => resolve({
|
|
status: res.statusCode,
|
|
body: data,
|
|
headers: res.headers,
|
|
}));
|
|
res.on('error', reject);
|
|
});
|
|
req.on('error', reject);
|
|
req.write(JSON.stringify({
|
|
model: extra.model ?? 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: extra.prompt ?? 'd58-default' }],
|
|
stream: true,
|
|
}));
|
|
req.end();
|
|
});
|
|
}
|
|
|
|
/**
|
|
* Fire an SSE request and abort it after `abortAfterMs` ms. Returns the
|
|
* partial body collected up to abort. Resolves on the abort event.
|
|
*/
|
|
function makeAbortableStreamRequest({ prompt, abortAfterMs }) {
|
|
return new Promise((resolve, reject) => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port: port28,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
}, res => {
|
|
let data = '';
|
|
res.on('data', d => { data += d.toString(); });
|
|
res.on('end', () => resolve({
|
|
status: res.statusCode,
|
|
body: data,
|
|
headers: res.headers,
|
|
aborted: false,
|
|
}));
|
|
res.on('error', () => resolve({
|
|
status: res.statusCode,
|
|
body: data,
|
|
headers: res.headers,
|
|
aborted: true,
|
|
}));
|
|
});
|
|
req.on('error', () => resolve({ aborted: true }));
|
|
req.write(JSON.stringify({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: prompt }],
|
|
stream: true,
|
|
}));
|
|
req.end();
|
|
setTimeout(() => {
|
|
try { req.destroy(); } catch { /* ignore */ }
|
|
}, abortAfterMs);
|
|
});
|
|
}
|
|
|
|
beforeEach(async () => {
|
|
const mod = await import('./server.mjs');
|
|
mod.__clearCache();
|
|
});
|
|
|
|
it('28a — single SSE request: X-OLP-Streaming-Inflight: source; cache populated; identical re-request → cache hit', async () => {
|
|
// D58 — ADR 0005 Amendment 8 §11: single client gets `source` role,
|
|
// X-OLP-Cache: miss. Subsequent identical request hits the cache.
|
|
const counter = installFakeStreamProvider(async function* (_ir) {
|
|
yield { type: 'delta', content: 'chunk-A' };
|
|
yield { type: 'delta', content: 'chunk-B' };
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
});
|
|
|
|
const r1 = await makeStreamRequest({ prompt: 'd58-28a' });
|
|
assert.equal(r1.status, 200, `r1 status ${r1.status}: ${r1.body.slice(0, 200)}`);
|
|
assert.equal(r1.headers['x-olp-streaming-inflight'], 'source',
|
|
'r1 must emit X-OLP-Streaming-Inflight: source');
|
|
assert.equal(r1.headers['x-olp-cache'], 'miss',
|
|
'r1 must be X-OLP-Cache: miss');
|
|
assert.ok(r1.body.includes('chunk-A'), 'r1 body must include chunk-A');
|
|
assert.ok(r1.body.includes('chunk-B'), 'r1 body must include chunk-B');
|
|
assert.ok(r1.body.includes('[DONE]'), 'r1 body must end with [DONE]');
|
|
assert.equal(counter.count, 1, 'exactly one spawn for r1');
|
|
|
|
// Second identical request — cache hit, no spawn.
|
|
const r2 = await makeStreamRequest({ prompt: 'd58-28a' });
|
|
assert.equal(r2.status, 200);
|
|
assert.equal(r2.headers['x-olp-cache'], 'hit', 'r2 must be cache hit');
|
|
// X-OLP-Streaming-Inflight is omitted on cache_hit (X-OLP-Cache: hit
|
|
// already signals that path per D58 design).
|
|
assert.equal(r2.headers['x-olp-streaming-inflight'], undefined,
|
|
'cache_hit path must omit X-OLP-Streaming-Inflight header');
|
|
assert.equal(counter.count, 1, 'no additional spawn for r2');
|
|
});
|
|
|
|
it('28b — 2 concurrent identical SSE: one spawn; first=source second=attached; identical chunks', async () => {
|
|
// D58 — ADR 0005 Amendment 8 §1, §11: two concurrent identical streams
|
|
// share one underlying spawn. First gets X-OLP-Streaming-Inflight:
|
|
// source; second gets attached. Both bodies contain the same chunks.
|
|
//
|
|
// Pacing: source yields chunks with a setTimeout gap so the second
|
|
// request has time to fire and attach mid-stream.
|
|
const counter = installFakeStreamProvider(async function* (_ir) {
|
|
// Slow pacing so the second request can attach before completion.
|
|
for (const t of ['c0', 'c1', 'c2', 'c3']) {
|
|
yield { type: 'delta', content: t };
|
|
await new Promise(r => setTimeout(r, 20));
|
|
}
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
});
|
|
|
|
// Fire request 1, then a few ms later fire request 2; both run to
|
|
// completion in parallel.
|
|
const p1 = makeStreamRequest({ prompt: 'd58-28b' });
|
|
// Give p1 a head start to register the inflight entry.
|
|
await new Promise(r => setTimeout(r, 10));
|
|
const p2 = makeStreamRequest({ prompt: 'd58-28b' });
|
|
|
|
const [r1, r2] = await Promise.all([p1, p2]);
|
|
assert.equal(r1.status, 200);
|
|
assert.equal(r2.status, 200);
|
|
assert.equal(counter.count, 1, 'only one underlying spawn for both requests');
|
|
|
|
// First caller (registered the inflight entry) gets source; the other
|
|
// gets attached. The order is set by who hit getOrComputeStreaming
|
|
// first — we asserted that with the 10ms head start above.
|
|
assert.equal(r1.headers['x-olp-streaming-inflight'], 'source',
|
|
'r1 must be source role');
|
|
assert.equal(r2.headers['x-olp-streaming-inflight'], 'attached',
|
|
'r2 must be attached role');
|
|
|
|
// Both responses must contain all 4 delta chunks + [DONE].
|
|
for (const tag of ['c0', 'c1', 'c2', 'c3', '[DONE]']) {
|
|
assert.ok(r1.body.includes(tag), `r1 missing ${tag}`);
|
|
assert.ok(r2.body.includes(tag), `r2 missing ${tag}`);
|
|
}
|
|
|
|
// Cache_status header: source=miss, attached=hit (cache_hit role)?
|
|
// No — attached client did NOT hit the cache (cache was empty). The
|
|
// server distinguishes by setting cache_status header to 'miss' for
|
|
// attached and 'miss' for source. Verify both are 'miss' on the wire
|
|
// (X-OLP-Cache header). cache_status='streaming_attached' is the AUDIT
|
|
// value, not the wire header value.
|
|
assert.equal(r1.headers['x-olp-cache'], 'miss');
|
|
assert.equal(r2.headers['x-olp-cache'], 'miss');
|
|
});
|
|
|
|
it('28c — TOCTOU regression: pre-populated cache + 2 concurrent identical streams → both cache_hit, no spawn', async () => {
|
|
// D58 — ADR 0005 Amendment 8 §6 / §11: when the cache is already
|
|
// populated, preCheckHit gates entry to the streaming-singleflight
|
|
// branch. The streaming-singleflight branch should not run at all.
|
|
// 2 concurrent identical requests both replay from cache; spawn count
|
|
// stays at 0.
|
|
const counter = installFakeStreamProvider(async function* (_ir) {
|
|
yield { type: 'delta', content: 'should-not-fire' };
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
});
|
|
|
|
// Populate the cache via a first request.
|
|
const r0 = await makeStreamRequest({ prompt: 'd58-28c' });
|
|
assert.equal(r0.status, 200);
|
|
assert.equal(counter.count, 1, 'r0 spawned once to populate cache');
|
|
|
|
// Now fire 2 concurrent identical requests — both should be cache hits.
|
|
const [r1, r2] = await Promise.all([
|
|
makeStreamRequest({ prompt: 'd58-28c' }),
|
|
makeStreamRequest({ prompt: 'd58-28c' }),
|
|
]);
|
|
assert.equal(r1.status, 200);
|
|
assert.equal(r2.status, 200);
|
|
assert.equal(r1.headers['x-olp-cache'], 'hit', 'r1 must be cache hit');
|
|
assert.equal(r2.headers['x-olp-cache'], 'hit', 'r2 must be cache hit');
|
|
// Neither should have entered the streaming-singleflight branch.
|
|
assert.equal(r1.headers['x-olp-streaming-inflight'], undefined,
|
|
'r1 must not emit X-OLP-Streaming-Inflight (served from buffered cache replay)');
|
|
assert.equal(r2.headers['x-olp-streaming-inflight'], undefined,
|
|
'r2 must not emit X-OLP-Streaming-Inflight (served from buffered cache replay)');
|
|
assert.equal(counter.count, 1, 'no additional spawns');
|
|
});
|
|
|
|
it('28d — mid-stream join (HTTP-level): 2 concurrent SSE requests share one spawn; both bodies identical', async () => {
|
|
// D58 — ADR 0005 Amendment 8 §5: late joiner gets accumulated burst +
|
|
// live tail. At the HTTP level we can't observe the burst-vs-live split
|
|
// precisely (it's internal to the cache layer), but we can verify the
|
|
// end-to-end invariant: both clients receive the same chunk sequence
|
|
// even when one joins mid-source. Suite 27c covers the burst-vs-live
|
|
// split at the cache-layer unit level.
|
|
const counter = installFakeStreamProvider(async function* (_ir) {
|
|
// Slower pacing so the second client clearly joins mid-stream.
|
|
for (const t of ['m0', 'm1', 'm2', 'm3', 'm4']) {
|
|
yield { type: 'delta', content: t };
|
|
await new Promise(r => setTimeout(r, 25));
|
|
}
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
});
|
|
|
|
const p1 = makeStreamRequest({ prompt: 'd58-28d' });
|
|
// p2 joins ~40ms in — at least 2 chunks have been accumulated by then.
|
|
await new Promise(r => setTimeout(r, 40));
|
|
const p2 = makeStreamRequest({ prompt: 'd58-28d' });
|
|
|
|
const [r1, r2] = await Promise.all([p1, p2]);
|
|
assert.equal(r1.status, 200);
|
|
assert.equal(r2.status, 200);
|
|
assert.equal(counter.count, 1, 'exactly one underlying spawn');
|
|
assert.equal(r1.headers['x-olp-streaming-inflight'], 'source');
|
|
assert.equal(r2.headers['x-olp-streaming-inflight'], 'attached');
|
|
for (const tag of ['m0', 'm1', 'm2', 'm3', 'm4']) {
|
|
assert.ok(r1.body.includes(tag), `r1 missing ${tag}`);
|
|
assert.ok(r2.body.includes(tag), `r2 missing ${tag} (late-joiner replay must include burst)`);
|
|
}
|
|
});
|
|
|
|
it('28f — one of N clients disconnects mid-stream: source NOT aborted; other client completes; cache populated', async () => {
|
|
// D58 — ADR 0005 Amendment 8 §9: when one of N attached clients drops,
|
|
// the source continues for the remaining clients. The cache write
|
|
// happens because the source completed normally for the surviving
|
|
// client.
|
|
const counter = installFakeStreamProvider(async function* (_ir) {
|
|
for (const t of ['s0', 's1', 's2', 's3', 's4', 's5']) {
|
|
yield { type: 'delta', content: t };
|
|
await new Promise(r => setTimeout(r, 25));
|
|
}
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
});
|
|
|
|
// r1 starts and registers the inflight entry.
|
|
const p1 = makeStreamRequest({ prompt: 'd58-28f' });
|
|
await new Promise(r => setTimeout(r, 15));
|
|
// r2 attaches but aborts after ~40ms.
|
|
const p2 = makeAbortableStreamRequest({ prompt: 'd58-28f', abortAfterMs: 40 });
|
|
|
|
const [r1, r2] = await Promise.all([p1, p2]);
|
|
// r1 (the source) completes normally with the full stream.
|
|
assert.equal(r1.status, 200);
|
|
assert.equal(r1.headers['x-olp-streaming-inflight'], 'source');
|
|
for (const tag of ['s0', 's1', 's2', 's3', 's4', 's5', '[DONE]']) {
|
|
assert.ok(r1.body.includes(tag), `r1 missing ${tag}`);
|
|
}
|
|
assert.equal(counter.count, 1, 'source spawn was NOT re-fired by the disconnect');
|
|
|
|
// r2 was aborted — partial body is okay; what matters is the source
|
|
// wasn't killed (verified above by r1 completing).
|
|
// Subsequent identical request must be a cache hit (cache was populated
|
|
// by the source on normal completion).
|
|
const r3 = await makeStreamRequest({ prompt: 'd58-28f' });
|
|
assert.equal(r3.status, 200);
|
|
assert.equal(r3.headers['x-olp-cache'], 'hit', 'cache populated after source completion');
|
|
assert.equal(counter.count, 1, 'no additional spawn for r3');
|
|
});
|
|
|
|
it('28g — ALL clients disconnect mid-stream: source aborted; no cache write; subsequent request respawns', async () => {
|
|
// D58 — ADR 0005 Amendment 8 §9 + §4: when all attached clients
|
|
// disconnect, the cache layer fires sourceAbortController.abort() and
|
|
// does NOT write the cache. A subsequent identical request must spawn
|
|
// afresh (no inflight entry left dangling).
|
|
//
|
|
// Implementation detail: the fake spawn's async generator must respect
|
|
// the abort signal (or simply have its iterator.return() called when
|
|
// the spawn is iterated by the tee task — see Suite 27 unit tests).
|
|
// For an async generator with `await new Promise(setTimeout)` between
|
|
// yields, calling .return() naturally propagates because the for-await
|
|
// exits cleanly via the try/finally.
|
|
let sourceFinished = false;
|
|
const counter = installFakeStreamProvider(async function* (_ir) {
|
|
try {
|
|
for (let i = 0; i < 30; i++) {
|
|
yield { type: 'delta', content: `g${i}` };
|
|
await new Promise(r => setTimeout(r, 20));
|
|
}
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
sourceFinished = true;
|
|
} finally {
|
|
// intentionally no-op; the "finished without abort" signal lives in
|
|
// sourceFinished above.
|
|
}
|
|
});
|
|
|
|
// Only one client; abort it after ~40ms (well before completion).
|
|
const r1 = await makeAbortableStreamRequest({ prompt: 'd58-28g', abortAfterMs: 40 });
|
|
assert.equal(counter.count, 1, 'spawn fired once');
|
|
// Wait for the cache layer to observe attachedClients.size === 0 and abort.
|
|
await new Promise(r => setTimeout(r, 100));
|
|
assert.equal(sourceFinished, false, 'source was aborted, not completed');
|
|
|
|
// Subsequent identical request must respawn (no cache, no inflight).
|
|
const r2 = await makeStreamRequest({ prompt: 'd58-28g' });
|
|
assert.equal(r2.status, 200);
|
|
assert.equal(r2.headers['x-olp-cache'], 'miss', 'no cache write after abort');
|
|
assert.equal(r2.headers['x-olp-streaming-inflight'], 'source', 'fresh source role');
|
|
assert.equal(counter.count, 2, 'r2 triggered a fresh spawn');
|
|
});
|
|
|
|
it('28h — CONCURRENCY_LIMIT fallthrough: different cacheKeys at maxConcurrent=1 → first succeeds; second falls through to buffered path', async () => {
|
|
// D58 — ADR 0005 Amendment 8 §7: CONCURRENCY_LIMIT thrown by
|
|
// sourceFactory falls through to the buffered path. Different
|
|
// cacheKeys do NOT share an inflight entry (they're not singleflight
|
|
// candidates), so request 2's sourceFactory throws and the streaming
|
|
// branch is bypassed for that request. The buffered path then
|
|
// re-attempts acquire; since maxConcurrent=1 and request 1 still has
|
|
// the slot, it surfaces a chain-exhausted 502 (single-hop saturation).
|
|
//
|
|
// We install a custom provider with maxConcurrent=1 + slow stream to
|
|
// hold the slot while request 2 fires.
|
|
let releaseSig;
|
|
const releaseGate = new Promise(r => { releaseSig = r; });
|
|
const counter = { count: 0 };
|
|
const fake = {
|
|
...savedAnthropic28,
|
|
hints: { ...savedAnthropic28.hints, maxConcurrent: 1 },
|
|
spawn: async function* (_ir, _ctx) {
|
|
counter.count++;
|
|
// First spawn holds the slot until releaseGate is signalled.
|
|
yield { type: 'delta', content: `h${counter.count}-0` };
|
|
await releaseGate;
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
},
|
|
};
|
|
lp28.set('anthropic', fake);
|
|
|
|
try {
|
|
// Fire request 1 with prompt-A; it acquires the slot and stalls.
|
|
const p1 = makeStreamRequest({ prompt: 'd58-28h-A' });
|
|
await new Promise(r => setTimeout(r, 30));
|
|
// Fire request 2 with prompt-B (different cache key) — should
|
|
// CONCURRENCY_LIMIT in factory, fall through to buffered path,
|
|
// which will also fail acquire → chain-exhausted 502.
|
|
const r2 = await makeStreamRequest({ prompt: 'd58-28h-B' });
|
|
// The buffered fallback engine surfaces a chain-exhausted error as
|
|
// 502 for single-hop saturation per pre-D58 behaviour.
|
|
assert.ok(r2.status === 502 || r2.status === 503,
|
|
`expected 502/503 chain-exhausted, got ${r2.status}: ${r2.body.slice(0, 200)}`);
|
|
|
|
// Release request 1's stall.
|
|
releaseSig();
|
|
const r1 = await p1;
|
|
assert.equal(r1.status, 200, 'r1 must complete normally');
|
|
// Exactly one spawn — request 2 never spawned (CONCURRENCY_LIMIT).
|
|
assert.equal(counter.count, 1, 'only one underlying spawn');
|
|
} finally {
|
|
// In case of an early failure, release the gate so promises settle.
|
|
try { releaseSig(); } catch { /* already released */ }
|
|
}
|
|
});
|
|
|
|
it('28i — stop-less exhaustion: source generator returns without {type:"stop"} → cache NOT populated (D58 follow-up to D58 reviewer P2-2)', async () => {
|
|
// ADR 0005 § "Cache write conditions" item 1 (D16 truncated-not-cached
|
|
// invariant): if the source generator exhausts without emitting a stop
|
|
// chunk, the response is treated as truncated and MUST NOT persist in
|
|
// cache. D57's cache layer is IR-agnostic and writes accumulatedChunks
|
|
// on any source exhaustion; D58's server.mjs handles the IR semantics by
|
|
// calling cacheStore.delete(...) immediately after on the no-stop path.
|
|
// This test pins the end-to-end behaviour from the HTTP layer.
|
|
const counter = { count: 0 };
|
|
const fake = {
|
|
...savedAnthropic28,
|
|
spawn: async function* (_ir, _ctx) {
|
|
counter.count++;
|
|
yield { type: 'delta', content: 'no-stop-1' };
|
|
yield { type: 'delta', content: 'no-stop-2' };
|
|
// Generator returns WITHOUT a stop chunk — truncation path.
|
|
},
|
|
};
|
|
lp28.set('anthropic', fake);
|
|
const r1 = await makeStreamRequest({ prompt: 'd58-28i' });
|
|
assert.equal(r1.status, 200, '28i r1: SSE response 200');
|
|
assert.equal(counter.count, 1, '28i r1: spawned once');
|
|
// The synthetic truncation marker {type:"stop", finish_reason:"length"}
|
|
// should appear (D26 F19 in-band signal); [DONE] terminator follows.
|
|
assert.ok(r1.body.includes('"finish_reason":"length"'),
|
|
`28i r1: truncation marker expected; body=${r1.body.slice(0, 300)}`);
|
|
assert.ok(r1.body.includes('[DONE]'), '28i r1: [DONE] terminator');
|
|
// Subsequent identical request must respawn (cache was NOT populated).
|
|
const r2 = await makeStreamRequest({ prompt: 'd58-28i' });
|
|
assert.equal(r2.status, 200, '28i r2: SSE response 200');
|
|
assert.equal(counter.count, 2, '28i r2: second request triggered a fresh spawn (no cache reuse for truncated entry)');
|
|
});
|
|
});
|
|
|
|
// ── Suite 29: D61 SSE heartbeat (ADR 0010 § Phase 4 D61-D63) ──────────────
|
|
//
|
|
// Tests the opt-in SSE heartbeat that emits `: keepalive\n\n` SSE comment
|
|
// frames during silent windows. Port of OCP `startHeartbeat` (server.mjs:
|
|
// 660-685) adapted to OLP's config (`streaming.heartbeat_interval_ms`),
|
|
// per-attached-client lifecycle, and eager-headers-post-spawn rule.
|
|
//
|
|
// Authority: ADR 0010 § Phase 4 D61-D63; OCP spec
|
|
// docs/superpowers/specs/2026-04-25-47-sse-heartbeat-design.md.
|
|
|
|
describe('Suite 29 — D61 SSE heartbeat (ADR 0010 § Phase 4 D61-D63)', () => {
|
|
let server29;
|
|
let port29;
|
|
let savedToken29;
|
|
let lp29;
|
|
let savedAnthropic29;
|
|
|
|
before(async () => {
|
|
savedToken29 = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-suite-29';
|
|
__setProvidersEnabled({ anthropic: true });
|
|
|
|
const mod = await import('./server.mjs');
|
|
lp29 = mod.loadedProviders;
|
|
savedAnthropic29 = lp29.get('anthropic');
|
|
mod.__clearCache();
|
|
|
|
server29 = mod.createOlpServer();
|
|
await new Promise((resolve, reject) => {
|
|
server29.listen(0, '127.0.0.1', resolve);
|
|
server29.once('error', reject);
|
|
});
|
|
port29 = server29.address().port;
|
|
});
|
|
|
|
after(async () => {
|
|
if (savedAnthropic29 !== undefined) {
|
|
lp29.set('anthropic', savedAnthropic29);
|
|
} else {
|
|
lp29.delete('anthropic');
|
|
}
|
|
__resetProvidersEnabled();
|
|
__resetSpawnImpl();
|
|
__resetStreamingConfig();
|
|
if (savedToken29 !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken29;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
if (!server29) return;
|
|
return new Promise(r => server29.close(r));
|
|
});
|
|
|
|
beforeEach(async () => {
|
|
const mod = await import('./server.mjs');
|
|
mod.__clearCache();
|
|
__setStreamingConfig({ heartbeat_interval_ms: 0 });
|
|
});
|
|
|
|
/** Install a fake provider whose async generator paces yields with `gapMs`. */
|
|
function installPacedProvider(chunks, gapMs) {
|
|
const counter = { count: 0 };
|
|
const fake = {
|
|
...savedAnthropic29,
|
|
spawn: async function* (_ir, _ctx) {
|
|
counter.count++;
|
|
for (const c of chunks) {
|
|
if (gapMs > 0) await new Promise(r => setTimeout(r, gapMs));
|
|
yield c;
|
|
}
|
|
},
|
|
};
|
|
lp29.set('anthropic', fake);
|
|
return counter;
|
|
}
|
|
|
|
/** Make a streaming request and return body + arrival timestamps. */
|
|
function makeTimedStreamRequest({ prompt }) {
|
|
return new Promise((resolve, reject) => {
|
|
const arrivals = [];
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port: port29,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
}, res => {
|
|
let data = '';
|
|
res.on('data', d => {
|
|
const s = d.toString();
|
|
arrivals.push({ ts: Date.now(), text: s });
|
|
data += s;
|
|
});
|
|
res.on('end', () => resolve({
|
|
status: res.statusCode,
|
|
body: data,
|
|
headers: res.headers,
|
|
arrivals,
|
|
}));
|
|
res.on('error', reject);
|
|
});
|
|
req.on('error', reject);
|
|
req.write(JSON.stringify({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: prompt }],
|
|
stream: true,
|
|
}));
|
|
req.end();
|
|
});
|
|
}
|
|
|
|
it('29a — heartbeat ENABLED (interval=40ms) + slow provider → keepalive frame(s) appear in body', async () => {
|
|
// Provider stalls 200ms before first delta, then ~80ms between deltas.
|
|
// With heartbeat=40ms the silent windows produce multiple keepalive
|
|
// frames. Eager-headers must fire so the frames are flushed.
|
|
installPacedProvider([
|
|
{ type: 'delta', content: 'h29a-1' },
|
|
{ type: 'delta', content: 'h29a-2' },
|
|
{ type: 'stop', finish_reason: 'stop' },
|
|
], 100);
|
|
__setStreamingConfig({ heartbeat_interval_ms: 40 });
|
|
|
|
const r = await makeTimedStreamRequest({ prompt: 'h29a-prompt' });
|
|
assert.equal(r.status, 200, `r status ${r.status}: ${r.body.slice(0, 200)}`);
|
|
// At least one keepalive comment frame must be present.
|
|
const keepaliveCount = (r.body.match(/: keepalive/g) ?? []).length;
|
|
assert.ok(keepaliveCount >= 1, `expected at least one keepalive frame, got ${keepaliveCount}; body=${r.body.slice(0, 400)}`);
|
|
// Content + [DONE] still present (heartbeat does not break the stream).
|
|
assert.ok(r.body.includes('h29a-1'), 'body must include h29a-1');
|
|
assert.ok(r.body.includes('h29a-2'), 'body must include h29a-2');
|
|
assert.ok(r.body.includes('[DONE]'), 'body must include [DONE]');
|
|
});
|
|
|
|
it('29b — heartbeat DISABLED (default 0) → NO keepalive frame in body', async () => {
|
|
// Same provider pacing but no heartbeat config → no keepalive frames.
|
|
installPacedProvider([
|
|
{ type: 'delta', content: 'h29b-1' },
|
|
{ type: 'delta', content: 'h29b-2' },
|
|
{ type: 'stop', finish_reason: 'stop' },
|
|
], 100);
|
|
__setStreamingConfig({ heartbeat_interval_ms: 0 });
|
|
|
|
const r = await makeTimedStreamRequest({ prompt: 'h29b-prompt' });
|
|
assert.equal(r.status, 200);
|
|
assert.equal(
|
|
(r.body.match(/: keepalive/g) ?? []).length,
|
|
0,
|
|
`heartbeat disabled but found keepalive frame; body=${r.body.slice(0, 400)}`,
|
|
);
|
|
assert.ok(r.body.includes('h29b-1'));
|
|
assert.ok(r.body.includes('[DONE]'));
|
|
});
|
|
|
|
it('29c — heartbeat timer resets on every real chunk (chunks at 30ms gaps with hb=50ms → 0 keepalives mid-stream)', async () => {
|
|
// With chunks arriving every 30ms and heartbeat=50ms, the timer resets
|
|
// before it can fire — there should be no keepalive frames between
|
|
// chunks (silent windows never exceed 50ms). Pacing the source by a
|
|
// long pre-first-chunk gap is avoided here: pre-first-chunk silence is
|
|
// covered by 29a.
|
|
installPacedProvider([
|
|
{ type: 'delta', content: 'r29c-1' },
|
|
{ type: 'delta', content: 'r29c-2' },
|
|
{ type: 'delta', content: 'r29c-3' },
|
|
{ type: 'delta', content: 'r29c-4' },
|
|
{ type: 'stop', finish_reason: 'stop' },
|
|
], 30);
|
|
__setStreamingConfig({ heartbeat_interval_ms: 50 });
|
|
|
|
const r = await makeTimedStreamRequest({ prompt: 'h29c-prompt' });
|
|
assert.equal(r.status, 200);
|
|
// Each chunk fires within 30ms of the previous, so the heartbeat (50ms)
|
|
// is repeatedly reset before it can fire. There may still be ONE
|
|
// keepalive frame from the pre-first-chunk window (30ms wait before
|
|
// first delta is borderline), but we expect zero mid-stream frames.
|
|
const keepaliveCount = (r.body.match(/: keepalive/g) ?? []).length;
|
|
assert.ok(keepaliveCount <= 1, `expected <= 1 keepalive (reset working), got ${keepaliveCount}; body=${r.body.slice(0, 400)}`);
|
|
for (const tag of ['r29c-1', 'r29c-2', 'r29c-3', 'r29c-4', '[DONE]']) {
|
|
assert.ok(r.body.includes(tag), `body missing ${tag}`);
|
|
}
|
|
});
|
|
|
|
it('29d — heartbeat cancelled on client disconnect (no further keepalive after abort)', async () => {
|
|
// Source stalls forever. Heartbeat=20ms fires repeatedly while the
|
|
// client is connected. After abort, heartbeat.stop() must fire — the
|
|
// surface check we do here: the server cleanly closes (no uncaught
|
|
// exception on continued setTimeout firing). We can't observe "no more
|
|
// keepalive frames" directly from the aborted client, but we CAN
|
|
// observe that the test runner doesn't time out — the dead-socket
|
|
// res.write inside the heartbeat is swallowed (no throw bubbling up).
|
|
let yieldedCount = 0;
|
|
const fake = {
|
|
...savedAnthropic29,
|
|
spawn: async function* (_ir, _ctx) {
|
|
// Hold open with a single delta then a long sleep.
|
|
yield { type: 'delta', content: 'first-delta' };
|
|
yieldedCount++;
|
|
try {
|
|
// Long stall; should be interrupted by iterator.return()
|
|
await new Promise(r => setTimeout(r, 1000));
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
} finally {
|
|
// intentional
|
|
}
|
|
},
|
|
};
|
|
lp29.set('anthropic', fake);
|
|
__setStreamingConfig({ heartbeat_interval_ms: 20 });
|
|
|
|
const result = await new Promise((resolve) => {
|
|
const req = httpRequest({
|
|
hostname: '127.0.0.1',
|
|
port: port29,
|
|
method: 'POST',
|
|
path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
}, res => {
|
|
let body = '';
|
|
res.on('data', d => { body += d.toString(); });
|
|
res.on('end', () => resolve({ body, aborted: false }));
|
|
res.on('error', () => resolve({ body, aborted: true }));
|
|
});
|
|
req.on('error', () => resolve({ body: '', aborted: true }));
|
|
req.write(JSON.stringify({
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: 'h29d-abort' }],
|
|
stream: true,
|
|
}));
|
|
req.end();
|
|
// Abort after 100ms (well into heartbeat firing window).
|
|
setTimeout(() => { try { req.destroy(); } catch { /* ignore */ } }, 100);
|
|
});
|
|
// Sanity: client saw at least the first chunk + maybe keepalives.
|
|
assert.ok(yieldedCount === 1, 'first delta was yielded once');
|
|
// Give the heartbeat loop time to fire post-abort — if heartbeat.stop()
|
|
// didn't fire, an uncaught exception or runaway timer would surface.
|
|
// Then assert that the server is still responsive to a new request,
|
|
// proving no fatal lingering state.
|
|
await new Promise(r => setTimeout(r, 60));
|
|
|
|
// Server health check: a fresh request must complete normally.
|
|
const probe = await fetch({ port: port29, method: 'GET', path: '/v1/models' });
|
|
assert.equal(probe.status, 200, 'server remains responsive after disconnect; heartbeat timer did not crash the process');
|
|
});
|
|
|
|
it('29e — eager-headers: heartbeat enabled → headers flushed BEFORE first content chunk', async () => {
|
|
// Pre-first-chunk silent window of 300ms; heartbeat=40ms ⇒ multiple
|
|
// keepalive frames must arrive BEFORE the first content delta. This
|
|
// is the OCP db11105 invariant: without eager-headers the keepalive
|
|
// writes would buffer (no headers sent yet) and the client wouldn't
|
|
// see anything until the first real chunk.
|
|
let firstDeltaIdx = -1;
|
|
let firstKeepaliveIdx = -1;
|
|
const fake = {
|
|
...savedAnthropic29,
|
|
spawn: async function* (_ir, _ctx) {
|
|
// Long pre-first-chunk silence.
|
|
await new Promise(r => setTimeout(r, 250));
|
|
yield { type: 'delta', content: 'eager-delta' };
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
},
|
|
};
|
|
lp29.set('anthropic', fake);
|
|
__setStreamingConfig({ heartbeat_interval_ms: 40 });
|
|
|
|
const r = await makeTimedStreamRequest({ prompt: 'h29e-prompt' });
|
|
assert.equal(r.status, 200);
|
|
// Headers received (status 200 was already received → headers flushed).
|
|
// Locate index of first arrival containing : keepalive and first arrival
|
|
// containing eager-delta.
|
|
for (let i = 0; i < r.arrivals.length; i++) {
|
|
if (firstKeepaliveIdx === -1 && r.arrivals[i].text.includes(': keepalive')) {
|
|
firstKeepaliveIdx = i;
|
|
}
|
|
if (firstDeltaIdx === -1 && r.arrivals[i].text.includes('eager-delta')) {
|
|
firstDeltaIdx = i;
|
|
}
|
|
}
|
|
assert.ok(firstKeepaliveIdx >= 0, `expected at least one keepalive arrival; arrivals=${JSON.stringify(r.arrivals.map(a => a.text.slice(0, 60)))}`);
|
|
assert.ok(firstDeltaIdx >= 0, 'expected the eager-delta to arrive');
|
|
assert.ok(firstKeepaliveIdx < firstDeltaIdx,
|
|
`keepalive (idx ${firstKeepaliveIdx}) must arrive BEFORE first delta (idx ${firstDeltaIdx}); proves eager-headers`);
|
|
});
|
|
});
|
|
|
|
// ── Suite 30: D62 recentErrors[20] ring buffer (ADR 0010 § Phase 4 D61-D63) ──
|
|
//
|
|
// Tests the in-memory ring of the last 20 server-side error events plus its
|
|
// filter (401/403 excluded so brute-force loops cannot flood the ring) and
|
|
// path-sanitization. Surfaced via /status (Suite 31).
|
|
|
|
describe('Suite 30 — D62 recentErrors[20] ring buffer (ADR 0010 § Phase 4 D61-D63)', () => {
|
|
let server30;
|
|
let port30;
|
|
let savedToken30;
|
|
let lp30;
|
|
let savedAnthropic30;
|
|
|
|
before(async () => {
|
|
savedToken30 = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-suite-30';
|
|
__setProvidersEnabled({ anthropic: true });
|
|
|
|
const mod = await import('./server.mjs');
|
|
lp30 = mod.loadedProviders;
|
|
savedAnthropic30 = lp30.get('anthropic');
|
|
mod.__clearCache();
|
|
|
|
server30 = mod.createOlpServer();
|
|
await new Promise((resolve, reject) => {
|
|
server30.listen(0, '127.0.0.1', resolve);
|
|
server30.once('error', reject);
|
|
});
|
|
port30 = server30.address().port;
|
|
});
|
|
|
|
after(async () => {
|
|
if (savedAnthropic30 !== undefined) {
|
|
lp30.set('anthropic', savedAnthropic30);
|
|
} else {
|
|
lp30.delete('anthropic');
|
|
}
|
|
__resetProvidersEnabled();
|
|
__resetSpawnImpl();
|
|
__clearRecentErrors();
|
|
if (savedToken30 !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken30;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
if (!server30) return;
|
|
return new Promise(r => server30.close(r));
|
|
});
|
|
|
|
beforeEach(async () => {
|
|
const mod = await import('./server.mjs');
|
|
mod.__clearCache();
|
|
__clearRecentErrors();
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
});
|
|
|
|
it('30a — provider error pushes an entry onto the recentErrors ring', async () => {
|
|
// Install a provider that throws SPAWN_FAILED before first chunk.
|
|
const fake = {
|
|
...savedAnthropic30,
|
|
spawn: async function* (_ir, _ctx) {
|
|
// Throw before yielding any chunk → propagates through the streaming
|
|
// branch as a pre-first-chunk error → 502 + _pushError fires.
|
|
throw new (await import('./lib/providers/base.mjs')).ProviderError(
|
|
'fake spawn failure for 30a',
|
|
'SPAWN_FAILED',
|
|
);
|
|
// eslint-disable-next-line no-unreachable
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
},
|
|
};
|
|
lp30.set('anthropic', fake);
|
|
|
|
const before = __snapshotRecentErrors().length;
|
|
const r = await fetch({
|
|
port: port30, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: '30a-trigger' }],
|
|
stream: true,
|
|
},
|
|
});
|
|
// The exact status varies (502 for pre-first-chunk error, or
|
|
// chain-exhausted 502 from the buffered path); both record _pushError.
|
|
assert.ok(r.status === 502 || r.status === 500,
|
|
`expected 5xx error, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
const after = __snapshotRecentErrors();
|
|
assert.ok(after.length > before, `ring should have grown; before=${before} after=${after.length}`);
|
|
const last = after[after.length - 1];
|
|
assert.ok(typeof last.time === 'string' && last.time.includes('T'),
|
|
'entry must have ISO8601 time');
|
|
assert.ok(typeof last.message === 'string' && last.message.length > 0,
|
|
'entry must have a message');
|
|
assert.ok(last.message.length <= 200, 'message must be capped at 200 chars');
|
|
assert.equal(last.path, '/v1/chat/completions', 'entry path captured');
|
|
assert.equal(typeof last.status_code, 'number', 'entry status_code captured');
|
|
});
|
|
|
|
it('30b — ring caps at 20 entries (oldest evicted on push)', async () => {
|
|
__clearRecentErrors();
|
|
// Push 25 synthetic errors via the public seam (we exercise this via a
|
|
// provider that throws repeatedly). Easier: call _pushError directly by
|
|
// hitting an endpoint that maps to a 5xx; install a fake that always
|
|
// throws and fire 25 requests. Each request adds one entry.
|
|
let pushCount = 0;
|
|
const fake = {
|
|
...savedAnthropic30,
|
|
spawn: async function* (_ir, _ctx) {
|
|
pushCount++;
|
|
throw new (await import('./lib/providers/base.mjs')).ProviderError(
|
|
`30b-err-${pushCount}`,
|
|
'SPAWN_FAILED',
|
|
);
|
|
// eslint-disable-next-line no-unreachable
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
},
|
|
};
|
|
lp30.set('anthropic', fake);
|
|
|
|
for (let i = 0; i < 25; i++) {
|
|
await fetch({
|
|
port: port30, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: `30b-iter-${i}` }],
|
|
stream: false,
|
|
},
|
|
});
|
|
}
|
|
const snap = __snapshotRecentErrors();
|
|
assert.equal(snap.length, 20, `ring must cap at 20, got ${snap.length}`);
|
|
// Oldest evicted → the first remaining entry's message should not be
|
|
// 30b-err-1 (which was pushed first and evicted). It should also not
|
|
// include `30b-err-2 .. 30b-err-5` (we pushed 25, last 20 retained →
|
|
// entries 6..25 remain).
|
|
const firstMsg = snap[0].message;
|
|
assert.ok(!firstMsg.includes('30b-err-1') || firstMsg.includes('30b-err-10') || firstMsg.includes('30b-err-11'),
|
|
`oldest 5 must be evicted; first message=${firstMsg}`);
|
|
});
|
|
|
|
it('30c — 401/403 auth failures are NOT pushed onto the ring (brute-force flood guard)', async () => {
|
|
__clearRecentErrors();
|
|
__setAuthConfig({ allow_anonymous: false, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
try {
|
|
const before = __snapshotRecentErrors().length;
|
|
// Fire 5 requests with no auth header → all 401s.
|
|
for (let i = 0; i < 5; i++) {
|
|
const r = await fetch({
|
|
port: port30, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: `30c-${i}` }] },
|
|
});
|
|
assert.equal(r.status, 401, `expected 401 on no-auth request, got ${r.status}`);
|
|
}
|
|
const after = __snapshotRecentErrors().length;
|
|
assert.equal(after, before, '401 responses must NOT push onto recentErrors');
|
|
} finally {
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
}
|
|
});
|
|
|
|
it('30d — path sanitization: filesystem-path-like tokens in message are replaced with [path]', async () => {
|
|
__clearRecentErrors();
|
|
const fake = {
|
|
...savedAnthropic30,
|
|
spawn: async function* (_ir, _ctx) {
|
|
// Error message embeds a filesystem path.
|
|
throw new (await import('./lib/providers/base.mjs')).ProviderError(
|
|
'ENOENT: no such file or directory /Users/private/secret-file.json',
|
|
'SPAWN_FAILED',
|
|
);
|
|
// eslint-disable-next-line no-unreachable
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
},
|
|
};
|
|
lp30.set('anthropic', fake);
|
|
|
|
await fetch({
|
|
port: port30, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: '30d-path-sanitize' }],
|
|
stream: false,
|
|
},
|
|
});
|
|
const snap = __snapshotRecentErrors();
|
|
assert.ok(snap.length >= 1, 'an entry must have been pushed');
|
|
const last = snap[snap.length - 1];
|
|
assert.ok(!last.message.includes('/Users/private/secret-file.json'),
|
|
`path leaked: ${last.message}`);
|
|
assert.ok(last.message.includes('[path]'),
|
|
`sanitization marker [path] expected; got ${last.message}`);
|
|
});
|
|
});
|
|
|
|
// ── Suite 31: D63 /v0/management/status combined endpoint (ADR 0010 § Phase 4) ──
|
|
//
|
|
// Owner-only_block management endpoint that returns process/provider/cache
|
|
// stats + recentErrors. Tests gating + payload shape + recent_errors wiring.
|
|
|
|
describe('Suite 31 — D63 /v0/management/status (ADR 0010 § Phase 4 D61-D63)', () => {
|
|
const GLOBAL_OLP_HOME_31 = process.env.OLP_HOME;
|
|
let TMP31, server31, port31;
|
|
let savedToken31;
|
|
let lp31;
|
|
let savedAnthropic31;
|
|
|
|
before(async () => {
|
|
TMP31 = mkdtempSync(pathJoin(tmpdir(), 'olp-test-31-'));
|
|
process.env.OLP_HOME = TMP31;
|
|
savedToken31 = process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-suite-31';
|
|
__setProvidersEnabled({ anthropic: true });
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
|
|
const mod = await import('./server.mjs');
|
|
lp31 = mod.loadedProviders;
|
|
savedAnthropic31 = lp31.get('anthropic');
|
|
mod.__clearCache();
|
|
mod.__clearRecentErrors();
|
|
mod.__resetRequestCounters();
|
|
|
|
server31 = mod.createOlpServer();
|
|
await new Promise((resolve, reject) => {
|
|
server31.listen(0, '127.0.0.1', resolve);
|
|
server31.once('error', reject);
|
|
});
|
|
port31 = server31.address().port;
|
|
});
|
|
|
|
after(async () => {
|
|
if (savedAnthropic31 !== undefined) {
|
|
lp31.set('anthropic', savedAnthropic31);
|
|
} else {
|
|
lp31.delete('anthropic');
|
|
}
|
|
__resetProvidersEnabled();
|
|
__resetSpawnImpl();
|
|
__clearRecentErrors();
|
|
__resetRequestCounters();
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
if (savedToken31 !== undefined) {
|
|
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken31;
|
|
} else {
|
|
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
|
}
|
|
process.env.OLP_HOME = GLOBAL_OLP_HOME_31;
|
|
if (!server31) return;
|
|
await new Promise(r => server31.close(r));
|
|
rmSync(TMP31, { recursive: true, force: true });
|
|
});
|
|
|
|
beforeEach(async () => {
|
|
const mod = await import('./server.mjs');
|
|
mod.__clearCache();
|
|
});
|
|
|
|
it('31a — owner identity GETs /v0/management/status → 200 + full payload shape', async () => {
|
|
const { plaintext_token } = createKey({
|
|
name: '31a-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP31,
|
|
});
|
|
const r = await fetch({
|
|
port: port31, method: 'GET', path: '/v0/management/status',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200, `expected 200, got ${r.status}: ${r.body.slice(0, 200)}`);
|
|
const body = JSON.parse(r.body);
|
|
// Top-level shape per ADR 0010 § Phase 4 D63 spec.
|
|
assert.equal(body.ok, true);
|
|
assert.equal(typeof body.version, 'string');
|
|
assert.equal(typeof body.uptime_ms, 'number');
|
|
assert.ok(body.uptime_ms >= 0);
|
|
assert.equal(typeof body.uptime_human, 'string');
|
|
assert.match(body.uptime_human, /^\d+h \d+m \d+s$/);
|
|
assert.equal(typeof body.started_at, 'string');
|
|
assert.ok(body.started_at.includes('T'));
|
|
assert.ok(typeof body.providers === 'object' && body.providers !== null);
|
|
assert.equal(typeof body.providers.enabled, 'number');
|
|
assert.equal(typeof body.providers.available, 'number');
|
|
assert.ok(typeof body.providers.status === 'object');
|
|
assert.ok(typeof body.stats === 'object');
|
|
assert.equal(typeof body.stats.total_requests, 'number');
|
|
assert.equal(typeof body.stats.active_requests, 'number');
|
|
assert.ok(typeof body.stats.cache === 'object');
|
|
assert.ok(Array.isArray(body.recent_errors));
|
|
assert.equal(typeof body.generated_at, 'string');
|
|
});
|
|
|
|
it('31b — non-owner (guest) → 401 owner_required', async () => {
|
|
const { plaintext_token } = createKey({
|
|
name: '31b-guest', owner_tier: 'guest', providers_enabled: '*', olpHome: TMP31,
|
|
});
|
|
const r = await fetch({
|
|
port: port31, method: 'GET', path: '/v0/management/status',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 401);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.error.type, 'owner_required');
|
|
});
|
|
|
|
it('31c — allow_anonymous=false + no header → 401 auth_required', async () => {
|
|
__setAuthConfig({ allow_anonymous: false, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
try {
|
|
const r = await fetch({ port: port31, method: 'GET', path: '/v0/management/status' });
|
|
assert.equal(r.status, 401);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.error.type, 'auth_required');
|
|
} finally {
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
}
|
|
});
|
|
|
|
it('31d — payload recent_errors populated after a triggered chain-exhausted error', async () => {
|
|
__clearRecentErrors();
|
|
// Install a provider that throws → triggers _pushError on the buffered path.
|
|
const fake = {
|
|
...savedAnthropic31,
|
|
spawn: async function* (_ir, _ctx) {
|
|
throw new (await import('./lib/providers/base.mjs')).ProviderError(
|
|
'fake spawn failure 31d',
|
|
'SPAWN_FAILED',
|
|
);
|
|
// eslint-disable-next-line no-unreachable
|
|
yield { type: 'stop', finish_reason: 'stop' };
|
|
},
|
|
};
|
|
lp31.set('anthropic', fake);
|
|
|
|
// Fire a non-streaming request → goes through executeWithFallback path
|
|
// → chain-exhausted → _pushError fires.
|
|
const trigger = await fetch({
|
|
port: port31, method: 'POST', path: '/v1/chat/completions',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: {
|
|
model: 'claude-sonnet-4-6',
|
|
messages: [{ role: 'user', content: '31d-trigger' }],
|
|
stream: false,
|
|
},
|
|
});
|
|
assert.ok(trigger.status === 502 || trigger.status === 500,
|
|
`expected 5xx error, got ${trigger.status}`);
|
|
|
|
// Restore real provider so the /status healthCheck doesn't itself throw.
|
|
lp31.set('anthropic', savedAnthropic31);
|
|
|
|
// Owner fetches /status — recent_errors must include our pushed entry.
|
|
const { plaintext_token } = createKey({
|
|
name: '31d-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP31,
|
|
});
|
|
const r = await fetch({
|
|
port: port31, method: 'GET', path: '/v0/management/status',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.ok(Array.isArray(body.recent_errors) && body.recent_errors.length >= 1,
|
|
`recent_errors expected to be populated; got ${JSON.stringify(body.recent_errors)}`);
|
|
const matchingEntry = body.recent_errors.find(e =>
|
|
typeof e.message === 'string' && e.message.includes('fake spawn failure 31d'),
|
|
);
|
|
assert.ok(matchingEntry,
|
|
`expected to find the triggered error in recent_errors; got ${JSON.stringify(body.recent_errors)}`);
|
|
assert.equal(matchingEntry.path, '/v1/chat/completions');
|
|
assert.equal(matchingEntry.provider, 'anthropic');
|
|
// total_requests must have advanced past 0 (we made at least 1 chat request).
|
|
assert.ok(body.stats.total_requests >= 1,
|
|
`total_requests expected >= 1, got ${body.stats.total_requests}`);
|
|
});
|
|
});
|
|
|
|
// ── Suite 32: D64 bin/olp.mjs CLI scaffold (ADR 0010 § Phase 4 D64-D67) ───
|
|
//
|
|
// Smoke tests for the operator CLI. The CLI is invoked in-process via its
|
|
// exported runCli() so we get exit codes without spawning a subprocess. The
|
|
// process.env is mutated per-test (saved + restored in before/after) so the
|
|
// runtime resolution paths (OLP_PROXY_URL / OLP_PORT / OLP_API_KEY / OLP_HOME)
|
|
// behave deterministically.
|
|
//
|
|
// The CLI sources are:
|
|
// bin/olp.mjs — runCli(argv, { out, err, useColor }) → exit code
|
|
// lib/doctor.mjs — runDoctor() consumed by `olp doctor`
|
|
//
|
|
// These tests do NOT spawn a real OLP server for the HTTP subcommands; instead
|
|
// each HTTP test starts an ephemeral createOlpServer() listening on port 0 and
|
|
// points the CLI at that port via --proxy-url. The owner_token used is created
|
|
// per-test via createKey({ owner_tier: 'owner' }).
|
|
|
|
import { runCli as runOlpCli, parseArgv as parseOlpArgv, resolveBearerToken as olpResolveBearerToken } from './bin/olp.mjs';
|
|
|
|
describe('Suite 32 — D64 bin/olp.mjs CLI scaffold (ADR 0010 § Phase 4 D64-D67)', () => {
|
|
const SAVED_ENV_32 = {
|
|
OLP_HOME: process.env.OLP_HOME,
|
|
OLP_PROXY_URL: process.env.OLP_PROXY_URL,
|
|
OLP_PORT: process.env.OLP_PORT,
|
|
OLP_API_KEY: process.env.OLP_API_KEY,
|
|
OLP_OWNER_TOKEN: process.env.OLP_OWNER_TOKEN,
|
|
};
|
|
let TMP32;
|
|
|
|
before(() => {
|
|
TMP32 = mkdtempSync(pathJoin(tmpdir(), 'olp-test-32-'));
|
|
process.env.OLP_HOME = TMP32;
|
|
// Clear conflicting env vars for deterministic resolution
|
|
delete process.env.OLP_PROXY_URL;
|
|
delete process.env.OLP_PORT;
|
|
delete process.env.OLP_API_KEY;
|
|
delete process.env.OLP_OWNER_TOKEN;
|
|
});
|
|
|
|
after(() => {
|
|
for (const [k, v] of Object.entries(SAVED_ENV_32)) {
|
|
if (v === undefined) delete process.env[k];
|
|
else process.env[k] = v;
|
|
}
|
|
rmSync(TMP32, { recursive: true, force: true });
|
|
});
|
|
|
|
// Helper: capture stdout / stderr buffers via IO writers
|
|
function makeCapture() {
|
|
let out = '', err = '';
|
|
return {
|
|
out: s => { out += s; },
|
|
err: s => { err += s; },
|
|
get stdout() { return out; },
|
|
get stderr() { return err; },
|
|
};
|
|
}
|
|
|
|
it('32a — parseArgv: --flag=value + --flag value + bare --flag', () => {
|
|
const a = parseOlpArgv(['--json', '--port=4567', '--name', 'foo', 'positional']);
|
|
assert.equal(a.flags.json, true);
|
|
assert.equal(a.flags.port, '4567');
|
|
assert.equal(a.flags.name, 'foo');
|
|
assert.deepEqual(a.positional, ['positional']);
|
|
});
|
|
|
|
it('32b — `olp` (no args) prints USAGE and returns exit code 1', async () => {
|
|
const cap = makeCapture();
|
|
const code = await runOlpCli([], { out: cap.out, err: cap.err, useColor: false });
|
|
assert.equal(code, 1, 'no-args should return 1');
|
|
assert.match(cap.stdout, /OLP operator CLI/);
|
|
assert.match(cap.stdout, /Subcommands:/);
|
|
});
|
|
|
|
it('32c — `olp help` prints USAGE and returns exit code 0', async () => {
|
|
const cap = makeCapture();
|
|
const code = await runOlpCli(['help'], { out: cap.out, err: cap.err, useColor: false });
|
|
assert.equal(code, 0);
|
|
assert.match(cap.stdout, /OLP operator CLI/);
|
|
});
|
|
|
|
it('32d — `olp unknown` prints error + USAGE on stderr, returns exit code 1', async () => {
|
|
const cap = makeCapture();
|
|
const code = await runOlpCli(['frobnicate'], { out: cap.out, err: cap.err, useColor: false });
|
|
assert.equal(code, 1);
|
|
assert.match(cap.stderr, /unknown subcommand "frobnicate"/);
|
|
assert.match(cap.stderr, /OLP operator CLI/);
|
|
});
|
|
|
|
it('32e — `olp providers` (local, no HTTP) → lists registry entries', async () => {
|
|
const cap = makeCapture();
|
|
const code = await runOlpCli(['providers'], { out: cap.out, err: cap.err, useColor: false });
|
|
assert.equal(code, 0);
|
|
// All three shipped plugin keys must appear
|
|
assert.match(cap.stdout, /anthropic/);
|
|
assert.match(cap.stdout, /openai/);
|
|
assert.match(cap.stdout, /mistral/);
|
|
// No providers enabled in the empty TMP32 config → all show disabled
|
|
assert.match(cap.stdout, /disabled/);
|
|
});
|
|
|
|
it('32f — `olp providers --json` emits parseable JSON with providers[]', async () => {
|
|
const cap = makeCapture();
|
|
const code = await runOlpCli(['providers', '--json'], { out: cap.out, err: cap.err, useColor: false });
|
|
assert.equal(code, 0);
|
|
const body = JSON.parse(cap.stdout);
|
|
assert.ok(Array.isArray(body.providers));
|
|
assert.ok(body.providers.length >= 3);
|
|
const names = body.providers.map(p => p.name);
|
|
assert.ok(names.includes('anthropic'));
|
|
assert.ok(names.includes('openai'));
|
|
assert.ok(names.includes('mistral'));
|
|
});
|
|
|
|
it('32g — `olp chain show` with no chains in config prints "no chains configured"', async () => {
|
|
const cap = makeCapture();
|
|
const code = await runOlpCli(['chain', 'show'], { out: cap.out, err: cap.err, useColor: false });
|
|
assert.equal(code, 0);
|
|
assert.match(cap.stdout, /no chains configured/);
|
|
});
|
|
|
|
it('32h — `olp status` against an ephemeral OLP server with owner token → 200', async () => {
|
|
// Spin up an ephemeral server bound to a random port; CLI talks to it.
|
|
__setProvidersEnabled({});
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
const mod = await import('./server.mjs');
|
|
mod.__clearCache();
|
|
mod.__clearRecentErrors();
|
|
mod.__resetRequestCounters();
|
|
|
|
const server32 = mod.createOlpServer();
|
|
await new Promise((resolve, reject) => {
|
|
server32.listen(0, '127.0.0.1', resolve);
|
|
server32.once('error', reject);
|
|
});
|
|
const port32 = server32.address().port;
|
|
try {
|
|
// Create owner key + use it via OLP_API_KEY env (resolveBearerToken reads it first)
|
|
const { plaintext_token } = createKey({
|
|
name: '32h-owner', owner_tier: 'owner', providers_enabled: '*', olpHome: TMP32,
|
|
});
|
|
const SAVED_TOK = process.env.OLP_API_KEY;
|
|
process.env.OLP_API_KEY = plaintext_token;
|
|
try {
|
|
const cap = makeCapture();
|
|
const code = await runOlpCli(
|
|
['status', `--proxy-url=http://127.0.0.1:${port32}`, '--json'],
|
|
{ out: cap.out, err: cap.err, useColor: false },
|
|
);
|
|
assert.equal(code, 0, `status JSON exit non-zero; stderr=${cap.stderr}`);
|
|
const body = JSON.parse(cap.stdout);
|
|
assert.equal(body.ok, true);
|
|
assert.ok(typeof body.version === 'string');
|
|
assert.ok(typeof body.uptime_ms === 'number');
|
|
} finally {
|
|
if (SAVED_TOK === undefined) delete process.env.OLP_API_KEY;
|
|
else process.env.OLP_API_KEY = SAVED_TOK;
|
|
}
|
|
} finally {
|
|
await new Promise(r => server32.close(r));
|
|
__resetProvidersEnabled();
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
}
|
|
});
|
|
|
|
it('32i — `olp status` against a non-existent port → exit 2 ECONNREFUSED', async () => {
|
|
// Pick a random high port unlikely to be in use; the CLI must detect
|
|
// ECONNREFUSED and surface the helpful hint.
|
|
const cap = makeCapture();
|
|
const code = await runOlpCli(
|
|
['status', '--proxy-url=http://127.0.0.1:1'], // port 1 → ECONNREFUSED on macOS/Linux
|
|
{ out: cap.out, err: cap.err, useColor: false },
|
|
);
|
|
assert.equal(code, 2, `expected ECONNREFUSED → exit 2, got ${code}; stderr=${cap.stderr}`);
|
|
assert.match(cap.stderr, /unreachable|ECONNREFUSED/);
|
|
});
|
|
|
|
it('32j — resolveBearerToken precedence: OLP_API_KEY > OLP_OWNER_TOKEN > null', () => {
|
|
const SAVED_API = process.env.OLP_API_KEY;
|
|
const SAVED_OWN = process.env.OLP_OWNER_TOKEN;
|
|
try {
|
|
// 1. Neither set → null
|
|
delete process.env.OLP_API_KEY;
|
|
delete process.env.OLP_OWNER_TOKEN;
|
|
assert.equal(olpResolveBearerToken(), null);
|
|
|
|
// 2. Only OLP_OWNER_TOKEN → that wins
|
|
process.env.OLP_OWNER_TOKEN = 'owner-tok-32j';
|
|
assert.equal(olpResolveBearerToken(), 'owner-tok-32j');
|
|
|
|
// 3. OLP_API_KEY also set → API key wins
|
|
process.env.OLP_API_KEY = 'api-key-32j';
|
|
assert.equal(olpResolveBearerToken(), 'api-key-32j');
|
|
} finally {
|
|
if (SAVED_API === undefined) delete process.env.OLP_API_KEY;
|
|
else process.env.OLP_API_KEY = SAVED_API;
|
|
if (SAVED_OWN === undefined) delete process.env.OLP_OWNER_TOKEN;
|
|
else process.env.OLP_OWNER_TOKEN = SAVED_OWN;
|
|
}
|
|
});
|
|
});
|
|
|
|
// ── Suite 33: D65 lib/doctor.mjs framework (ADR 0010 § Phase 4 D64-D67) ───
|
|
//
|
|
// Framework tests for `olp doctor`. Uses runDoctor()'s opts.injectChecks
|
|
// hook to drive every kind branch (noop / fresh_install / fix_server /
|
|
// fix_oauth / fix_provider / fix_config) deterministically without depending
|
|
// on real filesystem or network state.
|
|
|
|
import {
|
|
runDoctor as runOlpDoctor,
|
|
buildBuiltinChecks as buildOlpBuiltinChecks,
|
|
collectProviderChecks as collectOlpProviderChecks,
|
|
deriveKind as deriveOlpKind,
|
|
deriveNextAction as deriveOlpNextAction,
|
|
resolveProxyUrl as resolveOlpProxyUrl,
|
|
DOCTOR_SCHEMA_VERSION,
|
|
} from './lib/doctor.mjs';
|
|
|
|
describe('Suite 33 — D65 lib/doctor.mjs framework (ADR 0010 § Phase 4 D64-D67)', () => {
|
|
|
|
// Helper: build a fake check with deterministic status + optional evidence
|
|
function fakeCheck(id, category, status, message = 'fake', evidence) {
|
|
return {
|
|
id,
|
|
category,
|
|
async run() {
|
|
return evidence ? { status, message, evidence } : { status, message };
|
|
},
|
|
};
|
|
}
|
|
|
|
it('33a — all-ok checks → kind=noop, next_action.ai_executable=[]', async () => {
|
|
const result = await runOlpDoctor({
|
|
injectChecks: [
|
|
fakeCheck('system.node_version', 'system', 'ok', 'Node 20'),
|
|
fakeCheck('server.running', 'server', 'ok', 'live'),
|
|
fakeCheck('auth.owner_key_exists', 'auth', 'ok', 'present'),
|
|
],
|
|
});
|
|
assert.equal(result.schema_version, DOCTOR_SCHEMA_VERSION);
|
|
assert.equal(result.fail_count, 0);
|
|
assert.equal(result.kind, 'noop');
|
|
assert.deepEqual(result.next_action.ai_executable, []);
|
|
assert.deepEqual(result.next_action.human_required, []);
|
|
assert.equal(result.next_action.verify, 'already healthy');
|
|
assert.match(result.summary, /all 3 checks ok/);
|
|
});
|
|
|
|
it('33b — server.running FAIL → kind=fix_server + ai_executable propagates', async () => {
|
|
const result = await runOlpDoctor({
|
|
injectChecks: [
|
|
fakeCheck('server.running', 'server', 'fail', 'unreachable', {
|
|
fix_commands: ['olp restart'],
|
|
}),
|
|
fakeCheck('auth.owner_key_exists', 'auth', 'ok', 'present'),
|
|
],
|
|
});
|
|
assert.equal(result.fail_count, 1);
|
|
assert.equal(result.kind, 'fix_server');
|
|
assert.deepEqual(result.next_action.ai_executable, ['olp restart']);
|
|
assert.equal(result.next_action.verify, 'olp doctor');
|
|
});
|
|
|
|
it('33c — auth.owner_key_exists FAIL (no server fail) → kind=fix_oauth', async () => {
|
|
const result = await runOlpDoctor({
|
|
injectChecks: [
|
|
fakeCheck('server.running', 'server', 'ok', 'live'),
|
|
fakeCheck('auth.owner_key_exists', 'auth', 'fail', 'no owner', {
|
|
fix_commands: ['npx olp-keys keygen --owner'],
|
|
}),
|
|
],
|
|
});
|
|
assert.equal(result.kind, 'fix_oauth');
|
|
assert.deepEqual(result.next_action.ai_executable, ['npx olp-keys keygen --owner']);
|
|
});
|
|
|
|
it('33d — config.exists FAIL beats everything else → kind=fresh_install', async () => {
|
|
const result = await runOlpDoctor({
|
|
injectChecks: [
|
|
fakeCheck('config.exists', 'config', 'fail', 'missing', {
|
|
fix_commands: ['mkdir -p ~/.olp'],
|
|
}),
|
|
fakeCheck('server.running', 'server', 'fail', 'unreachable', {
|
|
fix_commands: ['olp restart'],
|
|
}),
|
|
fakeCheck('auth.owner_key_exists', 'auth', 'fail', 'no owner'),
|
|
],
|
|
});
|
|
assert.equal(result.kind, 'fresh_install');
|
|
// ai_executable concatenates all FAIL evidence.fix_commands
|
|
assert.ok(result.next_action.ai_executable.includes('mkdir -p ~/.olp'));
|
|
assert.ok(result.next_action.ai_executable.includes('olp restart'));
|
|
});
|
|
|
|
it('33e — provider-category FAIL only → kind=fix_provider, human_steps propagate', async () => {
|
|
const result = await runOlpDoctor({
|
|
injectChecks: [
|
|
fakeCheck('server.running', 'server', 'ok', 'live'),
|
|
fakeCheck('auth.owner_key_exists', 'auth', 'ok', 'present'),
|
|
fakeCheck('anthropic.oauth_token_present', 'provider', 'fail', 'missing token', {
|
|
human_steps: ['run: claude (browser OAuth)'],
|
|
}),
|
|
],
|
|
});
|
|
assert.equal(result.kind, 'fix_provider');
|
|
assert.deepEqual(result.next_action.ai_executable, []);
|
|
assert.deepEqual(result.next_action.human_required, ['run: claude (browser OAuth)']);
|
|
});
|
|
|
|
it('33f — collectProviderChecks reads plugin doctorChecks() (ADR 0002 Amendment 7)', () => {
|
|
// Synthetic plugin map that includes one plugin WITH doctorChecks() + one
|
|
// plugin WITHOUT (default behaviour — back-compat — contributes nothing).
|
|
const providers = new Map();
|
|
providers.set('alpha', {
|
|
name: 'alpha',
|
|
doctorChecks() {
|
|
return [
|
|
{ id: 'alpha.cli_available', category: 'provider', async run() { return { status: 'ok', message: 'alpha bin ok' }; } },
|
|
{ id: 'alpha.auth_present', category: 'provider', async run() { return { status: 'fail', message: 'alpha missing' }; } },
|
|
];
|
|
},
|
|
});
|
|
providers.set('legacy', {
|
|
name: 'legacy',
|
|
// No doctorChecks() — plugin pre-amendment 7; collectProviderChecks ignores it
|
|
});
|
|
const checks = collectOlpProviderChecks({ providersOverride: providers });
|
|
const ids = checks.map(c => c.id).sort();
|
|
assert.deepEqual(ids, ['alpha.auth_present', 'alpha.cli_available']);
|
|
// Categories normalized to 'provider'
|
|
assert.ok(checks.every(c => c.category === 'provider'));
|
|
});
|
|
|
|
it('33g — plugin doctorChecks() that throws is captured (does not crash framework)', async () => {
|
|
const providers = new Map();
|
|
providers.set('bad', {
|
|
name: 'bad',
|
|
doctorChecks() { throw new Error('boom from bad plugin'); },
|
|
});
|
|
const checks = collectOlpProviderChecks({ providersOverride: providers });
|
|
assert.equal(checks.length, 1);
|
|
assert.equal(checks[0].id, 'bad.doctor_checks_threw');
|
|
// Run it — should resolve to fail without re-throwing
|
|
const r = await checks[0].run();
|
|
assert.equal(r.status, 'fail');
|
|
assert.match(r.message, /boom from bad plugin/);
|
|
});
|
|
|
|
it('33h — --check filter restricts to matching id/category/prefix', async () => {
|
|
// Filter by category
|
|
const r1 = await runOlpDoctor({
|
|
injectChecks: [
|
|
fakeCheck('a.x', 'provider', 'ok'),
|
|
fakeCheck('b.y', 'system', 'ok'),
|
|
],
|
|
checkFilter: 'system',
|
|
});
|
|
assert.equal(r1.checks.length, 1);
|
|
assert.equal(r1.checks[0].id, 'b.y');
|
|
|
|
// Filter by id prefix
|
|
const r2 = await runOlpDoctor({
|
|
injectChecks: [
|
|
fakeCheck('anthropic.cli_available', 'provider', 'ok'),
|
|
fakeCheck('anthropic.oauth_token_present', 'provider', 'ok'),
|
|
fakeCheck('openai.cli_available', 'provider', 'ok'),
|
|
],
|
|
checkFilter: 'anthropic',
|
|
});
|
|
assert.equal(r2.checks.length, 2);
|
|
assert.ok(r2.checks.every(c => c.id.startsWith('anthropic.')));
|
|
|
|
// Filter by exact id
|
|
const r3 = await runOlpDoctor({
|
|
injectChecks: [
|
|
fakeCheck('anthropic.cli_available', 'provider', 'ok'),
|
|
fakeCheck('anthropic.oauth_token_present', 'provider', 'ok'),
|
|
],
|
|
checkFilter: 'anthropic.cli_available',
|
|
});
|
|
assert.equal(r3.checks.length, 1);
|
|
assert.equal(r3.checks[0].id, 'anthropic.cli_available');
|
|
});
|
|
|
|
it('33i — built-in checks run against a temp OLP_HOME (no config → fresh_install)', async () => {
|
|
// skipNetwork: true so server.* checks don't run against the localhost port.
|
|
const tmpEmpty = mkdtempSync(pathJoin(tmpdir(), 'olp-doc-33i-'));
|
|
try {
|
|
const result = await runOlpDoctor({
|
|
olpHome: tmpEmpty,
|
|
skipNetwork: true,
|
|
// Don't load real provider plugins for this run; pass an empty Map.
|
|
providersOverride: new Map(),
|
|
});
|
|
assert.equal(result.schema_version, DOCTOR_SCHEMA_VERSION);
|
|
// config.exists must FAIL (no ~/.olp/config.json in tmpEmpty)
|
|
const configCheck = result.checks.find(c => c.id === 'config.exists');
|
|
assert.ok(configCheck, 'config.exists check should run');
|
|
assert.equal(configCheck.status, 'fail');
|
|
// kind must be fresh_install per the precedence rule
|
|
assert.equal(result.kind, 'fresh_install');
|
|
// ai_executable must include the mkdir + printf seed bootstrap
|
|
assert.ok(result.next_action.ai_executable.some(c => c.startsWith('mkdir -p')));
|
|
} finally {
|
|
rmSync(tmpEmpty, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('33j — anthropic plugin doctorChecks() returns the documented probe set', () => {
|
|
// Plugin contract: ADR 0002 Amendment 7. anthropic.mjs ships
|
|
// anthropic.cli_available + anthropic.oauth_token_present.
|
|
const checks = anthropic.doctorChecks();
|
|
assert.ok(Array.isArray(checks));
|
|
const ids = checks.map(c => c.id).sort();
|
|
assert.deepEqual(ids, ['anthropic.cli_available', 'anthropic.oauth_token_present']);
|
|
assert.ok(checks.every(c => c.category === 'provider'));
|
|
assert.ok(checks.every(c => typeof c.run === 'function'));
|
|
});
|
|
|
|
it('33k — resolveProxyUrl: explicit > env OLP_PROXY_URL > OLP_PORT > default', () => {
|
|
const SAVED_URL = process.env.OLP_PROXY_URL;
|
|
const SAVED_PORT = process.env.OLP_PORT;
|
|
try {
|
|
delete process.env.OLP_PROXY_URL;
|
|
delete process.env.OLP_PORT;
|
|
// 1. default = http://127.0.0.1:4567
|
|
assert.equal(resolveOlpProxyUrl(), 'http://127.0.0.1:4567');
|
|
// 2. OLP_PORT env
|
|
process.env.OLP_PORT = '9999';
|
|
assert.equal(resolveOlpProxyUrl(), 'http://127.0.0.1:9999');
|
|
// 3. OLP_PROXY_URL wins over OLP_PORT
|
|
process.env.OLP_PROXY_URL = 'https://example.com:8443';
|
|
assert.equal(resolveOlpProxyUrl(), 'https://example.com:8443');
|
|
// 4. Explicit opts wins over everything
|
|
assert.equal(resolveOlpProxyUrl({ proxyUrl: 'http://override:1234' }), 'http://override:1234');
|
|
} finally {
|
|
if (SAVED_URL === undefined) delete process.env.OLP_PROXY_URL;
|
|
else process.env.OLP_PROXY_URL = SAVED_URL;
|
|
if (SAVED_PORT === undefined) delete process.env.OLP_PORT;
|
|
else process.env.OLP_PORT = SAVED_PORT;
|
|
}
|
|
});
|
|
|
|
it('33l — deriveKind / deriveNextAction pure-function shape', () => {
|
|
// deriveKind unit
|
|
assert.equal(deriveOlpKind([{ status: 'ok', category: 'server', id: 'a' }]), 'noop');
|
|
assert.equal(deriveOlpKind([
|
|
{ status: 'fail', category: 'server', id: 's.r' },
|
|
{ status: 'fail', category: 'auth', id: 'a.o' },
|
|
]), 'fix_server'); // server beats auth
|
|
|
|
// deriveNextAction aggregates evidence
|
|
const na = deriveOlpNextAction([
|
|
{ status: 'fail', id: 'x', category: 'provider', evidence: { fix_commands: ['cmd1'], human_steps: ['step1'] } },
|
|
{ status: 'fail', id: 'y', category: 'provider', evidence: { fix_commands: ['cmd2'] } },
|
|
{ status: 'ok', id: 'z', category: 'provider' },
|
|
], 'fix_provider');
|
|
assert.deepEqual(na.ai_executable, ['cmd1', 'cmd2']);
|
|
assert.deepEqual(na.human_required, ['step1']);
|
|
assert.equal(na.verify, 'olp doctor');
|
|
});
|
|
});
|
|
|
|
// ── Suite 34: D68-D70 — /health.anonymousKey + plaintext_advertise (ADR 0011) ──
|
|
//
|
|
// Tests for:
|
|
// - lib/keys.mjs createKey({ plaintext_advertise: true }) writes the field;
|
|
// rejects on owner tier
|
|
// - lib/keys.mjs findAdvertisedKey() discovery semantics (skips revoked /
|
|
// skips no-advertise / returns first match)
|
|
// - server.mjs /health emits anonymousKey ONLY when all 3 prerequisites hold
|
|
// - bin/olp-keys.mjs `--anonymous --advertise` end-to-end (manifest +
|
|
// plaintext printed + stderr warning + advertise label)
|
|
// - bin/olp-keys.mjs rejects `--owner --advertise`
|
|
|
|
import {
|
|
findAdvertisedKey as findAdvertisedKey34,
|
|
} from './lib/keys.mjs';
|
|
|
|
describe('Suite 34 — D68-D70 /health.anonymousKey + plaintext_advertise (ADR 0011)', () => {
|
|
const SUITE34_GLOBAL_OLP_HOME = process.env.OLP_HOME;
|
|
|
|
// ── 34a-c: lib/keys.mjs unit tests ──────────────────────────────────────
|
|
|
|
describe('34a-c — lib/keys.mjs createKey + findAdvertisedKey', () => {
|
|
let TMP;
|
|
before(() => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-34abc-'));
|
|
});
|
|
after(() => {
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
__resetWriteLocks();
|
|
});
|
|
|
|
it('34a — createKey({ plaintext_advertise: true }) writes plaintext_advertise field on guest manifest', () => {
|
|
const { id, plaintext_token, manifest } = createKey({
|
|
name: '34a-advert', owner_tier: 'guest', providers_enabled: '*',
|
|
plaintext_advertise: true, olpHome: TMP,
|
|
});
|
|
assert.equal(manifest.plaintext_advertise, plaintext_token);
|
|
// Verify on-disk
|
|
const reread = readManifest(id, { olpHome: TMP });
|
|
assert.equal(reread.plaintext_advertise, plaintext_token);
|
|
// Token hash also correct
|
|
assert.equal(reread.token_hash, hashToken(plaintext_token));
|
|
});
|
|
|
|
it('34a-2 — createKey() default (no plaintext_advertise) does NOT write the field (ADR 0007 § 5 invariant preserved)', () => {
|
|
const { id, manifest } = createKey({
|
|
name: '34a2-normal', owner_tier: 'guest', providers_enabled: '*', olpHome: TMP,
|
|
});
|
|
assert.ok(!('plaintext_advertise' in manifest), 'manifest must NOT carry plaintext_advertise by default');
|
|
const reread = readManifest(id, { olpHome: TMP });
|
|
assert.ok(!('plaintext_advertise' in reread), 'on-disk manifest must NOT carry plaintext_advertise by default');
|
|
});
|
|
|
|
it('34b — createKey({ owner_tier:"owner", plaintext_advertise:true }) is rejected (ADR 0011 tier restriction)', () => {
|
|
assert.throws(
|
|
() => createKey({
|
|
name: '34b-fail', owner_tier: 'owner', plaintext_advertise: true, olpHome: TMP,
|
|
}),
|
|
/plaintext_advertise requires owner_tier="guest"/,
|
|
);
|
|
});
|
|
|
|
it('34c-1 — findAdvertisedKey() returns null when no key has plaintext_advertise', () => {
|
|
const TMP2 = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-34c1-'));
|
|
try {
|
|
createKey({ name: 'plain', owner_tier: 'guest', olpHome: TMP2 });
|
|
assert.equal(findAdvertisedKey34({ olpHome: TMP2 }), null);
|
|
} finally {
|
|
rmSync(TMP2, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('34c-2 — findAdvertisedKey() returns the advertised manifest with plaintext_advertise field intact', () => {
|
|
const TMP3 = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-34c2-'));
|
|
try {
|
|
const { plaintext_token } = createKey({
|
|
name: '34c2', owner_tier: 'guest', plaintext_advertise: true, olpHome: TMP3,
|
|
});
|
|
const found = findAdvertisedKey34({ olpHome: TMP3 });
|
|
assert.ok(found !== null, 'findAdvertisedKey must return non-null');
|
|
assert.equal(found.plaintext_advertise, plaintext_token);
|
|
assert.equal(found.owner_tier, 'guest');
|
|
} finally {
|
|
rmSync(TMP3, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('34c-3 — findAdvertisedKey() skips revoked advertised keys', async () => {
|
|
const TMP4 = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-34c3-'));
|
|
try {
|
|
const { id } = createKey({
|
|
name: '34c3', owner_tier: 'guest', plaintext_advertise: true, olpHome: TMP4,
|
|
});
|
|
assert.ok(findAdvertisedKey34({ olpHome: TMP4 }) !== null, 'pre-revoke: must find it');
|
|
await revokeKey({ id, olpHome: TMP4 });
|
|
assert.equal(findAdvertisedKey34({ olpHome: TMP4 }), null, 'post-revoke: must skip');
|
|
} finally {
|
|
rmSync(TMP4, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|
|
|
|
// ── 34d-g: /health.anonymousKey HTTP integration (D69 prereqs) ──────────
|
|
|
|
describe('34d-g — /health.anonymousKey HTTP integration', () => {
|
|
let TMP, server, port;
|
|
|
|
async function makeSuite34Server() {
|
|
__setProvidersEnabled({});
|
|
const srv = createOlpServer();
|
|
return new Promise(resolve => {
|
|
srv.listen(0, '127.0.0.1', () => resolve({ server: srv, port: srv.address().port }));
|
|
});
|
|
}
|
|
function teardownSuite34() {
|
|
return new Promise(resolve => {
|
|
__setProvidersEnabled({});
|
|
__clearCache();
|
|
if (server) server.close(() => resolve());
|
|
else resolve();
|
|
});
|
|
}
|
|
|
|
before(async () => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-34dg-'));
|
|
process.env.OLP_HOME = TMP;
|
|
({ server, port } = await makeSuite34Server());
|
|
});
|
|
|
|
after(async () => {
|
|
await teardownSuite34();
|
|
process.env.OLP_HOME = SUITE34_GLOBAL_OLP_HOME;
|
|
// Restore test-global auth config so subsequent suites are unaffected
|
|
__setAuthConfig({ allow_anonymous: true, owner_only_endpoints: [], fallback_detail_header_policy: 'all' });
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
});
|
|
|
|
beforeEach(() => {
|
|
// Clean slate between cases — each test installs the precise auth config
|
|
// it needs (default-off / advertise-but-no-anon / advertise-but-no-key /
|
|
// all three prerequisites met) before issuing the /health request.
|
|
__setAuthConfig({
|
|
allow_anonymous: false,
|
|
owner_only_endpoints: ['/health'],
|
|
fallback_detail_header_policy: 'owner_only',
|
|
advertise_anonymous_key: false,
|
|
});
|
|
});
|
|
|
|
it('34d — default (advertise_anonymous_key: false) — /health response has NO anonymousKey field', async () => {
|
|
// Create an advertised key on disk but don't enable the flag.
|
|
createKey({
|
|
name: '34d-key', owner_tier: 'guest', plaintext_advertise: true, olpHome: TMP,
|
|
});
|
|
// Owner identity to access full payload (since allow_anonymous: false)
|
|
const { plaintext_token } = createKey({
|
|
name: '34d-owner', owner_tier: 'owner', olpHome: TMP,
|
|
});
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/health',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.ok(!('anonymousKey' in body), '/health MUST NOT include anonymousKey when advertise_anonymous_key=false');
|
|
});
|
|
|
|
it('34e — advertise_anonymous_key: true + allow_anonymous: false → field omitted (prereq 2 fails)', async () => {
|
|
createKey({
|
|
name: '34e-key', owner_tier: 'guest', plaintext_advertise: true, olpHome: TMP,
|
|
});
|
|
__setAuthConfig({
|
|
allow_anonymous: false, // prereq 2 NOT satisfied
|
|
advertise_anonymous_key: true,
|
|
owner_only_endpoints: ['/health'],
|
|
fallback_detail_header_policy: 'owner_only',
|
|
});
|
|
// Use owner identity (must — allow_anonymous=false rejects anonymous)
|
|
const { plaintext_token } = createKey({
|
|
name: '34e-owner', owner_tier: 'owner', olpHome: TMP,
|
|
});
|
|
const r = await fetch({
|
|
port, method: 'GET', path: '/health',
|
|
headers: { Authorization: `Bearer ${plaintext_token}` },
|
|
});
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.ok(!('anonymousKey' in body), '/health MUST NOT include anonymousKey when allow_anonymous=false even if advertise=true');
|
|
});
|
|
|
|
it('34f — advertise_anonymous_key: true + allow_anonymous: true + NO advertised key → field omitted (prereq 3 fails)', async () => {
|
|
// Fresh isolated tmpdir so no advertised key exists
|
|
const TMP_F = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-34f-'));
|
|
const SAVED = process.env.OLP_HOME;
|
|
process.env.OLP_HOME = TMP_F;
|
|
__setAuthConfig({
|
|
allow_anonymous: true,
|
|
advertise_anonymous_key: true,
|
|
owner_only_endpoints: [], // /health full payload to anonymous
|
|
fallback_detail_header_policy: 'all',
|
|
});
|
|
try {
|
|
const r = await fetch({ port, method: 'GET', path: '/health' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.ok(!('anonymousKey' in body), '/health MUST NOT include anonymousKey when no plaintext_advertise key exists');
|
|
} finally {
|
|
process.env.OLP_HOME = SAVED;
|
|
rmSync(TMP_F, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('34g — all 3 prerequisites met → /health.anonymousKey exposes the plaintext token', async () => {
|
|
const TMP_G = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-34g-'));
|
|
const SAVED = process.env.OLP_HOME;
|
|
process.env.OLP_HOME = TMP_G;
|
|
try {
|
|
const { plaintext_token } = createKey({
|
|
name: '34g-anon', owner_tier: 'guest', plaintext_advertise: true, olpHome: TMP_G,
|
|
});
|
|
__setAuthConfig({
|
|
allow_anonymous: true,
|
|
advertise_anonymous_key: true,
|
|
owner_only_endpoints: [], // anonymous sees full payload too
|
|
fallback_detail_header_policy: 'all',
|
|
});
|
|
const r = await fetch({ port, method: 'GET', path: '/health' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
assert.equal(body.anonymousKey, plaintext_token);
|
|
assert.match(body.anonymousKey, /^olp_[A-Za-z0-9_-]{43}$/);
|
|
} finally {
|
|
process.env.OLP_HOME = SAVED;
|
|
rmSync(TMP_G, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('34g-trimmed — anonymousKey also surfaces in trimmed /health (anonymous client zero-config path)', async () => {
|
|
// /health is gated as owner-only; anonymous client gets trimmed payload
|
|
// but anonymousKey must still surface — it's the whole point of D69.
|
|
const TMP_H = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-34gt-'));
|
|
const SAVED = process.env.OLP_HOME;
|
|
process.env.OLP_HOME = TMP_H;
|
|
try {
|
|
const { plaintext_token } = createKey({
|
|
name: '34gt-anon', owner_tier: 'guest', plaintext_advertise: true, olpHome: TMP_H,
|
|
});
|
|
__setAuthConfig({
|
|
allow_anonymous: true,
|
|
advertise_anonymous_key: true,
|
|
owner_only_endpoints: ['/health'], // trim gating ACTIVE
|
|
fallback_detail_header_policy: 'owner_only',
|
|
});
|
|
const r = await fetch({ port, method: 'GET', path: '/health' });
|
|
assert.equal(r.status, 200);
|
|
const body = JSON.parse(r.body);
|
|
// Trimmed shape (no providers) BUT anonymousKey is present
|
|
assert.ok(!('providers' in body), 'trimmed /health must omit providers');
|
|
assert.equal(body.anonymousKey, plaintext_token);
|
|
} finally {
|
|
process.env.OLP_HOME = SAVED;
|
|
rmSync(TMP_H, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|
|
|
|
// ── 34h-j: bin/olp-keys.mjs --anonymous --advertise CLI ─────────────────
|
|
|
|
describe('34h-j — bin/olp-keys.mjs --anonymous --advertise', () => {
|
|
let TMP;
|
|
before(() => {
|
|
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-34hj-'));
|
|
});
|
|
after(() => {
|
|
rmSync(TMP, { recursive: true, force: true });
|
|
__resetWriteLocks();
|
|
});
|
|
|
|
it('34h — keygen --anonymous --advertise creates guest+advertise key + prints WARNING to stderr', async () => {
|
|
let out = '';
|
|
let err = '';
|
|
const code = await runOlpKeysCli(
|
|
['keygen', '--anonymous', '--advertise', '--olp-home', TMP],
|
|
{ out: s => { out += s; }, err: s => { err += s; } },
|
|
);
|
|
assert.equal(code, 0);
|
|
assert.match(out, /token \(plaintext\):\s+olp_[A-Za-z0-9_-]{43}/);
|
|
assert.match(out, /owner_tier:\s+guest/);
|
|
assert.match(out, /name:\s+anonymous/);
|
|
assert.match(out, /advertise:\s+YES/);
|
|
assert.match(err, /WARNING:.*plaintext is now stored on disk/);
|
|
assert.match(err, /ADR 0011/);
|
|
// Manifest carries plaintext_advertise + matches printed plaintext
|
|
const advManifest = findAdvertisedKey34({ olpHome: TMP });
|
|
assert.ok(advManifest !== null);
|
|
const printedTokenMatch = out.match(/token \(plaintext\):\s+(olp_[A-Za-z0-9_-]{43})/);
|
|
assert.ok(printedTokenMatch);
|
|
assert.equal(advManifest.plaintext_advertise, printedTokenMatch[1]);
|
|
});
|
|
|
|
it('34i — keygen --owner --advertise → exit 1 with ADR 0011 pointer (tier mismatch rejected at CLI layer)', async () => {
|
|
const TMP_I = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-34i-'));
|
|
try {
|
|
let err = '';
|
|
const code = await runOlpKeysCli(
|
|
['keygen', '--owner', '--advertise', '--olp-home', TMP_I],
|
|
{ out: () => {}, err: s => { err += s; } },
|
|
);
|
|
assert.equal(code, 1);
|
|
assert.match(err, /--advertise requires guest tier/);
|
|
assert.match(err, /ADR 0011/);
|
|
// Nothing was written
|
|
assert.equal(listKeys({ olpHome: TMP_I }).length, 0);
|
|
} finally {
|
|
rmSync(TMP_I, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('34j — keygen --anonymous (without --advertise) creates guest key WITHOUT plaintext_advertise field', async () => {
|
|
const TMP_J = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-34j-'));
|
|
try {
|
|
let out = '';
|
|
const code = await runOlpKeysCli(
|
|
['keygen', '--anonymous', '--olp-home', TMP_J],
|
|
{ out: s => { out += s; }, err: () => {} },
|
|
);
|
|
assert.equal(code, 0);
|
|
assert.match(out, /owner_tier:\s+guest/);
|
|
assert.match(out, /name:\s+anonymous/);
|
|
assert.doesNotMatch(out, /advertise:\s+YES/);
|
|
// Manifest does NOT carry plaintext_advertise
|
|
assert.equal(findAdvertisedKey34({ olpHome: TMP_J }), null);
|
|
} finally {
|
|
rmSync(TMP_J, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|
|
});
|
|
|
|
// ── Suite 35: D71 olp-plugin/ smoke tests (ADR 0010 § Phase 4 D71-D73) ─────
|
|
//
|
|
// The plugin's default export takes an OpenClaw `api` object and registers a
|
|
// command — we can't fully unit-test the registration without mocking the
|
|
// gateway. Instead we test the exported helpers + dispatcher with the global
|
|
// fetch swapped out for a mock. These cover:
|
|
//
|
|
// 35a — mono() wrapping for chat surfaces
|
|
// 35b — bar() boundary behaviour (clamping + width)
|
|
// 35c — resolveProxyUrl precedence (env > env-port > config > default)
|
|
// 35d — truncateForChat truncation behaviour
|
|
// 35e — fmtStatus / fmtHealth / fmtUsage / fmtModels / fmtCache /
|
|
// fmtProviders / fmtChainShow / fmtDoctor render expected fields
|
|
// 35f — dispatch() unknown subcommand → help message
|
|
// 35g — dispatch() routes status/health/usage/models/cache to the right URL
|
|
// with Authorization: Bearer header
|
|
// 35h — dispatch() doctor returns the "use SSH" advisory (HTTP doctor
|
|
// endpoint not yet shipped)
|
|
// 35i — dispatch() catches fetch errors and emits "OLP error: ..."
|
|
|
|
import {
|
|
mono as plugMono,
|
|
bar as plugBar,
|
|
statusIcon as plugStatusIcon,
|
|
truncateForChat as plugTruncate,
|
|
resolveProxyUrl as plugResolveProxyUrl,
|
|
fmtStatus as plugFmtStatus,
|
|
fmtHealth as plugFmtHealth,
|
|
fmtUsage as plugFmtUsage,
|
|
fmtModels as plugFmtModels,
|
|
fmtCache as plugFmtCache,
|
|
fmtProviders as plugFmtProviders,
|
|
fmtChainShow as plugFmtChainShow,
|
|
fmtDoctor as plugFmtDoctor,
|
|
cmdHelp as plugCmdHelp,
|
|
dispatch as plugDispatch,
|
|
} from './olp-plugin/index.js';
|
|
|
|
describe('Suite 35 — D71 olp-plugin/ smoke tests (ADR 0010 § Phase 4 D71-D73)', () => {
|
|
|
|
// ── 35a — mono() ────────────────────────────────────────────────────────
|
|
|
|
it('35a — mono() wraps text in triple-backtick code block', () => {
|
|
const out = plugMono('hello\nworld');
|
|
assert.equal(out, '```\nhello\nworld\n```');
|
|
});
|
|
|
|
// ── 35b — bar() ─────────────────────────────────────────────────────────
|
|
|
|
it('35b — bar() clamps pct ∈ [0,1] and respects width', () => {
|
|
assert.equal(plugBar(0, 8), '░░░░░░░░');
|
|
assert.equal(plugBar(1, 8), '████████');
|
|
assert.equal(plugBar(0.5, 8), '████░░░░');
|
|
assert.equal(plugBar(-1, 8), '░░░░░░░░', 'negative pct clamps to 0');
|
|
assert.equal(plugBar(2, 8), '████████', 'pct > 1 clamps to 1');
|
|
assert.equal(plugBar(NaN, 8), '░░░░░░░░', 'NaN treated as 0');
|
|
assert.equal(plugBar(0.5).length, 16, 'default width = 16');
|
|
});
|
|
|
|
it('35b-icon — statusIcon() maps ok/degraded/fail to expected glyphs', () => {
|
|
assert.equal(plugStatusIcon('ok'), '🟢');
|
|
assert.equal(plugStatusIcon(true), '🟢');
|
|
assert.equal(plugStatusIcon('degraded'), '🟡');
|
|
assert.equal(plugStatusIcon('warn'), '🟡');
|
|
assert.equal(plugStatusIcon('fail'), '🔴');
|
|
assert.equal(plugStatusIcon(false), '🔴');
|
|
});
|
|
|
|
// ── 35c — resolveProxyUrl precedence ────────────────────────────────────
|
|
|
|
it('35c — resolveProxyUrl precedence: OLP_PROXY_URL > OLP_PORT > config.proxyUrl > default', () => {
|
|
// 1. Default fallback
|
|
assert.equal(plugResolveProxyUrl({ env: {}, config: {} }), 'http://127.0.0.1:4567');
|
|
// 2. config.proxyUrl wins over default
|
|
assert.equal(
|
|
plugResolveProxyUrl({ env: {}, config: { proxyUrl: 'http://10.0.0.5:9999' } }),
|
|
'http://10.0.0.5:9999',
|
|
);
|
|
// 3. OLP_PORT wins over config.proxyUrl
|
|
assert.equal(
|
|
plugResolveProxyUrl({ env: { OLP_PORT: '8000' }, config: { proxyUrl: 'http://config:1' } }),
|
|
'http://127.0.0.1:8000',
|
|
);
|
|
// 4. OLP_PROXY_URL wins over OLP_PORT + config
|
|
assert.equal(
|
|
plugResolveProxyUrl({
|
|
env: { OLP_PROXY_URL: 'http://env-url:1234', OLP_PORT: '8000' },
|
|
config: { proxyUrl: 'http://config:1' },
|
|
}),
|
|
'http://env-url:1234',
|
|
);
|
|
});
|
|
|
|
// ── 35d — truncateForChat ───────────────────────────────────────────────
|
|
|
|
it('35d — truncateForChat passes short text untouched + truncates long text with marker', () => {
|
|
const short = 'hello world';
|
|
assert.equal(plugTruncate(short), short);
|
|
const long = 'A'.repeat(5000);
|
|
const truncated = plugTruncate(long, 1000);
|
|
assert.ok(truncated.length <= 1000);
|
|
assert.match(truncated, /truncated, use SSH for full/);
|
|
assert.ok(truncated.startsWith('AAA'));
|
|
});
|
|
|
|
// ── 35e — Formatters ────────────────────────────────────────────────────
|
|
|
|
it('35e-1 — fmtStatus() renders version + uptime + provider list + recent errors', () => {
|
|
const body = {
|
|
ok: true,
|
|
version: '0.4.0-phase4',
|
|
uptime_human: '1h 2m 3s',
|
|
providers: {
|
|
enabled: 2,
|
|
available: 3,
|
|
status: {
|
|
anthropic: { ok: true, activeSpawns: 0 },
|
|
openai: { ok: false, error: 'CLI not found', activeSpawns: 0 },
|
|
},
|
|
},
|
|
stats: { total_requests: 42, active_requests: 1, cache: { hits: 10, misses: 5, size: 7 } },
|
|
recent_errors: [{ time: '2026-05-26T11:22:33Z', provider: 'openai', message: 'spawn EACCES' }],
|
|
};
|
|
const out = plugFmtStatus(body);
|
|
assert.match(out, /0\.4\.0-phase4/);
|
|
assert.match(out, /1h 2m 3s/);
|
|
assert.match(out, /Providers: 2 enabled \/ 3 available/);
|
|
assert.match(out, /anthropic/);
|
|
assert.match(out, /openai.*CLI not found/);
|
|
assert.match(out, /Requests: 42 total/);
|
|
assert.match(out, /Cache: 10 hit \/ 5 miss/);
|
|
assert.match(out, /Recent errors \(1\)/);
|
|
assert.match(out, /spawn EACCES/);
|
|
});
|
|
|
|
it('35e-2 — fmtHealth() renders ok + version + provider list', () => {
|
|
const body = {
|
|
ok: true,
|
|
version: '0.4.0-phase4',
|
|
uptime_human: '5m',
|
|
providers: { anthropic: { ok: true }, openai: { ok: false } },
|
|
};
|
|
const out = plugFmtHealth(body);
|
|
assert.match(out, /Status: ok/);
|
|
assert.match(out, /v0\.4\.0-phase4/);
|
|
assert.match(out, /anthropic/);
|
|
assert.match(out, /openai/);
|
|
});
|
|
|
|
it('35e-3 — fmtUsage() renders 24h window + per-provider quota + top fallback chains', () => {
|
|
const body = {
|
|
cache_hit_24h: 0.42,
|
|
quota: [
|
|
{ name: 'anthropic', percent_used: 33 },
|
|
{ name: 'mistral', percent_used: null },
|
|
],
|
|
top_fallback_chains_24h: [
|
|
{ count: 5, chain: ['anthropic', 'openai'] },
|
|
],
|
|
};
|
|
const out = plugFmtUsage(body);
|
|
assert.match(out, /Cache hit \(24h\): 42\.0%/);
|
|
assert.match(out, /anthropic.*33%/);
|
|
assert.match(out, /mistral.*no quota api/);
|
|
assert.match(out, /5 {2}anthropic → openai/);
|
|
});
|
|
|
|
it('35e-4 — fmtModels() handles empty + populated', () => {
|
|
assert.equal(plugFmtModels({ data: [] }), 'No models.');
|
|
const out = plugFmtModels({
|
|
data: [
|
|
{ id: 'claude-sonnet-4-5', owned_by: 'anthropic' },
|
|
{ id: 'gpt-5', owned_by: 'openai' },
|
|
],
|
|
});
|
|
assert.match(out, /Models \(2\)/);
|
|
assert.match(out, /claude-sonnet-4-5\s+\(anthropic\)/);
|
|
assert.match(out, /gpt-5\s+\(openai\)/);
|
|
});
|
|
|
|
it('35e-5 — fmtCache() renders hit/miss/size/inflight', () => {
|
|
const out = plugFmtCache({ size: 12, hits: 100, misses: 25, inflightCount: 2, evictions: 1 });
|
|
assert.match(out, /Entries: {8}12/);
|
|
assert.match(out, /Hits: {11}100/);
|
|
assert.match(out, /Misses: {9}25/);
|
|
assert.match(out, /Inflight: {7}2/);
|
|
assert.match(out, /Evictions: {6}1/);
|
|
});
|
|
|
|
it('35e-6 — fmtProviders() lists registry providers + enabled flags', () => {
|
|
const registry = {
|
|
providers: {
|
|
anthropic: { tier: 'D', models: ['m1', 'm2'] },
|
|
openai: { tier: 'D', models: ['m3'], candidate: true },
|
|
},
|
|
};
|
|
const out = plugFmtProviders(registry, { anthropic: true });
|
|
assert.match(out, /anthropic\s+enabled/);
|
|
assert.match(out, /openai\s+disabled.*\(candidate\)/);
|
|
});
|
|
|
|
it('35e-7 — fmtChainShow() handles empty / all / single-target', () => {
|
|
assert.equal(plugFmtChainShow({}), 'No chains configured.');
|
|
const chains = {
|
|
'claude-sonnet-4-5': [{ provider: 'anthropic', model: 'claude-sonnet-4-5' }],
|
|
'gpt-5': ['openai:gpt-5'],
|
|
};
|
|
const all = plugFmtChainShow(chains);
|
|
assert.match(all, /claude-sonnet-4-5/);
|
|
assert.match(all, /gpt-5/);
|
|
const one = plugFmtChainShow(chains, 'gpt-5');
|
|
assert.match(one, /gpt-5:/);
|
|
assert.match(one, /→ openai:gpt-5/);
|
|
const missing = plugFmtChainShow(chains, 'no-such-model');
|
|
assert.match(missing, /not in routing\.chains/);
|
|
});
|
|
|
|
it('35e-8 — fmtDoctor() renders summary + checks + next_action', () => {
|
|
const body = {
|
|
summary: '2 OK, 1 WARN',
|
|
checks: [
|
|
{ id: 'server.running', status: 'ok', message: 'reachable' },
|
|
{ id: 'auth.owner_key_exists', status: 'warn', message: 'no owner key configured' },
|
|
],
|
|
fail_count: 0, warn_count: 1, ok_count: 1,
|
|
kind: 'fix_config',
|
|
next_action: {
|
|
ai_executable: ['npx olp-keys keygen --owner'],
|
|
human_required: ['capture the printed token'],
|
|
},
|
|
};
|
|
const out = plugFmtDoctor(body);
|
|
assert.match(out, /2 OK, 1 WARN/);
|
|
assert.match(out, /server\.running/);
|
|
assert.match(out, /auth\.owner_key_exists/);
|
|
assert.match(out, /fail=0 warn=1 ok=1/);
|
|
assert.match(out, /kind=fix_config/);
|
|
assert.match(out, /npx olp-keys keygen --owner/);
|
|
assert.match(out, /capture the printed token/);
|
|
});
|
|
|
|
// ── 35f — dispatch: unknown / help / "" ──────────────────────────────────
|
|
|
|
it('35f-1 — dispatch("") returns cmdHelp() body', async () => {
|
|
const r = await plugDispatch('', { proxyUrl: 'http://x', registry: { providers: {} } });
|
|
assert.equal(r.text, plugCmdHelp());
|
|
});
|
|
|
|
it('35f-2 — dispatch("frobnicate") returns "Unknown subcommand:" + help', async () => {
|
|
const r = await plugDispatch('frobnicate', { proxyUrl: 'http://x', registry: { providers: {} } });
|
|
assert.match(r.text, /Unknown subcommand: frobnicate/);
|
|
assert.match(r.text, /OLP Commands/);
|
|
});
|
|
|
|
it('35f-3 — dispatch("help") returns cmdHelp() with read-only disclaimer', async () => {
|
|
const r = await plugDispatch('help', { proxyUrl: 'http://x', registry: { providers: {} } });
|
|
assert.match(r.text, /OLP Commands/);
|
|
assert.match(r.text, /Mutating commands.*NOT/);
|
|
});
|
|
|
|
// ── 35g — dispatch: routes status/health/usage/models/cache ──────────────
|
|
|
|
it('35g — dispatch routes each subcommand to the right URL with Authorization header', async () => {
|
|
const calls = [];
|
|
const mockFetch = async (url, opts) => {
|
|
calls.push({ url, headers: opts?.headers ?? {} });
|
|
// Per-route response stubs
|
|
if (url.endsWith('/v0/management/status')) {
|
|
return { ok: true, status: 200, json: async () => ({
|
|
ok: true, version: '0.4.0', uptime_human: '1m',
|
|
providers: { enabled: 0, available: 3, status: {} },
|
|
stats: { total_requests: 0, active_requests: 0 },
|
|
recent_errors: [],
|
|
}) };
|
|
}
|
|
if (url.endsWith('/health')) {
|
|
return { ok: true, status: 200, json: async () => ({ ok: true, version: '0.4.0' }) };
|
|
}
|
|
if (url.endsWith('/v0/management/dashboard-data')) {
|
|
return { ok: true, status: 200, json: async () => ({ cache_hit_24h: 0.5 }) };
|
|
}
|
|
if (url.endsWith('/v1/models')) {
|
|
return { ok: true, status: 200, json: async () => ({ data: [{ id: 'm1' }] }) };
|
|
}
|
|
if (url.endsWith('/cache/stats')) {
|
|
return { ok: true, status: 200, json: async () => ({ size: 0, hits: 0, misses: 0, inflightCount: 0 }) };
|
|
}
|
|
throw new Error(`unmocked url: ${url}`);
|
|
};
|
|
|
|
const opts = {
|
|
proxyUrl: 'http://test:9999',
|
|
apiKey: 'olp_TESTKEY',
|
|
registry: { providers: { anthropic: { tier: 'D', models: [] } } },
|
|
fetchFn: mockFetch,
|
|
};
|
|
|
|
for (const sub of ['status', 'health', 'usage', 'models', 'cache']) {
|
|
calls.length = 0;
|
|
const r = await plugDispatch(sub, opts);
|
|
assert.equal(calls.length, 1, `${sub} should make exactly 1 HTTP call`);
|
|
assert.equal(calls[0].headers.Authorization, 'Bearer olp_TESTKEY', `${sub} must include Bearer header`);
|
|
assert.ok(r.text.length > 0, `${sub} should produce non-empty output`);
|
|
assert.ok(!/^OLP error/.test(r.text), `${sub} should not error; got: ${r.text}`);
|
|
}
|
|
|
|
// URL routing per subcommand
|
|
calls.length = 0;
|
|
await plugDispatch('status', opts);
|
|
assert.equal(calls[0].url, 'http://test:9999/v0/management/status');
|
|
|
|
calls.length = 0;
|
|
await plugDispatch('usage', opts);
|
|
assert.equal(calls[0].url, 'http://test:9999/v0/management/dashboard-data');
|
|
});
|
|
|
|
it('35g-providers — dispatch("providers") uses registry locally without HTTP', async () => {
|
|
let calls = 0;
|
|
const r = await plugDispatch('providers', {
|
|
proxyUrl: 'http://test:9999',
|
|
registry: { providers: { anthropic: { tier: 'D', models: [] }, openai: { tier: 'D', models: [] } } },
|
|
fetchFn: async () => { calls++; return { ok: true, status: 200, json: async () => ({}) }; },
|
|
});
|
|
assert.equal(calls, 0, 'providers must NOT make an HTTP call');
|
|
assert.match(r.text, /anthropic/);
|
|
assert.match(r.text, /openai/);
|
|
});
|
|
|
|
it('35g-chain — dispatch("chain show <model>") uses local chainsLocal arg', async () => {
|
|
const r = await plugDispatch('chain show foo-model', {
|
|
proxyUrl: 'http://test:9999',
|
|
registry: { providers: {} },
|
|
chainsLocal: { 'foo-model': [{ provider: 'anthropic', model: 'foo' }] },
|
|
fetchFn: async () => { throw new Error('should not fetch'); },
|
|
});
|
|
assert.match(r.text, /foo-model:/);
|
|
assert.match(r.text, /→.*anthropic/);
|
|
});
|
|
|
|
it('35g-chain-usage — dispatch("chain bogus") prints usage hint', async () => {
|
|
const r = await plugDispatch('chain bogus', {
|
|
proxyUrl: 'http://test:9999',
|
|
registry: { providers: {} },
|
|
fetchFn: async () => { throw new Error('should not fetch'); },
|
|
});
|
|
assert.match(r.text, /Usage: \/olp chain show \[model\]/);
|
|
});
|
|
|
|
// ── 35h — dispatch: doctor advisory ─────────────────────────────────────
|
|
|
|
it('35h — dispatch("doctor") returns the "use SSH" advisory (HTTP endpoint not yet shipped)', async () => {
|
|
let calls = 0;
|
|
const r = await plugDispatch('doctor', {
|
|
proxyUrl: 'http://test:9999',
|
|
registry: { providers: {} },
|
|
fetchFn: async () => { calls++; return { ok: true, status: 200, json: async () => ({}) }; },
|
|
});
|
|
assert.equal(calls, 0, 'doctor must NOT make an HTTP call yet (Phase 4 surface is local only)');
|
|
assert.match(r.text, /not yet wired through HTTP/);
|
|
assert.match(r.text, /Run `olp doctor` over SSH/);
|
|
});
|
|
|
|
// ── 35i — dispatch: error catching ──────────────────────────────────────
|
|
|
|
it('35i — dispatch catches fetch errors and emits "OLP error: ..."', async () => {
|
|
const r = await plugDispatch('status', {
|
|
proxyUrl: 'http://test:9999',
|
|
apiKey: 'olp_X',
|
|
registry: { providers: {} },
|
|
fetchFn: async () => { throw new Error('ECONNREFUSED'); },
|
|
});
|
|
assert.match(r.text, /^OLP error: ECONNREFUSED/);
|
|
});
|
|
|
|
it('35i-401 — dispatch surfaces 401 unauthorized with helpful hint', async () => {
|
|
const r = await plugDispatch('status', {
|
|
proxyUrl: 'http://test:9999',
|
|
apiKey: 'olp_BAD',
|
|
registry: { providers: {} },
|
|
fetchFn: async () => ({ ok: false, status: 401, statusText: 'Unauthorized', json: async () => ({}) }),
|
|
});
|
|
assert.match(r.text, /OLP error.*401 unauthorized/);
|
|
assert.match(r.text, /owner-tier/);
|
|
});
|
|
|
|
it('35i-403 — dispatch surfaces 403 forbidden with helpful hint', async () => {
|
|
const r = await plugDispatch('cache', {
|
|
proxyUrl: 'http://test:9999',
|
|
apiKey: 'olp_GUEST',
|
|
registry: { providers: {} },
|
|
fetchFn: async () => ({ ok: false, status: 403, statusText: 'Forbidden', json: async () => ({}) }),
|
|
});
|
|
assert.match(r.text, /OLP error.*403 forbidden/);
|
|
assert.match(r.text, /not owner-tier/);
|
|
});
|
|
});
|
|
|
|
// ── Suite 36: D74 v0.4.1 hotfix regression tests (maintainer-review findings) ─
|
|
//
|
|
// These tests pin the FIVE issues maintainer caught in independent post-v0.4.0
|
|
// review that previous D-day reviewers all missed (because they reviewed against
|
|
// spec text, not runtime contracts). Each test exercises a real codepath that,
|
|
// pre-D74, would have produced the documented wrong behavior.
|
|
|
|
import { writeFileSync as _writeFileSyncS36, readFileSync as _readFileSyncS36, mkdtempSync as _mkdtempSyncS36, rmSync as _rmSyncS36 } from 'node:fs';
|
|
import { tmpdir as _tmpdirS36 } from 'node:os';
|
|
import { join as _joinS36 } from 'node:path';
|
|
import { spawnSync as _spawnSyncS36 } from 'node:child_process';
|
|
|
|
describe('Suite 36 — D74 v0.4.1 hotfix regression (maintainer review findings)', () => {
|
|
it('36a (P1-1) — runDoctor accepts authHeaders + passes them to server.running probe', async () => {
|
|
// Pre-D74: lib/doctor.mjs called httpGet(`${proxyUrl}/health`) with no
|
|
// headers, so under default `auth.allow_anonymous: false` the probe got
|
|
// 401 and reported "server down" — false negative. D74 threads
|
|
// authHeaders through buildBuiltinChecks. This test confirms the wire.
|
|
const { buildBuiltinChecks } = await import('./lib/doctor.mjs');
|
|
const tmpHome = _mkdtempSyncS36(_joinS36(_tmpdirS36(), 'olp-d74a-'));
|
|
const checks = buildBuiltinChecks({
|
|
olpHome: tmpHome,
|
|
proxyUrl: 'http://127.0.0.1:1', // closed port — probe fails with ECONN
|
|
authHeaders: { Authorization: 'Bearer olp_FAKE' },
|
|
});
|
|
const serverRunning = checks.find(c => c.id === 'server.running');
|
|
assert.ok(serverRunning, 'server.running check must exist');
|
|
// We can't easily intercept httpGet here without DI, but the check at
|
|
// least accepts the opts and runs. Real wire-level coverage is 36b below.
|
|
_rmSyncS36(tmpHome, { recursive: true, force: true });
|
|
});
|
|
|
|
it('36b (P1-1) — server.running distinguishes 401 (server up, bad token) from "server down"', async () => {
|
|
__setAuthConfig({ allow_anonymous: false, owner_only_endpoints: ['/health'] });
|
|
const serverMod = await import('./server.mjs');
|
|
const s = serverMod.createOlpServer();
|
|
let port36b = 28300 + Math.floor(Math.random() * 100);
|
|
await new Promise((resolve, reject) => {
|
|
s.listen(port36b, '127.0.0.1', resolve);
|
|
s.once('error', e => {
|
|
if (e.code === 'EADDRINUSE') { port36b++; s.listen(port36b, '127.0.0.1', resolve); s.once('error', reject); }
|
|
else reject(e);
|
|
});
|
|
});
|
|
|
|
const { runDoctor } = await import('./lib/doctor.mjs');
|
|
try {
|
|
const result = await runDoctor({
|
|
olpHome: process.env.OLP_HOME,
|
|
proxyUrl: `http://127.0.0.1:${port36b}`,
|
|
// intentionally NO authHeaders — simulates user without OLP_API_KEY
|
|
});
|
|
const sr = result.checks.find(c => c.id === 'server.running');
|
|
assert.ok(sr, 'server.running present');
|
|
assert.equal(sr.status, 'fail', 'fails because no token');
|
|
assert.match(sr.message, /401|bearer token/i,
|
|
`must mention 401 / bearer token specifically — got: ${sr.message}`);
|
|
assert.ok(!/unreachable/i.test(sr.message),
|
|
'"unreachable" would falsely imply server is down (P1-1 bug)');
|
|
} finally {
|
|
await new Promise(r => s.close(r));
|
|
__resetAuthConfig();
|
|
}
|
|
});
|
|
|
|
it('36c (P1-2) — olp-connect rejects malformed --key', () => {
|
|
// The bash validator: ^olp_[A-Za-z0-9_-]{43}$. Anything else is rejected
|
|
// before it can touch a shell rc file or environment.d entry.
|
|
const result = _spawnSyncS36('bash', [
|
|
_joinS36(import.meta.dirname ?? process.cwd(), 'bin/olp-connect'),
|
|
'127.0.0.1',
|
|
'--port', '1',
|
|
'--key', 'not-an-olp-token',
|
|
'--dry-run',
|
|
], { encoding: 'utf8', timeout: 5000 });
|
|
assert.equal(result.status, 1, `expected exit 1 for malformed --key, got ${result.status}; stderr=${result.stderr.slice(0, 300)}`);
|
|
assert.match(result.stderr + result.stdout, /token format does not match/i,
|
|
'must surface the validator error');
|
|
});
|
|
|
|
it('36d (P1-2) — olp-connect accepts a properly-formed olp_ token', () => {
|
|
// Pad to 43 base64url chars after olp_. Don't actually hit the server —
|
|
// use --port 1 (closed) which makes the connectivity probe fail at
|
|
// the connect step (exit 2), but only AFTER the --key validator passes.
|
|
const goodKey = 'olp_' + 'A'.repeat(43);
|
|
const result = _spawnSyncS36('bash', [
|
|
_joinS36(import.meta.dirname ?? process.cwd(), 'bin/olp-connect'),
|
|
'127.0.0.1',
|
|
'--port', '1',
|
|
'--key', goodKey,
|
|
'--dry-run',
|
|
], { encoding: 'utf8', timeout: 5000 });
|
|
// Validator passes → reaches connectivity step → exits 2 (or whatever
|
|
// bash propagation gives) but NOT exit 1 from the validator.
|
|
assert.notEqual(result.status, 1, `expected validator to pass; got exit 1 with stderr=${result.stderr.slice(0, 300)}`);
|
|
assert.ok(!/token format does not match/i.test(result.stderr + result.stdout),
|
|
'validator must NOT trigger for a well-formed token');
|
|
});
|
|
|
|
it('36e (P2-3) — CacheStore.stats() returns {hits, misses, size, inflightCount} shape', async () => {
|
|
// Real cacheStore.stats() returns { hits, misses, size, inflightCount } per
|
|
// lib/cache/store.mjs. Pre-D74 cmdCache read body.entries / body.bytes /
|
|
// body.maxBytes — all undefined → output showed "entries: ?" and "bytes: 0".
|
|
// Pin the shape so a future server-side rename can't reintroduce the bug.
|
|
// Use the CacheStore class directly (no HTTP) since the field names ARE the
|
|
// contract — server.mjs handleCacheStats just sendJSON(s, 200, cacheStore.stats()).
|
|
const { CacheStore: CS36 } = await import('./lib/cache/store.mjs');
|
|
const cs = new CS36();
|
|
const stats = cs.stats();
|
|
assert.ok('size' in stats, 'CacheStore.stats() shape: size field present');
|
|
assert.ok('hits' in stats, 'CacheStore.stats() shape: hits field present');
|
|
assert.ok('misses' in stats, 'CacheStore.stats() shape: misses field present');
|
|
assert.ok('inflightCount' in stats, 'CacheStore.stats() shape: inflightCount field present');
|
|
assert.ok(!('entries' in stats), 'must NOT have OCP-era entries field');
|
|
assert.ok(!('bytes' in stats), 'must NOT have OCP-era bytes field');
|
|
assert.ok(!('maxBytes' in stats), 'must NOT have OCP-era maxBytes field');
|
|
// Also pin that cmdCache reads body.size (not body.entries) by grepping the source
|
|
const cliSrc = _readFileSyncS36(_joinS36(import.meta.dirname ?? process.cwd(), 'bin/olp.mjs'), 'utf8');
|
|
assert.ok(/body\.size\b/.test(cliSrc), 'cmdCache must read body.size for entries display');
|
|
assert.ok(/body\.inflightCount\b/.test(cliSrc), 'cmdCache must read body.inflightCount');
|
|
assert.ok(!/body\.entries\b/.test(cliSrc), 'cmdCache must NOT read body.entries (OCP-era field)');
|
|
});
|
|
|
|
it('36f (P2-3) — dashboard-data payload + cmdUsage source pin the wire-contract field names', () => {
|
|
// server.mjs handleManagementDashboardData (~line 2027) builds the payload
|
|
// inline. Pin by grepping the server source: the keys MUST be window_24h /
|
|
// cache_hit_24h / quota / spend_trend_30d / top_fallback_chains_24h /
|
|
// cache_stats. Pre-D74 cmdUsage read body.usage_24h.requests / body.providers
|
|
// / body.top_fallback_chains — none of which exist in the actual payload.
|
|
const serverSrc = _readFileSyncS36(_joinS36(import.meta.dirname ?? process.cwd(), 'server.mjs'), 'utf8');
|
|
assert.ok(/window_24h:\s*auditAggregateRequests/.test(serverSrc),
|
|
'handleManagementDashboardData must emit window_24h field');
|
|
assert.ok(/cache_hit_24h:\s*auditCacheHitRateWindow/.test(serverSrc),
|
|
'handleManagementDashboardData must emit cache_hit_24h field');
|
|
assert.ok(/top_fallback_chains_24h:/.test(serverSrc),
|
|
'handleManagementDashboardData must emit top_fallback_chains_24h field (NOT top_fallback_chains)');
|
|
assert.ok(/cache_stats:\s*cacheStore\.stats/.test(serverSrc),
|
|
'handleManagementDashboardData must emit cache_stats field');
|
|
// Now pin cmdUsage reads the matching keys
|
|
const cliSrc = _readFileSyncS36(_joinS36(import.meta.dirname ?? process.cwd(), 'bin/olp.mjs'), 'utf8');
|
|
assert.ok(/body\.window_24h\b/.test(cliSrc), 'cmdUsage must read body.window_24h');
|
|
assert.ok(/body\.cache_hit_24h\b/.test(cliSrc), 'cmdUsage must read body.cache_hit_24h');
|
|
assert.ok(/body\.top_fallback_chains_24h\b/.test(cliSrc),
|
|
'cmdUsage must read body.top_fallback_chains_24h (NOT body.top_fallback_chains)');
|
|
assert.ok(/body\.quota\b/.test(cliSrc), 'cmdUsage must read body.quota');
|
|
// Pre-D74 stale reads must be GONE from the CLI source
|
|
assert.ok(!/body\.usage_24h\.requests\b/.test(cliSrc),
|
|
'cmdUsage must NOT read body.usage_24h.requests (OCP-era field)');
|
|
});
|
|
|
|
it('36g (P2-4) — olp-plugin fmtHealth iterates providers.status (not providers.*)', async () => {
|
|
// Real /health full payload: providers: { enabled, available, status: { anthropic: {...} } }
|
|
// Pre-D74 fmtHealth iterated Object.entries(body.providers) and the body
|
|
// contained `status: {...}` so the loop printed a pseudo-provider named "status".
|
|
const { fmtHealth } = await import('./olp-plugin/index.js');
|
|
const realServerShape = {
|
|
ok: true,
|
|
version: '0.4.1',
|
|
uptime_human: '1m 3s',
|
|
providers: {
|
|
enabled: 2,
|
|
available: 3,
|
|
status: {
|
|
anthropic: { ok: true, activeSpawns: 0 },
|
|
openai: { ok: false, error: 'oauth-missing', activeSpawns: 0 },
|
|
},
|
|
},
|
|
};
|
|
const out = fmtHealth(realServerShape);
|
|
assert.match(out, /anthropic/, 'must list anthropic by name');
|
|
assert.match(out, /openai/, 'must list openai by name');
|
|
// assert.notMatch isn't available on assert/strict in some Node versions;
|
|
// use the explicit negation pattern to stay portable.
|
|
assert.ok(!/^\s*[🟢🔴⚪]\s*status\s*$/m.test(out),
|
|
'must NOT list "status" as a pseudo-provider (the P2-4 bug)');
|
|
assert.ok(!/^\s*[🟢🔴⚪]\s*enabled\s*$/m.test(out),
|
|
'must NOT list "enabled" as a pseudo-provider');
|
|
assert.ok(!/^\s*[🟢🔴⚪]\s*available\s*$/m.test(out),
|
|
'must NOT list "available" as a pseudo-provider');
|
|
// Also confirm the new enabled/available header
|
|
assert.match(out, /2 enabled \/ 3 available/);
|
|
});
|
|
|
|
it('36h (P3-5) — server startup banner does not hardcode a stale phase', () => {
|
|
// Pre-D74 banner literal said "Phase 1 in progress" forever.
|
|
const serverSrc = _readFileSyncS36(_joinS36(import.meta.dirname ?? process.cwd(), 'server.mjs'), 'utf8');
|
|
assert.ok(!/Phase 1 in progress/.test(serverSrc),
|
|
'startup banner must not hardcode "Phase 1 in progress" (stale post v0.4.0)');
|
|
assert.ok(!/Phase 2 in progress/.test(serverSrc));
|
|
assert.ok(!/Phase 3 in progress/.test(serverSrc));
|
|
});
|
|
});
|