taodeng/olp
1
0
Fork 0
mirror of https://github.com/dtzp555-max/olp.git synced 2026-07-21 21:15:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(server)+fix(obs): D18 — populate /v1/models + standard X-OLP-* on error paths (Findings 10 + 11)

Browse Source 
cold-audit catch from 2026-05-23

Cold-audit Findings 10 + 11 (both P3, server + observability). F10:
/v1/models returned {data: []} unconditionally; README claimed it
lists from models-registry.json — a client calling /v1/models to
discover what OLP serves would conclude OLP has no models. F11: error
responses (chain-exhausted, pre-routing 4xx) omitted the standard
5-header set; ADR 0004 § Observability requires every response to
carry X-OLP-{Provider-Used, Model-Used, Fallback-Hops, Cache, Latency-Ms}.

Changes (2 files, +427 / -9):

1. server.mjs handleModels: populate from loaded providers × canonical
   model IDs. Each entry has exactly {id, object: 'model', created,
   owned_by} — no invented fields (Rule 2(b)). `created` is computed
   once per request as Math.floor(Date.now()/1000). Iteration order is
   insertion-order of loadedProviders Map × insertion-order of each
   provider's models[]. Empty case (no providers enabled) returns
   {object: 'list', data: []} via natural empty iteration. Aliases are
   NOT included — per D17 SPOT, models[] is canonical-only; the
   /v1/models endpoint surfaces canonical IDs only.

2. server.mjs sendError extended with optional 5th arg extraHeaders.
   Non-breaking — existing call sites (10 of them) use the default {}.
   Three pre-routing call sites inside handleChatCompletions now pass
   X-OLP-Latency-Ms (computed at-error-time as Date.now() - startMs):
   - 415 Content-Type mismatch
   - 400 invalid JSON body
   - 400 IR parse error
   The 404 (route not found) and 500 (top-level catch) paths are
   outside handleChatCompletions and have no startMs — they remain
   without latency rather than synthesize a value.

3. server.mjs chain-exhausted error path: replaced minimal-header
   block with olpHeaders({...}) call producing the full standard
   5-tuple. X-OLP-Fallback-Exhausted is preserved as an additional
   flag layered on top. The 5 header values reflect engine state per
   ADR 0004 step 4 ("preserve A's identity — return FIRST hop's
   provider/model and original error to user"): providerUsed =
   chain[0].provider, modelUsed = chain[0].model, fallbackHops =
   attempted count, cacheStatus = 'miss'.

4. test-features.mjs Suite 17 — 7 new tests:
   - 17a: /v1/models with anthropic enabled → 3 entries, all
     owned_by='anthropic', no aliases in response
   - 17b: /v1/models with no providers → {object:'list', data:[]}
   - 17c: /v1/models entries have only {id, object, created, owned_by}
     (no invented fields — Rule 2(b) assertion)
   - 17d: /v1/models with all 3 providers → canonical IDs present,
     aliases absent (sonnet/devstral not in response)
   - 17e: chain-exhausted response has all 5 standard X-OLP-* headers
     present + X-OLP-Fallback-Exhausted
   - 17f: 2-hop chain both fail → X-OLP-Fallback-Hops: '2' value check
   - 17g: pre-routing 400 invalid JSON has X-OLP-Latency-Ms (non-negative
     integer); X-OLP-Provider-Used absent (no provider context — honest)

Tests: 317 → 324 (+7). All pass on Node 20.

Pre-commit fold-in (per evidence-first checkpoint #4):

- **D18 reviewer flagged Concern #1**: original inline comment described
  providerUsed as "last-attempted provider name." This was wrong —
  lib/fallback/engine.mjs returns chain[0].provider on chain-exhausted
  (per ADR 0004 step 4 "preserve A's identity"), which is the FIRST
  hop, not the last. The "last-attempted" framing came from my own
  dispatch brief and the implementer accurately reflected the brief.
  Folded in: corrected the comment to honestly describe what the
  engine returns. Same class of doc-vs-reality drift as D11 / D16 /
  D17 — the cold-audit + diff-review combination is catching these
  consistently across the P3 batch.

Authority:
- ADR 0002 § Loading model — `models: string[]` enumeration
- ADR 0004 § Observability headers — the 5-header standard set
  https://github.com/dtzp555-max/olp/blob/main/docs/adr/0004-fallback-engine.md
- ADR 0004 step 4 — "preserve A's identity" on chain exhaustion
- OpenAI Chat Completions spec /v1/models response shape
  https://platform.openai.com/docs/api-reference/models/list
- ALIGNMENT.md Rule 2(b) — only spec-defined fields in OpenAI responses
- CC 开发铁律 v1.6 § 10.x — Cold Audit Findings 10 + 11

Reviewer (Iron Rule v1.6 § 10.x Mode A, fresh-context opus, independent
of drafter): APPROVE_WITH_MINOR. Verified all 5 X-OLP-* headers fire
on chain-exhausted path; verified Math.floor(Date.now()/1000) computed
once per request (not per-model); verified aliases excluded; verified
404 + 500 paths honestly omit latency rather than synthesize. Caught
the providerUsed-comment drift which was folded in.

Follow-up items (reviewer's non-blocking observations, NOT in this PR):

- 4 other error sites inside handleChatCompletions have startMs
  available but don't emit X-OLP-Latency-Ms (503 no_enabled_provider,
  503 provider-not-enabled-streaming, 502 streaming post-error, 500
  fallback programming error). The "Latency-Ms-when-startMs-is-available"
  rule should be uniform; file as 11b or future cleanup
- Tests 17e/17f could strengthen by asserting specific values not just
  presence (e.g., x-olp-provider-used === 'anthropic'); already done
  for fallback-hops in 17f
- Test 17e/17f setup duplication — extractable helper for cosmetic
  cleanup

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
taodeng
2026-05-24 12:08:19 +10:00
co-authored by Claude Opus 4.7
parent cb86807009
commit 82ff00784d
2 changed files with 431 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test-features.mjs
+377
View File
@@ -5038,6 +5038,383 @@ describe('Streaming cache-miss real-time (Suite 15)', () => {
});
// ── Suite 17: D18 — /v1/models population + X-OLP-* headers on errors ────────
//
// Finding 10: /v1/models returned empty data regardless of loaded providers.
// Finding 11: chain-exhausted (and other) error responses omitted the standard
// X-OLP-* observability headers required by ADR 0004 § Observability.
//
// Tests:
// 17a: /v1/models with anthropic enabled → 200 + 3 entries, owned_by='anthropic'
// 17b: /v1/models with no providers enabled → 200 + data:[]
// 17c: /v1/models entries contain only canonical IDs (no alias like 'sonnet')
// 17d: /v1/models entries match OpenAI spec shape (id/object/created/owned_by only)
// 17e: chain-exhausted error response carries all 5 X-OLP-* headers
// 17f: chain-exhausted error X-OLP-Fallback-Hops reflects hops attempted
// 17g: pre-routing 400 (invalid JSON body) carries X-OLP-Latency-Ms header
//
// Authority: OpenAI /v1/models spec (https://platform.openai.com/docs/api-reference/models);
// ADR 0004 § Observability headers (5-header set required on every response)
import {
createOlpServer as createServer17,
__setProvidersEnabled as setProviders17,
__resetProvidersEnabled as resetProviders17,
__setFallbackConfig as setFallbackConfig17,
__resetFallbackConfig as resetFallbackConfig17,
__clearCache as clearCache17,
} from './server.mjs';
describe('/v1/models population + X-OLP-* error headers (Suite 17)', () => {
// ── 17a: /v1/models with anthropic enabled → 3 model entries ─────────────
it('17a: /v1/models with anthropic enabled → 200 + 3 entries with owned_by="anthropic"', async () => {
setProviders17({ anthropic: true });
const s = createServer17();
const p = 25456 + Math.floor(Math.random() * 400);
await new Promise((resolve, reject) => {
s.listen(p, '127.0.0.1', resolve);
s.once('error', reject);
});
try {
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
assert.equal(r.status, 200);
const body = JSON.parse(r.body);
assert.equal(body.object, 'list');
assert.ok(Array.isArray(body.data), 'data must be an array');
// Anthropic has 3 canonical models in models-registry.json
assert.equal(body.data.length, 3, `Expected 3 anthropic models, got ${body.data.length}`);
for (const entry of body.data) {
assert.equal(entry.owned_by, 'anthropic', `Expected owned_by='anthropic', got '${entry.owned_by}'`);
}
} finally {
resetProviders17();
await new Promise(r => s.close(r));
}
});
// ── 17b: /v1/models with no providers enabled → data:[] ──────────────────
it('17b: /v1/models with no providers enabled → 200 + data:[]', async () => {
setProviders17({});
const s = createServer17();
const p = 25460 + Math.floor(Math.random() * 400);
await new Promise((resolve, reject) => {
s.listen(p, '127.0.0.1', resolve);
s.once('error', reject);
});
try {
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
assert.equal(r.status, 200);
const body = JSON.parse(r.body);
assert.equal(body.object, 'list');
assert.deepEqual(body.data, [], 'data must be empty when no providers are enabled');
} finally {
resetProviders17();
await new Promise(r => s.close(r));
}
});
// ── 17c: /v1/models contains only canonical IDs, no aliases ───────────────
it('17c: /v1/models returns canonical IDs only (no aliases like "sonnet")', async () => {
setProviders17({ anthropic: true });
const s = createServer17();
const p = 25464 + Math.floor(Math.random() * 400);
await new Promise((resolve, reject) => {
s.listen(p, '127.0.0.1', resolve);
s.once('error', reject);
});
try {
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
assert.equal(r.status, 200);
const body = JSON.parse(r.body);
const ids = body.data.map(e => e.id);
// Aliases that must NOT appear
const knownAliases = ['sonnet', 'opus', 'haiku', 'claude', 'codex', 'devstral'];
for (const alias of knownAliases) {
assert.ok(!ids.includes(alias), `Alias '${alias}' must not appear in /v1/models data`);
}
// The canonical ID must appear
assert.ok(ids.includes('claude-sonnet-4-6'), 'claude-sonnet-4-6 must appear in /v1/models data');
} finally {
resetProviders17();
await new Promise(r => s.close(r));
}
});
// ── 17d: /v1/models entries match OpenAI spec shape ───────────────────────
it('17d: /v1/models entries match OpenAI spec shape (id/object/created/owned_by)', async () => {
setProviders17({ anthropic: true });
const s = createServer17();
const p = 25468 + Math.floor(Math.random() * 400);
await new Promise((resolve, reject) => {
s.listen(p, '127.0.0.1', resolve);
s.once('error', reject);
});
try {
const r = await fetch({ port: p, method: 'GET', path: '/v1/models' });
assert.equal(r.status, 200);
const body = JSON.parse(r.body);
assert.ok(body.data.length > 0, 'Expected at least one model entry');
for (const entry of body.data) {
// Must have spec-defined fields
assert.ok(typeof entry.id === 'string' && entry.id.length > 0, 'id must be a non-empty string');
assert.equal(entry.object, 'model', "object must be 'model'");
assert.ok(typeof entry.created === 'number' && entry.created > 0, 'created must be a positive number (Unix epoch seconds)');
assert.ok(typeof entry.owned_by === 'string' && entry.owned_by.length > 0, 'owned_by must be a non-empty string');
// Must NOT have invented fields beyond the spec
const allowedKeys = new Set(['id', 'object', 'created', 'owned_by']);
for (const key of Object.keys(entry)) {
assert.ok(allowedKeys.has(key), `Unexpected field '${key}' in /v1/models entry (not in OpenAI spec)`);
}
}
} finally {
resetProviders17();
await new Promise(r => s.close(r));
}
});
// ── 17e: chain-exhausted error response carries all 5 X-OLP-* headers ─────
it('17e: chain-exhausted 502 response carries all 5 standard X-OLP-* observability headers', async () => {
// Set up a 2-hop chain where both providers fail (SPAWN_FAILED via exit code 1).
// Both providers must be enabled for the chain to be built.
setProviders17({ anthropic: true, openai: true });
setFallbackConfig17({
chains: {
'claude-sonnet-4-6': [
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
{ provider: 'openai', model: 'gpt-5.5' },
],
},
soft_triggers: {},
providersEnabled: { anthropic: true, openai: true },
});
clearCache17();
// Fake auth for both providers
const savedToken = process.env.CLAUDE_CODE_OAUTH_TOKEN;
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-17e-anthropic';
const { writeFileSync: wfs17e, unlinkSync: uls17e } = await import('node:fs');
const { tmpdir: td17e } = await import('node:os');
const { join: pj17e } = await import('node:path');
const tmpAuth17e = pj17e(td17e(), `olp-test-17e-codex-${Date.now()}.json`);
wfs17e(tmpAuth17e, JSON.stringify({ accessToken: 'fake-codex-17e' }), 'utf8');
const savedCodexPath = process.env.OPENAI_CODEX_AUTH_PATH;
process.env.OPENAI_CODEX_AUTH_PATH = tmpAuth17e;
// Both providers: exit code 1 → SPAWN_FAILED hard trigger
function makeFailSpawn17() {
return function (_bin, _args, _opts) {
const proc = new EventEmitter();
proc.stdout = new EventEmitter();
proc.stderr = new EventEmitter();
proc.stdin = {
write: () => {},
end: () => {
setImmediate(() => {
proc.stdout.emit('end');
proc.stderr.emit('end');
proc.emit('close', 1, null);
});
},
};
proc.killed = false;
proc.kill = () => {};
return proc;
};
}
__setSpawnImpl(makeFailSpawn17());
codexSetSpawnImpl(makeFailSpawn17());
const s = createServer17();
const p = 25472 + Math.floor(Math.random() * 400);
await new Promise((resolve, reject) => {
s.listen(p, '127.0.0.1', resolve);
s.once('error', reject);
});
try {
const r = await fetch({
port: p,
method: 'POST',
path: '/v1/chat/completions',
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: 'hi' }] },
});
assert.ok(r.status >= 400 && r.status < 600, `Expected error status, got ${r.status}`);
// All 5 standard X-OLP-* headers must be present per ADR 0004 § Observability
assert.ok(r.headers['x-olp-provider-used'] !== undefined, 'X-OLP-Provider-Used must be present on error response');
assert.ok(r.headers['x-olp-model-used'] !== undefined, 'X-OLP-Model-Used must be present on error response');
assert.ok(r.headers['x-olp-fallback-hops'] !== undefined, 'X-OLP-Fallback-Hops must be present on error response');
assert.ok(r.headers['x-olp-cache'] !== undefined, 'X-OLP-Cache must be present on error response');
assert.ok(r.headers['x-olp-latency-ms'] !== undefined, 'X-OLP-Latency-Ms must be present on error response');
} finally {
resetProviders17();
resetFallbackConfig17();
__resetSpawnImpl();
codexResetSpawnImpl();
if (savedToken !== undefined) {
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken;
} else {
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
}
if (savedCodexPath !== undefined) {
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexPath;
} else {
delete process.env.OPENAI_CODEX_AUTH_PATH;
}
try { uls17e(tmpAuth17e); } catch { /* ignore */ }
await new Promise(r => s.close(r));
}
});
// ── 17f: chain-exhausted X-OLP-Fallback-Hops reflects hops attempted ─────
it('17f: chain-exhausted error X-OLP-Fallback-Hops reflects hops attempted (2-hop → "2")', async () => {
// 2-hop chain where both fail → fallbackHops should reflect 2 hops tried
setProviders17({ anthropic: true, openai: true });
setFallbackConfig17({
chains: {
'claude-sonnet-4-6': [
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
{ provider: 'openai', model: 'gpt-5.5' },
],
},
soft_triggers: {},
providersEnabled: { anthropic: true, openai: true },
});
clearCache17();
const savedToken = process.env.CLAUDE_CODE_OAUTH_TOKEN;
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'fake-token-17f-anthropic';
const { writeFileSync: wfs17f, unlinkSync: uls17f } = await import('node:fs');
const { tmpdir: td17f } = await import('node:os');
const { join: pj17f } = await import('node:path');
const tmpAuth17f = pj17f(td17f(), `olp-test-17f-codex-${Date.now()}.json`);
wfs17f(tmpAuth17f, JSON.stringify({ accessToken: 'fake-codex-17f' }), 'utf8');
const savedCodexPath17f = process.env.OPENAI_CODEX_AUTH_PATH;
process.env.OPENAI_CODEX_AUTH_PATH = tmpAuth17f;
function makeFailSpawn17f() {
return function (_bin, _args, _opts) {
const proc = new EventEmitter();
proc.stdout = new EventEmitter();
proc.stderr = new EventEmitter();
proc.stdin = {
write: () => {},
end: () => {
setImmediate(() => {
proc.stdout.emit('end');
proc.stderr.emit('end');
proc.emit('close', 1, null);
});
},
};
proc.killed = false;
proc.kill = () => {};
return proc;
};
}
__setSpawnImpl(makeFailSpawn17f());
codexSetSpawnImpl(makeFailSpawn17f());
const s = createServer17();
const p = 25876 + Math.floor(Math.random() * 400);
await new Promise((resolve, reject) => {
s.listen(p, '127.0.0.1', resolve);
s.once('error', reject);
});
try {
const r = await fetch({
port: p,
method: 'POST',
path: '/v1/chat/completions',
body: { model: 'claude-sonnet-4-6', messages: [{ role: 'user', content: 'hi' }] },
});
assert.ok(r.status >= 400, `Expected error status, got ${r.status}`);
// fallbackHops is set to chain.length (2) when chain exhausted per engine.mjs
assert.equal(r.headers['x-olp-fallback-hops'], '2',
`Expected X-OLP-Fallback-Hops: 2 (both hops tried), got: ${r.headers['x-olp-fallback-hops']}`);
} finally {
resetProviders17();
resetFallbackConfig17();
__resetSpawnImpl();
codexResetSpawnImpl();
if (savedToken !== undefined) {
process.env.CLAUDE_CODE_OAUTH_TOKEN = savedToken;
} else {
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
}
if (savedCodexPath17f !== undefined) {
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexPath17f;
} else {
delete process.env.OPENAI_CODEX_AUTH_PATH;
}
try { uls17f(tmpAuth17f); } catch { /* ignore */ }
await new Promise(r => s.close(r));
}
});
// ── 17g: pre-routing 400 (invalid JSON) carries X-OLP-Latency-Ms ─────────
it('17g: pre-routing 400 (invalid JSON body) carries X-OLP-Latency-Ms', async () => {
// Pre-routing errors inside handleChatCompletions (invalid JSON body, wrong
// Content-Type, IR parse failure) have access to startMs and emit
// X-OLP-Latency-Ms to allow latency instrumentation even when there is no
// provider context. The other 4 X-OLP-* headers are omitted (no provider was
// attempted). This is consistent with ADR 0004 § Observability which requires
// the full set for chain-exhausted paths; partial emission on pre-route errors
// is the minimum viable observability for those paths.
setProviders17({});
const s = createServer17();
const p = 25880 + Math.floor(Math.random() * 400);
await new Promise((resolve, reject) => {
s.listen(p, '127.0.0.1', resolve);
s.once('error', reject);
});
try {
const result = await new Promise((resolve, reject) => {
const req = httpRequest({
hostname: '127.0.0.1',
port: p,
method: 'POST',
path: '/v1/chat/completions',
headers: { 'Content-Type': 'application/json', 'Content-Length': '5' },
}, res => {
let data = '';
res.on('data', c => { data += c; });
res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: data }));
});
req.on('error', reject);
req.write('{bad}');
req.end();
});
assert.equal(result.status, 400, `Expected 400 for invalid JSON body`);
// X-OLP-Latency-Ms must be present (D18 pre-routing observability)
assert.ok(result.headers['x-olp-latency-ms'] !== undefined,
'Pre-routing 400 error must carry X-OLP-Latency-Ms for latency instrumentation');
const latencyMs = parseInt(result.headers['x-olp-latency-ms'], 10);
assert.ok(!isNaN(latencyMs) && latencyMs >= 0,
`X-OLP-Latency-Ms must be a non-negative integer, got: ${result.headers['x-olp-latency-ms']}`);
// The other 4 headers are NOT present (no provider context)
assert.ok(result.headers['x-olp-provider-used'] === undefined,
'Pre-routing 400 must NOT carry X-OLP-Provider-Used (no provider context)');
} finally {
resetProviders17();
await new Promise(r => s.close(r));
}
});
});
// ── Suite 16: Spawn timeout (P1.3) ───────────────────────────────────────────
//
// Tests that: