taodeng/olp
1
0
Fork 0
mirror of https://github.com/dtzp555-max/olp.git synced 2026-07-19 09:45:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(phase-1): land fallback engine (D9)

Browse Source 
Phase 1 Day 6 — the architectural milestone of Phase 1. OLP is no longer
a single-provider proxy with cache; it is now a multi-provider router
with idempotent-failure-safe fallback. ADR 0004 ratified in v0.1 governance
is fully implemented for D9 scope. Configuration-driven chains activate
when a user populates ~/.olp/config.json routing.chains; at v0.1 default
(empty chains), behaviour is single-hop pass-through identical to D5.

Files:
  NEW:  lib/fallback/engine.mjs (~470 lines) — Trigger taxonomy, chain
        advancement, first-chunk safety, observability annotation.
  MOD:  server.mjs (+80 lines net) — wires executeWithFallback between IR
        construction and provider.spawn. Per-hop cache key isolation
        (ADR 0005 § Cross-provider fallback). Test seams added:
        __setFallbackConfig, __resetFallbackConfig, __clearCache.
  MOD:  test-features.mjs (+845 lines, 6 new suites = +55 tests).

Authority citations:
  ADR 0004 § Decision § Trigger taxonomy — Hard / Soft / Deterministic
    (deferred) / Cost-aware (deferred) implemented exactly per spec.
  ADR 0004 § Decision § Fallback safety — first-chunk rule satisfied at
    D9 by buffering composition (executeHopFn = collectAllChunks; server
    writes to res only after engine returns).
  ADR 0004 § Decision § Chain advancement — one-at-a-time iteration;
    originalError preserved from FIRST hop (not last) on exhaustion.
  ADR 0004 § Decision § No fallback for client-side errors — HTTP 400/
    401/403/404/422 stop the chain immediately; AUTH_MISSING also stops
    immediately (user-config failure, not provider-quota).
  ADR 0004 § Decision § Observability headers — X-OLP-Fallback-Hops set
    from engine return value; X-OLP-Fallback-Exhausted lists tried
    providers on exhaustion.
  ADR 0005 § Cross-provider fallback cache behavior — each fallback hop
    computes a fresh cache key with the hop's (provider, model) tuple;
    primary's cache entries cannot leak to secondary's hop.

Reviewer chain (Iron Rule 10):
  Implementer: sonnet (general-purpose).
  Fresh-context reviewer: opus. Verdict APPROVE_WITH_MINOR.
  Reviewer ran npm test (277/277 pass), read ADR 0004 + ADR 0005 end-
    to-end, verified first-chunk-safety composition, checked all
    trigger taxonomy mappings.

Reviewer non-blocking findings folded in this commit:
  1. Test label at test-features.mjs:3756 clarified. Original title
     "client error (400) → does NOT fall back" was misleading because
     the test actually exercises both-fail SPAWN_FAILED → exhausted
     path (400-no-fallback semantic is covered at unit level instead).
     Renamed + commented to match actual behaviour.
  2. triedProviders semantic (includes soft-skipped) documented in
     engine.mjs comment near the soft-trigger branch.
  3. SOFT_TRIGGER synthetic code documented in engine.mjs as engine-
     internal, NOT a member of base.mjs PROVIDER_ERROR_CODES.

Reviewer non-blocking suggestions deferred (open D-later questions):
  - Q1 cacheStatus on fallback hops: conservative 'miss', acceptable
    per ADR 0005 § Per-model isolation.
  - Q2 X-OLP-Fallback-Exhausted only when triedProviders > 1:
    acceptable per ADR 0004 spec.
  - Q3 soft-trigger quotaSnapshot always null at D9: acceptable per
    ADR 0004 § Consequences/Negative graceful-degrade clause. Phase 2
    will add quota poll-worker.
  - Q4 loadFallbackConfigSync sync-only: acceptable for bounded
    startup I/O.

Test count: 222 (D8) → 277 (D9). 6 new test suites cover trigger
taxonomy, engine chain advancement (including the load-bearing
originalError-from-FIRST-hop property), HTTP integration through
real server.mjs, soft trigger pre-fetch path, exhausted-chain
header emission.

Verification:
  node --check on all touched files: clean.
  npm test on Node 25.8.0: 277/277 pass in 266ms.
  Hygiene grep: zero personal-name/path/token hits.
  No new external npm deps.

Co-Authored-By: Claude Opus 4.7 (noreply@anthropic.com)
This commit is contained in:
taodeng
2026-05-23 22:09:39 +10:00
co-authored by Claude Opus 4.7 (noreply@anthropic.com)
parent 95dc072245
commit 5960486542
3 changed files with 1502 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lib/fallback/engine.mjs
+473
View File
@@ -0,0 +1,473 @@
/**
* lib/fallback/engine.mjs — OLP Fallback Engine
*
* Authority: ADR 0004 — Fallback Engine Semantics and Safety
*
* Implements:
* - Trigger taxonomy (Hard / Soft per ADR 0004 § Decision § Trigger taxonomy)
* - Chain advancement one-at-a-time (ADR 0004 § Decision § Chain advancement)
* - First-chunk safety rule (ADR 0004 § Decision § Fallback safety)
* - Observability return values for header annotation (ADR 0004 § Decision § Observability headers)
* - No fallback for client-side errors (ADR 0004 § Decision § No fallback for client-side errors)
*
* Sealed at D9. Do NOT modify without an ADR 0004 amendment.
*/
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { join } from 'node:path';
import { ProviderError } from '../providers/base.mjs';
// ── Hard-trigger evaluation ────────────────────────────────────────────────
/**
* Maps ProviderError codes to hard-trigger decisions.
* Per ADR 0004 § Decision § Trigger taxonomy — Hard triggers:
* - QUOTA_EXHAUSTED → hard trigger
* - RATE_LIMITED → hard trigger
* - SPAWN_FAILED → hard trigger (provider CLI failed)
* - CLI_NOT_FOUND → hard trigger (binary missing)
* - AUTH_MISSING → NOT a hard trigger (user-config failure; user must fix)
* - OUTPUT_PARSE_ERROR → hard trigger
*
* @type {Record<string, boolean>}
*/
const HARD_TRIGGER_CODES = {
QUOTA_EXHAUSTED: true,
RATE_LIMITED: true,
SPAWN_FAILED: true,
CLI_NOT_FOUND: true,
AUTH_MISSING: false, // user config problem — never fall over (ADR 0004 § Decision)
OUTPUT_PARSE_ERROR: true,
};
/**
* HTTP status codes that are client errors and MUST NOT trigger fallback.
* Per ADR 0004 § Decision § "No fallback for client-side errors":
* HTTP 400, 401, 403, 404, 422 from a provider are NOT fallback triggers.
*
* @type {Set<number>}
*/
const CLIENT_ERROR_STATUSES = new Set([400, 401, 403, 404, 422]);
/**
* Returns true if `status` is a client-side error that must NOT trigger fallback.
* Exported for tests.
*
* @param {number} status
* @returns {boolean}
*/
export function isClientError(status) {
return CLIENT_ERROR_STATUSES.has(status);
}
/**
* Evaluates whether an error from a provider qualifies as a Hard trigger.
*
* Per ADR 0004 § Decision § Trigger taxonomy — Hard triggers:
* HTTP 5xx and quota-exhaustion 4xx (surfaced via ProviderError codes)
* trigger mandatory chain advancement.
*
* Per ADR 0004 § Decision § "No fallback for client-side errors":
* Client-side HTTP errors (400/401/403/404/422) are NOT hard triggers.
* AUTH_MISSING is also not a hard trigger (user must fix their config).
*
* @param {Error} error — the error thrown by executeHopFn
* @param {object} [_providerHints] — reserved for future hints-based logic
* @returns {boolean} — true if this error should trigger chain advancement
*/
export function evaluateHardTriggers(error, _providerHints = {}) {
if (!error) return false;
// ProviderError with a code: map via HARD_TRIGGER_CODES lookup.
// AUTH_MISSING is explicitly false in the table — do not fall over.
if (error instanceof ProviderError && error.code) {
return HARD_TRIGGER_CODES[error.code] === true;
}
// HTTP status code present on the error object.
const status = error.statusCode ?? error.status ?? null;
if (status !== null && typeof status === 'number') {
// Client-side errors: never fall over (ADR 0004 § No fallback for client-side errors)
if (CLIENT_ERROR_STATUSES.has(status)) {
return false;
}
// 5xx: always a hard trigger (ADR 0004 § Hard triggers: HTTP 5xx from provider)
if (status >= 500) {
return true;
}
// Non-client 4xx (e.g. 429 / 529): hard trigger (quota exhaustion semantics)
if (status >= 400) {
return true;
}
}
// Unknown error type — conservative default: do NOT trigger fallback.
// Per ADR 0004 § Alternatives considered (a): "any error" fallback was rejected.
return false;
}
// ── Soft-trigger evaluation ────────────────────────────────────────────────
/**
* Evaluates whether a chain hop's soft triggers fire against the current
* quota snapshot for that provider.
*
* Per ADR 0004 § Decision § Trigger taxonomy — Soft triggers:
* - credit_pool_percent_threshold: fires when percentUsed >= threshold
* - daily_request_count_threshold: fires when dailyCount >= threshold
* - five_hour_window_percent_threshold: fires when fiveHourWindowPercent >= threshold
*
* Per ADR 0004 § Consequences/Negative (Mitigations):
* null quotaStatus → treat as "don't fire" — degrade gracefully.
*
* @param {object|null|undefined} triggerConfig — per-hop soft-trigger config, e.g.
* { credit_pool_percent_threshold: 90, daily_request_count_threshold: 1000 }
* @param {object|null|undefined} quotaSnapshot — from provider.quotaStatus(), e.g.
* { percentUsed: 95, dailyCount: 500, fiveHourWindowPercent: 80 }
* May be null if the provider cannot retrieve quota.
* @returns {boolean} — true if any configured soft trigger fires
*/
export function evaluateSoftTriggers(triggerConfig, quotaSnapshot) {
// No trigger config at all → soft triggers never fire
if (!triggerConfig || typeof triggerConfig !== 'object') return false;
// Per ADR 0004 § Consequences/Negative: null quotaStatus → don't fire
if (quotaSnapshot == null) return false;
// credit_pool_percent_threshold: fires when percentUsed >= threshold
if (
triggerConfig.credit_pool_percent_threshold !== undefined &&
quotaSnapshot.percentUsed !== undefined &&
quotaSnapshot.percentUsed !== null &&
quotaSnapshot.percentUsed >= triggerConfig.credit_pool_percent_threshold
) {
return true;
}
// daily_request_count_threshold: fires when dailyCount >= threshold
if (
triggerConfig.daily_request_count_threshold !== undefined &&
quotaSnapshot.dailyCount !== undefined &&
quotaSnapshot.dailyCount !== null &&
quotaSnapshot.dailyCount >= triggerConfig.daily_request_count_threshold
) {
return true;
}
// five_hour_window_percent_threshold: fires when fiveHourWindowPercent >= threshold
if (
triggerConfig.five_hour_window_percent_threshold !== undefined &&
quotaSnapshot.fiveHourWindowPercent !== undefined &&
quotaSnapshot.fiveHourWindowPercent !== null &&
quotaSnapshot.fiveHourWindowPercent >= triggerConfig.five_hour_window_percent_threshold
) {
return true;
}
return false;
}
// ── Fallback engine ────────────────────────────────────────────────────────
/**
* @typedef {Object} ChainHop
* @property {string} provider — provider name (e.g. 'anthropic')
* @property {string} model — model string for this hop
* @property {object} [softTriggers] — optional soft-trigger config for this hop
* @property {object|null} [quotaSnapshot] — optional pre-fetched quota snapshot
*/
/**
* @typedef {Object} FallbackResult
* @property {Array<object>|null} chunks — IR chunk array on success; null if exhausted
* @property {string} providerUsed — provider name that served the request (or first that failed)
* @property {string} modelUsed — model string that served (or first that failed)
* @property {number} fallbackHops — chain index of the serving hop (0=primary, 1=first fallback, etc.)
* @property {Error|null} originalError — first-hop error if exhausted; null on success
* @property {string[]} triedProviders — all providers tried, in chain order
*/
/**
* Executes a provider chain with fallback semantics.
*
* Per ADR 0004 § Decision § Chain advancement (one-at-a-time):
* 1. Try each hop in order.
* 2. Before calling executeHopFn: evaluate soft triggers. If fired, skip this hop.
* 3. On error: check for client error / AUTH_MISSING (stop immediately).
* Otherwise evaluate hard triggers. If hard-triggered, advance to next hop.
* 4. If chain exhausted: return first-hop's originalError (not last).
*
* Per ADR 0004 § Decision § Fallback safety — first-chunk rule:
* At D9, executeHopFn is collectAllChunks() — fully buffered before return.
* An exception from executeHopFn means zero bytes were written to the client.
* First-chunk safety is trivially satisfied by the buffering pattern.
*
* @param {ChainHop[]} chain — ordered list of hops
* @param {object} irRequest — IR request object (ADR 0003)
* @param {function} executeHopFn — async (provider, model, ir) => IR chunk array; throws on failure
* @param {object} [options]
* @param {function} [options.logEvent] — optional structured logger (level, event, data)
* @returns {Promise<FallbackResult>}
*/
export async function executeWithFallback(chain, irRequest, executeHopFn, options = {}) {
const { logEvent = () => {} } = options;
if (!Array.isArray(chain) || chain.length === 0) {
throw new Error('executeWithFallback: chain must be a non-empty array');
}
const triedProviders = [];
let originalError = null; // Per ADR 0004: first-hop error is the canonical signal
let firstErrorRecorded = false;
for (let i = 0; i < chain.length; i++) {
const hop = chain[i];
const { provider, model, softTriggers = null, quotaSnapshot = null } = hop;
// ── Soft-trigger check (BEFORE spawn) ──────────────────────────────
// Per ADR 0004 § Decision: soft triggers evaluated before spawn;
// if fired, advance without attempting this provider at all.
//
// Note on `triedProviders` semantics (D9 review-2 finding): we push the
// provider name onto triedProviders even when soft-skipped (no spawn).
// ADR 0004 § Observability headers describes X-OLP-Fallback-Exhausted as
// "tried" providers; per ADR 0004 § Soft triggers a soft-fired hop is
// counted as tried-and-skipped (the proxy advances to the next chain
// entry without attempting the primary at all). The header therefore
// includes soft-skipped hops. A future enhancement could split into
// `triedProviders` + `softSkippedProviders` for finer-grained debug
// visibility; D9 keeps them unified per ADR-0004 spec.
if (evaluateSoftTriggers(softTriggers, quotaSnapshot)) {
logEvent('info', 'fallback_soft_trigger', {
hop: i,
provider,
model,
reason: 'soft_trigger_fired',
});
triedProviders.push(provider);
// Record a synthetic originalError for the first fired soft trigger.
// `code: 'SOFT_TRIGGER'` is a fallback-engine-internal marker, NOT a
// member of PROVIDER_ERROR_CODES (base.mjs) — those are provider
// plugin error codes. Downstream consumers inspecting originalError.code
// should treat 'SOFT_TRIGGER' as engine-synthetic. D9 review-2 noted
// this; documented here so future readers do not try to add SOFT_TRIGGER
// to the PROVIDER_ERROR_CODES closed enum.
if (!firstErrorRecorded) {
originalError = Object.assign(
new Error(`Soft trigger fired for provider ${provider}: quota threshold exceeded`),
{ code: 'SOFT_TRIGGER', provider },
);
firstErrorRecorded = true;
}
continue;
}
// ── Execute hop ────────────────────────────────────────────────────
triedProviders.push(provider);
try {
const chunks = await executeHopFn(provider, model, irRequest);
// Per ADR 0004 § Fallback safety — first-chunk rule:
// executeHopFn returned successfully = chunks buffered = no client writes
// blocked = safe to return. Any retry attempt after this point would be
// post-first-chunk and is therefore forbidden. We return immediately.
logEvent('info', 'fallback_hop_success', { hop: i, provider, model });
return {
chunks,
providerUsed: provider,
modelUsed: model,
fallbackHops: i,
originalError: null,
triedProviders,
};
} catch (err) {
// Record FIRST hop error as the canonical signal
// Per ADR 0004 § Chain advancement step 4: return first error, not last.
if (!firstErrorRecorded) {
originalError = err;
firstErrorRecorded = true;
}
logEvent('warn', 'fallback_hop_error', {
hop: i,
provider,
model,
error: err.message,
code: err.code ?? null,
});
// ── Client error: stop immediately ─────────────────────────────
// Per ADR 0004 § "No fallback for client-side errors"
const status = err.statusCode ?? err.status ?? null;
if (status !== null && CLIENT_ERROR_STATUSES.has(status)) {
logEvent('warn', 'fallback_client_error_no_fallback', { hop: i, provider, status });
return {
chunks: null,
providerUsed: provider,
modelUsed: model,
fallbackHops: i,
originalError: err,
triedProviders,
};
}
// ── AUTH_MISSING: stop immediately ─────────────────────────────
// Per ADR 0004 § Decision: AUTH_MISSING = HARD_TRIGGER_CODES[AUTH_MISSING]=false.
// Silently masking it by trying next provider prevents user from discovering
// the misconfiguration.
if (err instanceof ProviderError && err.code === 'AUTH_MISSING') {
logEvent('warn', 'fallback_auth_missing_no_fallback', { hop: i, provider });
return {
chunks: null,
providerUsed: provider,
modelUsed: model,
fallbackHops: i,
originalError: err,
triedProviders,
};
}
// ── Hard-trigger check: advance chain ──────────────────────────
// Per ADR 0004 § Decision § Fallback safety — first-chunk rule:
// At D9, executeHopFn threw, so zero chunks were emitted.
// Fallback is eligible if the error is a hard trigger.
if (evaluateHardTriggers(err)) {
logEvent('info', 'fallback_hard_trigger', {
hop: i,
provider,
model,
error: err.message,
code: err.code ?? null,
advance_to_hop: i + 1,
});
continue; // advance to next hop
}
// ── Non-trigger error: surface immediately ──────────────────────
// Per ADR 0004 § Alternatives considered (a): "any error" fallback rejected.
logEvent('warn', 'fallback_non_trigger_error', {
hop: i,
provider,
error: err.message,
code: err.code ?? null,
});
return {
chunks: null,
providerUsed: provider,
modelUsed: model,
fallbackHops: i,
originalError: err,
triedProviders,
};
}
}
// Chain exhausted
// Per ADR 0004 § Decision § Chain advancement step 4:
// "return A's original error (not B's, not C's)"
logEvent('error', 'fallback_chain_exhausted', {
chain: chain.map(h => h.provider),
triedProviders,
});
return {
chunks: null,
providerUsed: chain[0].provider,
modelUsed: chain[0].model,
fallbackHops: chain.length,
originalError,
triedProviders,
};
}
// ── Chain builder ──────────────────────────────────────────────────────────
/**
* Builds the execution chain for a given model string from config.
*
* Per D9 implementation notes:
* - If fallbackChainConfig has an explicit entry for this model, use it.
* - Otherwise, use a single-hop chain with the first matched provider (no fallback).
* - v0.1 ships with empty chain config; fallback activates only when the user
* populates ~/.olp/config.json's routing.chains.
*
* @param {string} modelString — user-requested model string
* @param {Map<string, object>} loadedProviders — name → provider plugin
* @param {object} [fallbackChainConfig] — routing.chains from config.json
* @param {object} [softTriggerConfig] — routing.soft_triggers from config.json
* @returns {ChainHop[]|null} — ordered chain, or null if no provider found for model
*/
export function buildDefaultChain(
modelString,
loadedProviders,
fallbackChainConfig = {},
softTriggerConfig = {},
) {
// Check for explicit chain in config for this model
const explicitChain = fallbackChainConfig[modelString];
if (explicitChain && Array.isArray(explicitChain) && explicitChain.length > 0) {
return explicitChain.map(hop => ({
provider: hop.provider,
model: hop.model ?? modelString,
softTriggers: softTriggerConfig[hop.provider] ?? null,
quotaSnapshot: null, // populated at runtime if provider.quotaStatus() is called
}));
}
// No explicit chain: find first loaded provider that serves this model
for (const [name, provider] of loadedProviders.entries()) {
if (provider.models && provider.models.includes(modelString)) {
return [{
provider: name,
model: modelString,
softTriggers: softTriggerConfig[name] ?? null,
quotaSnapshot: null,
}];
}
}
// No provider found for this model
return null;
}
// ── Config loader ──────────────────────────────────────────────────────────
/**
* Default path for the OLP config file.
* @returns {string}
*/
function defaultConfigPath() {
return join(homedir(), '.olp', 'config.json');
}
/**
* Loads fallback chain configuration from ~/.olp/config.json (or the given path).
*
* Per D9 implementation notes:
* v0.1 ships with no config file. Fallback engine is wired but single-hop
* until the user populates routing.chains.
*
* Returns empty config (no chains, no soft triggers) if the file is absent,
* unreadable, or malformed.
*
* @param {string} [configPath] — override path (for testing — do NOT write to ~/.olp/config.json in tests)
* @returns {{ chains: object, soft_triggers: object }}
*/
export function loadFallbackConfigSync(configPath) {
try {
const path = configPath ?? defaultConfigPath();
const raw = readFileSync(path, 'utf8');
const parsed = JSON.parse(raw);
const routing = parsed?.routing ?? {};
return {
chains: routing.chains ?? {},
soft_triggers: routing.soft_triggers ?? {},
};
} catch {
// File absent, unreadable, or malformed → no fallback config (single-hop mode)
return { chains: {}, soft_triggers: {} };
}
}
server.mjs
+168 -87
View File
@@ -33,6 +33,12 @@ import { loadProviders, getProviderForModel, listAllProviderNames } from './lib/
import { ProviderError } from './lib/providers/base.mjs';
import { computeCacheKey, hasCacheControl, extractCacheControlMarkers } from './lib/cache/keys.mjs';
import { CacheStore } from './lib/cache/store.mjs';
import {
evaluateHardTriggers,
executeWithFallback,
buildDefaultChain,
loadFallbackConfigSync,
} from './lib/fallback/engine.mjs';
// ── Config ────────────────────────────────────────────────────────────────
@@ -48,6 +54,27 @@ const BODY_LIMIT = 5 * 1024 * 1024; // 5 MB
// Empty config → empty loaded map → all POST /v1/chat/completions → 503.
const loadedProviders = loadProviders({ enabled: {} });
// ── Fallback config ───────────────────────────────────────────────────────
// Read ~/.olp/config.json routing.chains at startup. Empty at v0.1.
// Per ADR 0004 § D9: fallback engine is wired; activates when user populates chains.
// Tests may inject a synthetic fallbackConfig via __setFallbackConfig().
let _fallbackConfig = loadFallbackConfigSync();
/** @internal — test seam: inject a synthetic fallback config (no file I/O) */
export function __setFallbackConfig(config) {
_fallbackConfig = config ?? { chains: {}, soft_triggers: {} };
}
/** @internal — reset to file-based config */
export function __resetFallbackConfig() {
_fallbackConfig = loadFallbackConfigSync();
}
/** @internal — clear the cache store (for tests that need a fresh cache state) */
export function __clearCache() {
cacheStore.clear();
}
// ── Cache layer ───────────────────────────────────────────────────────────
// D1 per-key isolation + D4 singleflight per ADR 0005.
// keyId: '__anonymous__' at D5 — Phase 2 multi-key infrastructure wires in
@@ -132,9 +159,10 @@ function sendError(res, status, message, type) {
/**
* Returns the standard OLP diagnostic headers.
* Per spec: X-OLP-Provider-Used, X-OLP-Model-Used, X-OLP-Fallback-Hops,
* Per spec §4.7 and ADR 0004 § Observability headers:
* X-OLP-Provider-Used, X-OLP-Model-Used, X-OLP-Fallback-Hops,
* X-OLP-Cache, X-OLP-Latency-Ms.
* Fallback-Hops is always 0 at D5 (no fallback engine yet — ADR 0004).
* Fallback-Hops reflects which chain index served the request (0=primary).
* Cache reflects actual hit/miss/bypass status from the cache layer (ADR 0005).
*
* @param {object} opts
@@ -142,13 +170,14 @@ function sendError(res, status, message, type) {
* @param {string} opts.modelUsed
* @param {number} opts.startMs
* @param {'hit'|'miss'|'bypass'} [opts.cacheStatus='miss']
* @param {number} [opts.fallbackHops=0] — from executeWithFallback result
* @returns {Record<string,string>}
*/
function olpHeaders({ providerUsed, modelUsed, startMs, cacheStatus = 'miss' }) {
function olpHeaders({ providerUsed, modelUsed, startMs, cacheStatus = 'miss', fallbackHops = 0 }) {
return {
'X-OLP-Provider-Used': providerUsed,
'X-OLP-Model-Used': modelUsed,
'X-OLP-Fallback-Hops': '0',
'X-OLP-Fallback-Hops': String(fallbackHops),
'X-OLP-Cache': cacheStatus,
'X-OLP-Latency-Ms': String(Date.now() - startMs),
};
@@ -181,7 +210,13 @@ function handleModels(req, res) {
/**
* POST /v1/chat/completions
* Core dispatch path: OpenAI request → IR → provider.spawn → OpenAI response.
* Core dispatch path: OpenAI request → IR → fallback engine → provider.spawn → OpenAI response.
*
* D9: Fallback engine (ADR 0004) is wired between IR construction and provider.spawn.
* Chain advancement, soft/hard trigger evaluation, and first-chunk safety are all
* handled by executeWithFallback(). At v0.1 with empty routing.chains config, this
* is a transparent single-hop pass-through. Multi-hop fallback activates when the
* user populates ~/.olp/config.json.
*
* @param {import('node:http').IncomingMessage} req
* @param {import('node:http').ServerResponse} res
@@ -202,7 +237,7 @@ async function handleChatCompletions(req, res) {
return sendError(res, e.statusCode ?? 400, e.message, 'invalid_request_error');
}
// Translate OpenAI → IR
// Translate OpenAI → IR (ADR 0003)
let ir;
try {
ir = openAIToIR(body);
@@ -213,9 +248,23 @@ async function handleChatCompletions(req, res) {
throw e;
}
// Find a provider for the requested model
const match = getProviderForModel(loadedProviders, ir.model);
if (!match) {
// Auth context is null at D5/D9 — providers fall back to their own credential
// discovery (env var, keychain, credentials file). Phase 2 multi-key
// infrastructure will pass a real authContext carrying the per-key OLP token.
const authContext = null;
// ── Fallback engine: build chain (ADR 0004) ─────────────────────────────
// buildDefaultChain returns null if no enabled provider serves this model.
// Per ADR 0004 § D9: at v0.1, chain is single-hop (no fallback) unless the
// user has populated ~/.olp/config.json routing.chains.
const chain = buildDefaultChain(
ir.model,
loadedProviders,
_fallbackConfig.chains,
_fallbackConfig.soft_triggers,
);
if (!chain) {
// ALIGNMENT.md: 0 Enabled Providers at v0.1 → 503 per spec
return sendError(
res, 503,
@@ -224,49 +273,50 @@ async function handleChatCompletions(req, res) {
);
}
const { provider, name: providerName } = match;
const requestId = generateRequestId();
// Auth context is null at D5 — providers fall back to their own credential
// discovery (env var, keychain, credentials file). Phase 2 multi-key
// infrastructure will pass a real authContext carrying the per-key OLP token.
const authContext = null;
// ── Cache layer (ADR 0005) ──────────────────────────────────────────────
// keyId: '__anonymous__' at D5. Phase 2 multi-key infrastructure wires the
// keyId: '__anonymous__' at D5/D9. Phase 2 multi-key infrastructure wires the
// real OLP API key ID here for D1 per-key isolation.
const keyId = '__anonymous__';
const cacheKey = computeCacheKey(providerName, ir.model, ir);
// D2 bypass: if the request contains Anthropic cache_control markers,
// skip OLP's response cache entirely (the prompt cache lives at Anthropic's
// side; double-caching would shadow Anthropic's TTLs per ADR 0005 § D2).
//
// Note: hasCacheControl() checks the IR (ADR 0003). Since openAIToIR() does not
// preserve cache_control fields from the raw OpenAI message shape (those fields
// are Anthropic-specific extensions, not part of the IR schema), we also check
// the raw body.messages directly via extractCacheControlMarkers. This ensures
// D2 bypass is triggered even though the IR translator strips the field.
// skip OLP's response cache (prompt cache lives at Anthropic's side per ADR 0005 § D2).
const bypassCache = hasCacheControl(ir) || extractCacheControlMarkers(body?.messages ?? []).length > 0;
if (bypassCache) {
logEvent('debug', 'cache_bypass', { provider: providerName, model: ir.model, reason: 'cache_control_markers' });
logEvent('debug', 'cache_bypass', { model: ir.model, reason: 'cache_control_markers' });
}
// ── Collect chunks helper (used by both streaming and non-streaming paths) ──
// collectAllChunks wraps provider.spawn() to collect all IR chunks into an
// array. Used as the computeFn for getOrCompute (D4 singleflight).
// ── executeHopFn: per-hop spawn + cache wrapper ─────────────────────────
// This is the function executeWithFallback calls for each chain hop.
// Each hop gets its own (provider, model) cache key per ADR 0005 § Per-model isolation.
//
// Error semantics: if the provider emits a `type: 'error'` chunk, we throw
// a ProviderError instead of returning the chunk array. This prevents
// cache_store.set() from being called on an error-terminated response, per
// ADR 0005 § "Cache write conditions" item 1 ("The response completed
// successfully (no truncation, no error mid-stream)"). The thrown error
// propagates to the catch block at the call site, which returns 502 to
// the client without writing to cache.
// First-chunk safety (ADR 0004 § Fallback safety):
// collectAllChunks fully buffers the provider response before returning.
// Therefore, if executeHopFn throws, zero bytes have been written to `res`.
// The fallback engine safely advances the chain on hard triggers.
// If executeHopFn returns successfully, the chunks are buffered and we write
// them to `res` only AFTER executeWithFallback returns — ensuring no writes
// occur during chain iteration.
async function executeHopFn(hopProvider, hopModel, irReq) {
const hopCacheKey = computeCacheKey(hopProvider, hopModel, irReq);
const hopProviderPlugin = loadedProviders.get(hopProvider);
if (!hopProviderPlugin) {
// Provider in the chain is not loaded (config references a disabled provider)
throw Object.assign(
new Error(`Provider ${hopProvider} is not enabled`),
{ statusCode: 503 },
);
}
// Collect all chunks from this provider, throwing on error chunks.
// Error semantics: ProviderError thrown here propagates to executeWithFallback
// which decides whether to advance the chain.
async function collectAllChunks() {
const chunks = [];
for await (const irChunk of provider.spawn(ir, authContext)) {
for await (const irChunk of hopProviderPlugin.spawn(irReq, authContext)) {
chunks.push(irChunk);
if (irChunk.type === 'error') {
throw new ProviderError(
@@ -279,36 +329,92 @@ async function handleChatCompletions(req, res) {
return chunks;
}
// Pre-check: stats-neutral peek before calling getOrCompute so we can
// reliably report hit/miss in the response header. peek() does NOT touch
// hit/miss counters (fix for codex-flagged double-count); getOrCompute()
// increments at the semantic boundary.
const preCheckHit = bypassCache ? false : await cacheStore.peek(keyId, cacheKey);
if (ir.stream) {
// Streaming response path
// D5 simplified D3: cache stores full collected chunks, replays them one
// at a time without timing fidelity. Full D3 (timing-accurate replay) lands
// in a later Phase per ADR 0005 § D3.
let chunks;
let cacheStatus;
try {
if (bypassCache) {
cacheStatus = 'bypass';
chunks = await collectAllChunks();
} else {
// D4 singleflight + D1 per-key isolation
chunks = await cacheStore.getOrCompute(keyId, cacheKey, collectAllChunks);
cacheStatus = preCheckHit ? 'hit' : 'miss';
return collectAllChunks();
}
// D4 singleflight + D1 per-key isolation per ADR 0005.
// Each hop has its own (provider, model) key — cross-provider contamination
// is structurally impossible (ADR 0005 § Per-model isolation).
return cacheStore.getOrCompute(keyId, hopCacheKey, collectAllChunks);
}
// ── Execute with fallback (ADR 0004) ────────────────────────────────────
// Pre-check for cache status reporting uses first hop's key (primary provider).
const firstHopCacheKey = computeCacheKey(chain[0].provider, chain[0].model, ir);
const preCheckHit = bypassCache ? false : await cacheStore.peek(keyId, firstHopCacheKey);
let fallbackResult;
try {
fallbackResult = await executeWithFallback(chain, ir, executeHopFn, {
logEvent,
});
} catch (e) {
logEvent('error', 'spawn_error', { provider: providerName, model: ir.model, error: e.message });
sendError(res, e instanceof ProviderError ? 502 : 500, e.message ?? 'Provider error', 'provider_error');
// executeWithFallback throws only on programming errors (empty chain).
logEvent('error', 'fallback_engine_error', { error: e.message });
return sendError(res, 500, 'Internal server error', 'internal_error');
}
const {
chunks,
providerUsed,
modelUsed,
fallbackHops,
originalError,
triedProviders,
} = fallbackResult;
// ── Chain exhausted or non-trigger error ─────────────────────────────────
if (chunks === null) {
logEvent('error', 'spawn_error', {
model: ir.model,
providerUsed,
fallbackHops,
triedProviders,
error: originalError?.message,
});
// Emit exhausted header if more than one provider was tried
const exhaustedHeader = triedProviders.length > 1
? { 'X-OLP-Fallback-Exhausted': triedProviders.join(',') }
: {};
// Determine status: preserve client errors (400/401/403/404/422) as-is.
// Otherwise map ProviderError → 502, unknown → 500.
let errStatus = 502;
if (originalError) {
const httpStatus = originalError.statusCode ?? originalError.status ?? null;
if (httpStatus !== null) {
errStatus = httpStatus;
} else if (!(originalError instanceof ProviderError)) {
errStatus = 500;
}
}
// Send error with exhausted header
const payload = JSON.stringify({
error: {
message: originalError?.message ?? 'Provider error',
type: 'provider_error',
},
});
res.writeHead(errStatus, {
'Content-Type': 'application/json',
'Content-Length': Buffer.byteLength(payload),
...exhaustedHeader,
});
res.end(payload);
return;
}
const headers = olpHeaders({ providerUsed: providerName, modelUsed: ir.model, startMs, cacheStatus });
// ── Success: emit response ─────────────────────────────────────────────
// Per ADR 0004 § Observability headers: X-OLP-Fallback-Hops reflects the
// chain index of the serving hop; 0 = primary served, 1 = first fallback, etc.
const cacheStatus = bypassCache ? 'bypass' : (preCheckHit && fallbackHops === 0 ? 'hit' : 'miss');
const headers = olpHeaders({ providerUsed, modelUsed, startMs, cacheStatus, fallbackHops });
if (ir.stream) {
// Streaming response path (D3 simplified: burst replay, no timing fidelity)
res.writeHead(200, {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
@@ -317,7 +423,6 @@ async function handleChatCompletions(req, res) {
...headers,
});
// D3 simplified replay: emit collected chunks sequentially
for (const irChunk of chunks) {
res.write(irChunkToOpenAISSE(irChunk, requestId, ir.model));
if (irChunk.type === 'stop' || irChunk.type === 'error') break;
@@ -326,31 +431,7 @@ async function handleChatCompletions(req, res) {
res.end();
} else {
// Non-streaming response path
let cacheStatus;
let responseObj;
try {
if (bypassCache) {
cacheStatus = 'bypass';
const chunks = await collectAllChunks();
responseObj = irResponseToOpenAINonStream(chunks, requestId, ir.model);
} else {
// D4 singleflight: concurrent identical requests share one spawn.
// We cache the full IR chunk array and assemble the OpenAI response on
// each retrieval so the requestId is fresh per request (per OpenAI spec
// each response has a unique id).
const chunks = await cacheStore.getOrCompute(keyId, cacheKey, collectAllChunks);
cacheStatus = preCheckHit ? 'hit' : 'miss';
responseObj = irResponseToOpenAINonStream(chunks, requestId, ir.model);
}
} catch (e) {
logEvent('error', 'spawn_error', { provider: providerName, model: ir.model, error: e.message });
const status = e instanceof ProviderError ? 502 : 500;
sendError(res, status, e.message ?? 'Provider error', 'provider_error');
return;
}
const headers = olpHeaders({ providerUsed: providerName, modelUsed: ir.model, startMs, cacheStatus });
const responseObj = irResponseToOpenAINonStream(chunks, requestId, ir.model);
sendJSON(res, 200, responseObj, headers);
}
}
test-features.mjs
+845
View File
@@ -2999,3 +2999,848 @@ describe('Mistral Vibe plugin (D8)', () => {
});
});
// ── Suite 13: Fallback engine (D9) ───────────────────────────────────────
//
// All tests are unit tests. No real provider CLIs are invoked.
// Tests cover:
// 13a: Trigger taxonomy (evaluateHardTriggers / evaluateSoftTriggers)
// 13b: executeWithFallback engine behaviour
// 13c: First-chunk safety (buffering semantics at D9)
// 13d: Soft trigger skipping spawn
// 13e: Header annotation (providerUsed / modelUsed / fallbackHops / originalError)
// 13f: HTTP integration with __setFallbackConfig test seam
//
// Authority: ADR 0004 — Fallback Engine Semantics and Safety
import {
evaluateHardTriggers,
evaluateSoftTriggers,
executeWithFallback,
buildDefaultChain,
loadFallbackConfigSync,
isClientError,
} from './lib/fallback/engine.mjs';
import {
createOlpServer,
__setFallbackConfig,
__resetFallbackConfig,
__clearCache,
} from './server.mjs';
// ── 13a: Trigger taxonomy ────────────────────────────────────────────────
describe('Fallback engine — trigger taxonomy (D9)', () => {
// ── evaluateHardTriggers ─────────────────────────────────────────────
it('evaluateHardTriggers: HTTP 500 → fires (5xx hard trigger)', () => {
const err = Object.assign(new Error('Server error'), { statusCode: 500 });
assert.equal(evaluateHardTriggers(err), true);
});
it('evaluateHardTriggers: HTTP 503 → fires (5xx hard trigger)', () => {
const err = Object.assign(new Error('Service unavailable'), { statusCode: 503 });
assert.equal(evaluateHardTriggers(err), true);
});
it('evaluateHardTriggers: HTTP 400 → does NOT fire (client error)', () => {
const err = Object.assign(new Error('Bad request'), { statusCode: 400 });
assert.equal(evaluateHardTriggers(err), false);
});
it('evaluateHardTriggers: HTTP 401 → does NOT fire (client error)', () => {
const err = Object.assign(new Error('Unauthorized'), { statusCode: 401 });
assert.equal(evaluateHardTriggers(err), false);
});
it('evaluateHardTriggers: HTTP 403 → does NOT fire (client error)', () => {
const err = Object.assign(new Error('Forbidden'), { statusCode: 403 });
assert.equal(evaluateHardTriggers(err), false);
});
it('evaluateHardTriggers: HTTP 404 → does NOT fire (client error)', () => {
const err = Object.assign(new Error('Not found'), { statusCode: 404 });
assert.equal(evaluateHardTriggers(err), false);
});
it('evaluateHardTriggers: HTTP 422 → does NOT fire (client error)', () => {
const err = Object.assign(new Error('Unprocessable'), { statusCode: 422 });
assert.equal(evaluateHardTriggers(err), false);
});
it('evaluateHardTriggers: HTTP 429 → fires (quota-like 4xx not in client-error set)', () => {
const err = Object.assign(new Error('Too Many Requests'), { statusCode: 429 });
assert.equal(evaluateHardTriggers(err), true);
});
it('evaluateHardTriggers: ProviderError QUOTA_EXHAUSTED → fires', () => {
const err = new ProviderError('Quota exhausted', 'QUOTA_EXHAUSTED');
assert.equal(evaluateHardTriggers(err), true);
});
it('evaluateHardTriggers: ProviderError RATE_LIMITED → fires', () => {
const err = new ProviderError('Rate limited', 'RATE_LIMITED');
assert.equal(evaluateHardTriggers(err), true);
});
it('evaluateHardTriggers: ProviderError SPAWN_FAILED → fires', () => {
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
assert.equal(evaluateHardTriggers(err), true);
});
it('evaluateHardTriggers: ProviderError CLI_NOT_FOUND → fires', () => {
const err = new ProviderError('CLI not found', 'CLI_NOT_FOUND');
assert.equal(evaluateHardTriggers(err), true);
});
it('evaluateHardTriggers: ProviderError AUTH_MISSING → does NOT fire (user must fix)', () => {
const err = new ProviderError('Auth missing', 'AUTH_MISSING');
assert.equal(evaluateHardTriggers(err), false);
});
it('evaluateHardTriggers: ProviderError OUTPUT_PARSE_ERROR → fires', () => {
const err = new ProviderError('Parse error', 'OUTPUT_PARSE_ERROR');
assert.equal(evaluateHardTriggers(err), true);
});
it('evaluateHardTriggers: generic Error with no statusCode → does NOT fire', () => {
const err = new Error('Something went wrong');
assert.equal(evaluateHardTriggers(err), false);
});
it('evaluateHardTriggers: null error → does NOT fire', () => {
assert.equal(evaluateHardTriggers(null), false);
});
// ── evaluateSoftTriggers ─────────────────────────────────────────────
it('evaluateSoftTriggers: credit_pool_percent_threshold at exactly threshold → fires', () => {
const cfg = { credit_pool_percent_threshold: 90 };
const quota = { percentUsed: 90 };
assert.equal(evaluateSoftTriggers(cfg, quota), true);
});
it('evaluateSoftTriggers: credit_pool_percent_threshold above threshold → fires', () => {
const cfg = { credit_pool_percent_threshold: 90 };
const quota = { percentUsed: 95 };
assert.equal(evaluateSoftTriggers(cfg, quota), true);
});
it('evaluateSoftTriggers: credit_pool_percent_threshold below threshold → does NOT fire', () => {
const cfg = { credit_pool_percent_threshold: 90 };
const quota = { percentUsed: 89 };
assert.equal(evaluateSoftTriggers(cfg, quota), false);
});
it('evaluateSoftTriggers: daily_request_count_threshold at threshold → fires', () => {
const cfg = { daily_request_count_threshold: 500 };
const quota = { dailyCount: 500 };
assert.equal(evaluateSoftTriggers(cfg, quota), true);
});
it('evaluateSoftTriggers: daily_request_count_threshold below threshold → does NOT fire', () => {
const cfg = { daily_request_count_threshold: 500 };
const quota = { dailyCount: 499 };
assert.equal(evaluateSoftTriggers(cfg, quota), false);
});
it('evaluateSoftTriggers: five_hour_window_percent_threshold at threshold → fires', () => {
const cfg = { five_hour_window_percent_threshold: 85 };
const quota = { fiveHourWindowPercent: 85 };
assert.equal(evaluateSoftTriggers(cfg, quota), true);
});
it('evaluateSoftTriggers: null quotaSnapshot → does NOT fire (graceful degrade)', () => {
const cfg = { credit_pool_percent_threshold: 90 };
assert.equal(evaluateSoftTriggers(cfg, null), false);
});
it('evaluateSoftTriggers: undefined quotaSnapshot → does NOT fire (graceful degrade)', () => {
const cfg = { credit_pool_percent_threshold: 90 };
assert.equal(evaluateSoftTriggers(cfg, undefined), false);
});
it('evaluateSoftTriggers: null triggerConfig → does NOT fire', () => {
const quota = { percentUsed: 95 };
assert.equal(evaluateSoftTriggers(null, quota), false);
});
it('evaluateSoftTriggers: empty triggerConfig → does NOT fire', () => {
const quota = { percentUsed: 95 };
assert.equal(evaluateSoftTriggers({}, quota), false);
});
// ── isClientError ────────────────────────────────────────────────────
it('isClientError: 400 → true', () => { assert.equal(isClientError(400), true); });
it('isClientError: 401 → true', () => { assert.equal(isClientError(401), true); });
it('isClientError: 403 → true', () => { assert.equal(isClientError(403), true); });
it('isClientError: 404 → true', () => { assert.equal(isClientError(404), true); });
it('isClientError: 422 → true', () => { assert.equal(isClientError(422), true); });
it('isClientError: 429 → false (quota, not client error)', () => { assert.equal(isClientError(429), false); });
it('isClientError: 500 → false', () => { assert.equal(isClientError(500), false); });
});
// ── 13b: executeWithFallback engine ─────────────────────────────────────
describe('Fallback engine — executeWithFallback (D9)', () => {
// helper: build a mock executeHopFn that succeeds or throws per provider name
function makeHopFn(outcomes) {
// outcomes: { [provider]: 'success' | Error }
return async function (provider, model, _ir) {
const outcome = outcomes[provider];
if (!outcome || outcome === 'success') {
return [{ type: 'delta', role: 'assistant', content: `response from ${provider}` },
{ type: 'stop', finish_reason: 'stop' }];
}
throw outcome;
};
}
const dummyIR = makeIR({ model: 'test-model' });
it('empty chain → throws Error', async () => {
let caught = null;
try {
await executeWithFallback([], dummyIR, makeHopFn({}));
} catch (e) {
caught = e;
}
assert.ok(caught instanceof Error, 'Expected Error for empty chain');
assert.ok(caught.message.includes('chain'));
});
it('single-hop chain with success → returns chunks + fallbackHops=0', async () => {
const chain = [{ provider: 'anthropic', model: 'claude-sonnet-4-6' }];
const result = await executeWithFallback(chain, dummyIR, makeHopFn({ anthropic: 'success' }));
assert.ok(Array.isArray(result.chunks), 'Expected chunks array');
assert.equal(result.fallbackHops, 0);
assert.equal(result.providerUsed, 'anthropic');
assert.equal(result.originalError, null);
assert.deepEqual(result.triedProviders, ['anthropic']);
});
it('single-hop chain with hard-triggered error → exhausted, returns originalError', async () => {
const err = new ProviderError('Quota exhausted', 'QUOTA_EXHAUSTED');
const chain = [{ provider: 'anthropic', model: 'claude-sonnet-4-6' }];
const result = await executeWithFallback(chain, dummyIR, makeHopFn({ anthropic: err }));
assert.equal(result.chunks, null, 'Expected null chunks on exhausted chain');
assert.equal(result.originalError, err);
assert.equal(result.fallbackHops, 1); // chain.length = 1, all exhausted
assert.deepEqual(result.triedProviders, ['anthropic']);
});
it('two-hop chain, primary fails with hard trigger → falls back to secondary, fallbackHops=1', async () => {
const err = new ProviderError('Spawn failed', 'SPAWN_FAILED');
const chain = [
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
{ provider: 'openai', model: 'gpt-5.5' },
];
const result = await executeWithFallback(chain, dummyIR, makeHopFn({ anthropic: err, openai: 'success' }));
assert.ok(Array.isArray(result.chunks), 'Expected chunks from secondary');
assert.equal(result.fallbackHops, 1);
assert.equal(result.providerUsed, 'openai');
assert.equal(result.originalError, null);
assert.deepEqual(result.triedProviders, ['anthropic', 'openai']);
});
it('three-hop chain, primary + secondary fail → tertiary returns chunks, fallbackHops=2', async () => {
const errA = new ProviderError('Rate limited', 'RATE_LIMITED');
const errB = Object.assign(new Error('Service unavailable'), { statusCode: 503 });
const chain = [
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
{ provider: 'openai', model: 'gpt-5.5' },
{ provider: 'mistral', model: 'devstral-2' },
];
const result = await executeWithFallback(
chain, dummyIR,
makeHopFn({ anthropic: errA, openai: errB, mistral: 'success' }),
);
assert.ok(Array.isArray(result.chunks));
assert.equal(result.fallbackHops, 2);
assert.equal(result.providerUsed, 'mistral');
assert.equal(result.originalError, null);
assert.deepEqual(result.triedProviders, ['anthropic', 'openai', 'mistral']);
});
it('three-hop chain, all fail → exhausted, originalError is from FIRST hop (not last)', async () => {
const errA = new ProviderError('Rate limited', 'RATE_LIMITED');
const errB = new ProviderError('Spawn failed', 'SPAWN_FAILED');
const errC = new ProviderError('CLI not found', 'CLI_NOT_FOUND');
const chain = [
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
{ provider: 'openai', model: 'gpt-5.5' },
{ provider: 'mistral', model: 'devstral-2' },
];
const result = await executeWithFallback(
chain, dummyIR,
makeHopFn({ anthropic: errA, openai: errB, mistral: errC }),
);
assert.equal(result.chunks, null);
assert.equal(result.originalError, errA, 'Should be first hop error, not last');
assert.equal(result.fallbackHops, 3); // chain.length = 3
assert.deepEqual(result.triedProviders, ['anthropic', 'openai', 'mistral']);
});
it('client error (400) on primary → does NOT fall back, surfaces error immediately', async () => {
const err = Object.assign(new Error('Bad request'), { statusCode: 400 });
const chain = [
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
{ provider: 'openai', model: 'gpt-5.5' },
];
let openaiCalled = false;
const hopFn = async (provider) => {
if (provider === 'anthropic') throw err;
openaiCalled = true;
return [{ type: 'stop', finish_reason: 'stop' }];
};
const result = await executeWithFallback(chain, dummyIR, hopFn);
assert.equal(result.chunks, null);
assert.equal(result.originalError, err);
assert.equal(openaiCalled, false, 'Second provider must NOT be called on client error');
assert.deepEqual(result.triedProviders, ['anthropic']);
});
it('AUTH_MISSING on primary → does NOT fall back (user must fix config)', async () => {
const err = new ProviderError('Auth missing', 'AUTH_MISSING');
const chain = [
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
{ provider: 'openai', model: 'gpt-5.5' },
];
let openaiCalled = false;
const hopFn = async (provider) => {
if (provider === 'anthropic') throw err;
openaiCalled = true;
return [{ type: 'stop', finish_reason: 'stop' }];
};
const result = await executeWithFallback(chain, dummyIR, hopFn);
assert.equal(result.chunks, null);
assert.equal(result.originalError, err);
assert.equal(openaiCalled, false, 'Second provider must NOT be called on AUTH_MISSING');
});
it('non-trigger error (generic) on primary → does NOT fall back', async () => {
const err = new Error('Unexpected internal error');
const chain = [
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
{ provider: 'openai', model: 'gpt-5.5' },
];
let openaiCalled = false;
const hopFn = async (provider) => {
if (provider === 'anthropic') throw err;
openaiCalled = true;
return [{ type: 'stop', finish_reason: 'stop' }];
};
const result = await executeWithFallback(chain, dummyIR, hopFn);
assert.equal(result.chunks, null);
assert.equal(openaiCalled, false, 'Must not fall back on non-trigger error');
});
});
// ── 13c: First-chunk safety ──────────────────────────────────────────────
describe('Fallback engine — first-chunk safety (D9)', () => {
// At D9, executeHopFn = collectAllChunks() which is fully buffered.
// Safety is enforced by construction: if executeHopFn returns successfully,
// chunks are already in memory (none written to res yet). Once chunks are
// returned, fallback is no longer attempted — we return immediately.
// If executeHopFn throws, zero bytes went to the client.
it('successful executeHopFn returns chunks immediately without retrying secondary', async () => {
const chain = [
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
{ provider: 'openai', model: 'gpt-5.5' },
];
let callCount = 0;
const hopFn = async (_provider) => {
callCount++;
return [{ type: 'stop', finish_reason: 'stop' }];
};
const result = await executeWithFallback(chain, makeIR({ model: 'test' }), hopFn);
assert.equal(result.fallbackHops, 0, 'Primary served — no fallback');
assert.equal(callCount, 1, 'executeHopFn called exactly once (first-chunk safety)');
});
it('error from executeHopFn (hard trigger) means zero chunks emitted — advance is safe', async () => {
// Simulate: anthropic throws (no bytes emitted), openai succeeds
const err = new ProviderError('Rate limited', 'RATE_LIMITED');
const chain = [
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
{ provider: 'openai', model: 'gpt-5.5' },
];
let writtenToClient = false;
const hopFn = async (provider) => {
if (provider === 'anthropic') {
// Throw BEFORE any "write" — simulates first-chunk rule
throw err;
}
// openai succeeds — simulate "writtenToClient" only AFTER returning
// (in real server, chunks are written after executeWithFallback returns)
writtenToClient = true;
return [{ type: 'stop', finish_reason: 'stop' }];
};
const result = await executeWithFallback(chain, makeIR({ model: 'test' }), hopFn);
// openai served, which means writtenToClient is set (simulates post-return write)
assert.equal(result.fallbackHops, 1, 'Fell back to openai');
// Crucially: the "write" happened AFTER executeHopFn returned (not during chain iteration)
assert.equal(writtenToClient, true, 'Client write happens post-return, not during chain iteration');
});
});
// ── 13d: Soft triggers skipping spawn ────────────────────────────────────
describe('Fallback engine — soft trigger skipping (D9)', () => {
const dummyIR = makeIR({ model: 'test-model' });
it('chain [A, B], A soft trigger fires → engine skips A entirely, calls B; fallbackHops=1', async () => {
let aCalled = false;
let bCalled = false;
const hopFn = async (provider) => {
if (provider === 'a') { aCalled = true; return [{ type: 'stop', finish_reason: 'stop' }]; }
if (provider === 'b') { bCalled = true; return [{ type: 'stop', finish_reason: 'stop' }]; }
throw new Error('Unexpected provider');
};
const chain = [
{
provider: 'a',
model: 'model-a',
softTriggers: { credit_pool_percent_threshold: 90 },
quotaSnapshot: { percentUsed: 95 }, // fires the trigger
},
{ provider: 'b', model: 'model-b' },
];
const result = await executeWithFallback(chain, dummyIR, hopFn);
assert.equal(aCalled, false, 'A must NOT be called (soft trigger skipped it)');
assert.equal(bCalled, true, 'B must be called');
assert.equal(result.fallbackHops, 1, 'B served as fallback hop 1');
assert.equal(result.providerUsed, 'b');
});
it('chain [A, B], neither soft trigger fires → A is called; if A succeeds, B never called', async () => {
let aCalled = false;
let bCalled = false;
const hopFn = async (provider) => {
if (provider === 'a') { aCalled = true; return [{ type: 'stop', finish_reason: 'stop' }]; }
if (provider === 'b') { bCalled = true; return [{ type: 'stop', finish_reason: 'stop' }]; }
throw new Error('Unexpected provider');
};
const chain = [
{
provider: 'a',
model: 'model-a',
softTriggers: { credit_pool_percent_threshold: 90 },
quotaSnapshot: { percentUsed: 85 }, // below threshold, trigger does NOT fire
},
{ provider: 'b', model: 'model-b' },
];
const result = await executeWithFallback(chain, dummyIR, hopFn);
assert.equal(aCalled, true, 'A must be called');
assert.equal(bCalled, false, 'B must NOT be called (A succeeded)');
assert.equal(result.fallbackHops, 0);
assert.equal(result.providerUsed, 'a');
});
it('chain [A], A soft trigger fires (null quotaSnapshot) → trigger does NOT fire, A called', async () => {
// Per ADR 0004: null quotaStatus → treat as "don't fire"
let aCalled = false;
const hopFn = async (provider) => {
if (provider === 'a') { aCalled = true; return [{ type: 'stop', finish_reason: 'stop' }]; }
throw new Error('Unexpected provider');
};
const chain = [
{
provider: 'a',
model: 'model-a',
softTriggers: { credit_pool_percent_threshold: 90 },
quotaSnapshot: null, // null → trigger never fires
},
];
const result = await executeWithFallback(chain, dummyIR, hopFn);
assert.equal(aCalled, true, 'A must be called (null quota → trigger does not fire)');
assert.equal(result.fallbackHops, 0);
});
});
// ── 13e: Header annotation ───────────────────────────────────────────────
describe('Fallback engine — observability / header annotation (D9)', () => {
const dummyIR = makeIR({ model: 'test-model' });
it('success on primary: providerUsed + modelUsed match primary hop', async () => {
const chain = [{ provider: 'anthropic', model: 'claude-sonnet-4-6' }];
const result = await executeWithFallback(chain, dummyIR, async () => [{ type: 'stop', finish_reason: 'stop' }]);
assert.equal(result.providerUsed, 'anthropic');
assert.equal(result.modelUsed, 'claude-sonnet-4-6');
});
it('success on fallback: providerUsed + modelUsed match the serving hop', async () => {
const err = new ProviderError('Rate limited', 'RATE_LIMITED');
const chain = [
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
{ provider: 'openai', model: 'gpt-5.5' },
];
const hopFn = async (provider) => {
if (provider === 'anthropic') throw err;
return [{ type: 'stop', finish_reason: 'stop' }];
};
const result = await executeWithFallback(chain, dummyIR, hopFn);
assert.equal(result.providerUsed, 'openai');
assert.equal(result.modelUsed, 'gpt-5.5');
assert.equal(result.fallbackHops, 1);
});
it('chain exhausted: originalError is from FIRST hop, not second or third', async () => {
const errA = new ProviderError('Rate limited', 'RATE_LIMITED');
const errB = new ProviderError('Spawn failed', 'SPAWN_FAILED');
const errC = new ProviderError('CLI not found', 'CLI_NOT_FOUND');
const chain = [
{ provider: 'a', model: 'model-a' },
{ provider: 'b', model: 'model-b' },
{ provider: 'c', model: 'model-c' },
];
const hopFn = async (provider) => {
if (provider === 'a') throw errA;
if (provider === 'b') throw errB;
if (provider === 'c') throw errC;
throw new Error('Unexpected');
};
const result = await executeWithFallback(chain, dummyIR, hopFn);
assert.equal(result.originalError, errA, 'originalError must be first-hop error');
assert.notEqual(result.originalError, errB, 'Must NOT be second-hop error');
assert.notEqual(result.originalError, errC, 'Must NOT be third-hop error');
});
it('triedProviders lists all attempted hops in chain order', async () => {
const err = new ProviderError('Rate limited', 'RATE_LIMITED');
const chain = [
{ provider: 'a', model: 'model-a' },
{ provider: 'b', model: 'model-b' },
{ provider: 'c', model: 'model-c' },
];
const hopFn = async (provider) => {
if (provider === 'a') throw err;
if (provider === 'b') throw err;
return [{ type: 'stop', finish_reason: 'stop' }];
};
const result = await executeWithFallback(chain, dummyIR, hopFn);
assert.deepEqual(result.triedProviders, ['a', 'b', 'c']);
assert.equal(result.fallbackHops, 2);
assert.equal(result.providerUsed, 'c');
});
});
// ── 13f: HTTP integration ────────────────────────────────────────────────
describe('Fallback engine — HTTP integration (D9)', () => {
let server;
let port;
before(async () => {
__resetSpawnImpl();
codexResetSpawnImpl();
mistralResetSpawnImpl();
__resetFallbackConfig();
// Start test server with all 3 providers enabled
const testProviders = loadProviders({ enabled: { anthropic: true, openai: true, mistral: true } });
const { loadedProviders: lp, cacheStore: cs } = await import('./server.mjs');
// Inject test providers
for (const [name, p] of testProviders) {
lp.set(name, p);
}
cs.clear();
server = createOlpServer();
port = 20456 + Math.floor(Math.random() * 500);
await new Promise((resolve, reject) => {
server.listen(port, '127.0.0.1', resolve);
server.once('error', (e) => {
if (e.code === 'EADDRINUSE') {
port++;
server.listen(port, '127.0.0.1', resolve);
server.once('error', reject);
} else reject(e);
});
});
});
after(() => {
__resetFallbackConfig();
__resetSpawnImpl();
codexResetSpawnImpl();
mistralResetSpawnImpl();
if (!server) return;
return new Promise(r => server.close(r));
});
// Shared mock spawn builder that returns a successful response
function makeSuccessSpawn(content = 'Hello from mock') {
return function mockSpawnImpl(_bin, _args, _opts) {
const proc = new EventEmitter();
proc.stdout = new EventEmitter();
proc.stderr = new EventEmitter();
proc.stdin = {
write: () => {},
end: () => {
setImmediate(() => {
// Emit a minimal SSE-like response that anthropic plugin parses
// (Anthropic mock format: JSON lines per the real anthropic.mjs parser)
const lines = [
JSON.stringify({ type: 'content_block_delta', index: 0, delta: { type: 'text_delta', text: content } }),
JSON.stringify({ type: 'message_stop' }),
];
for (const line of lines) {
proc.stdout.emit('data', Buffer.from(`data: ${line}\n\n`));
}
proc.stdout.emit('end');
proc.stderr.emit('end');
proc.emit('close', 0, null);
});
},
};
proc.killed = false;
proc.kill = () => {};
return proc;
};
}
it('no fallback config: POST with no enabled provider → 503', async () => {
// Remove all providers temporarily
const { loadedProviders: lp } = await import('./server.mjs');
const savedMap = new Map(lp);
lp.clear();
try {
const r = await fetch({
port,
method: 'POST',
path: '/v1/chat/completions',
body: { model: 'unknown-model', messages: [{ role: 'user', content: 'Hi' }] },
});
assert.equal(r.status, 503);
} finally {
for (const [name, p] of savedMap) lp.set(name, p);
}
});
it('no fallback config + mock anthropic: POST claude-sonnet-4-6 → 200 + X-OLP-Fallback-Hops: 0', async () => {
__setSpawnImpl(makeSuccessSpawn('Test response'));
try {
const r = await fetch({
port,
method: 'POST',
path: '/v1/chat/completions',
body: {
model: 'claude-sonnet-4-6',
messages: [{ role: 'user', content: 'Hi' }],
max_tokens: 10,
},
});
assert.equal(r.status, 200, `Expected 200, got ${r.status}: ${r.body.slice(0, 200)}`);
assert.equal(r.headers['x-olp-fallback-hops'], '0');
assert.equal(r.headers['x-olp-provider-used'], 'anthropic');
} finally {
__resetSpawnImpl();
}
});
it('fallback config: mock anthropic fails (SPAWN_FAILED), chain [anthropic→openai], openai mock succeeds → 200 + X-OLP-Fallback-Hops: 1 + X-OLP-Provider-Used: openai', async () => {
// Clear cache to prevent cache-hit from previous test polluting this one.
// ADR 0005: each (provider, model) pair is independently cached; a hit
// from the previous test would cause executeHopFn to skip the spawn and
// return cached chunks, masking the SPAWN_FAILED mock.
__clearCache();
// Provide fake codex auth so AUTH_MISSING doesn't stop the chain before
// the codex mock can respond. Codex checks readAuthArtifact() before spawnImpl.
// Write a temp file and set OPENAI_CODEX_AUTH_PATH to point at it.
const { writeFileSync, unlinkSync } = await import('node:fs');
const { tmpdir } = await import('node:os');
const { join: pathJoin } = await import('node:path');
const tmpAuthFile = pathJoin(tmpdir(), `olp-test-codex-auth-${Date.now()}.json`);
writeFileSync(tmpAuthFile, JSON.stringify({ accessToken: 'fake-test-token-for-codex' }), 'utf8');
const savedCodexAuthPath = process.env.OPENAI_CODEX_AUTH_PATH;
process.env.OPENAI_CODEX_AUTH_PATH = tmpAuthFile;
// Set up a 2-hop chain: anthropic → openai for claude-sonnet-4-6
__setFallbackConfig({
chains: {
'claude-sonnet-4-6': [
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
{ provider: 'openai', model: 'gpt-5.5' },
],
},
soft_triggers: {},
});
// Anthropic mock: always exits with code 1 → SPAWN_FAILED (hard trigger)
__setSpawnImpl(function (_bin, _args, _opts) {
const proc = new EventEmitter();
proc.stdout = new EventEmitter();
proc.stderr = new EventEmitter();
proc.stdin = {
write: () => {},
end: () => {
setImmediate(() => {
proc.stderr.emit('data', Buffer.from('spawn failed\n'));
proc.stdout.emit('end');
proc.stderr.emit('end');
proc.emit('close', 1, null); // non-zero exit → SPAWN_FAILED
});
},
};
proc.killed = false;
proc.kill = () => {};
return proc;
});
// Codex (openai) mock: succeeds with a delta+stop response
codexSetSpawnImpl(function (_bin, _args, _opts) {
const proc = new EventEmitter();
proc.stdout = new EventEmitter();
proc.stderr = new EventEmitter();
proc.stdin = {
write: () => {},
end: () => {
setImmediate(() => {
proc.stdout.emit('data', Buffer.from('{"content":"Fallback response from openai"}\n'));
proc.stdout.emit('data', Buffer.from('{"type":"stop"}\n'));
proc.stdout.emit('end');
proc.stderr.emit('end');
proc.emit('close', 0, null);
});
},
};
proc.killed = false;
proc.kill = () => {};
return proc;
});
try {
const r = await fetch({
port,
method: 'POST',
path: '/v1/chat/completions',
body: {
model: 'claude-sonnet-4-6',
messages: [{ role: 'user', content: 'Hi' }],
max_tokens: 10,
},
});
assert.equal(r.status, 200, `Expected 200 on fallback, got ${r.status}: ${r.body.slice(0, 300)}`);
assert.equal(r.headers['x-olp-fallback-hops'], '1', `Expected fallback-hops=1, got: ${r.headers['x-olp-fallback-hops']}`);
assert.equal(r.headers['x-olp-provider-used'], 'openai', `Expected provider-used=openai, got: ${r.headers['x-olp-provider-used']}`);
} finally {
__resetFallbackConfig();
__resetSpawnImpl();
codexResetSpawnImpl();
if (savedCodexAuthPath !== undefined) {
process.env.OPENAI_CODEX_AUTH_PATH = savedCodexAuthPath;
} else {
delete process.env.OPENAI_CODEX_AUTH_PATH;
}
try { unlinkSync(tmpAuthFile); } catch { /* ignore */ }
}
});
it('fallback config: two-hop chain with both providers failing SPAWN_FAILED → exhausted-chain path → error response with X-OLP-Fallback-Exhausted header', async () => {
// NOTE on this test's renaming (D9 review-2): the original sonnet draft
// titled this as a "client error (400) → no fallback" HTTP test. But the
// anthropic plugin surfaces non-zero exit codes as ProviderError
// SPAWN_FAILED (a HARD trigger), not as a typed-400 client error. So at
// the HTTP integration layer we can't easily inject a synthetic 400
// without monkey-patching executeHopFn. The "client error stops chain
// immediately" semantic IS covered at the UNIT level in
// "Engine: two-hop chain primary client error 400 → secondary NOT
// called" (line 3289+ in this file). This HTTP test now exercises the
// complementary path: both hops fail with SPAWN_FAILED → chain
// exhausted → originalError surfaces with X-OLP-Fallback-Exhausted
// header listing both providers.
__clearCache();
// Set up a 2-hop chain
__setFallbackConfig({
chains: {
'claude-sonnet-4-6': [
{ provider: 'anthropic', model: 'claude-sonnet-4-6' },
{ provider: 'openai', model: 'gpt-5.5' },
],
},
soft_triggers: {},
});
// Both providers fail with SPAWN_FAILED
__setSpawnImpl(function (_bin, _args, _opts) {
const proc = new EventEmitter();
proc.stdout = new EventEmitter();
proc.stderr = new EventEmitter();
proc.stdin = {
write: () => {},
end: () => {
setImmediate(() => {
proc.stdout.emit('end');
proc.stderr.emit('end');
proc.emit('close', 1, null);
});
},
};
proc.killed = false;
proc.kill = () => {};
return proc;
});
codexSetSpawnImpl(function (_bin, _args, _opts) {
const proc = new EventEmitter();
proc.stdout = new EventEmitter();
proc.stderr = new EventEmitter();
proc.stdin = {
write: () => {},
end: () => {
setImmediate(() => {
proc.stdout.emit('end');
proc.stderr.emit('end');
proc.emit('close', 1, null);
});
},
};
proc.killed = false;
proc.kill = () => {};
return proc;
});
try {
const r = await fetch({
port,
method: 'POST',
path: '/v1/chat/completions',
body: {
model: 'claude-sonnet-4-6',
messages: [{ role: 'user', content: 'Hi' }],
max_tokens: 10,
},
});
// Both providers fail → exhausted chain → 502 error response with exhausted header
assert.ok(r.status >= 400 && r.status < 600, `Expected error status, got ${r.status}`);
// X-OLP-Fallback-Exhausted header should be present since both providers tried
assert.ok(
r.headers['x-olp-fallback-exhausted'] !== undefined,
`Expected X-OLP-Fallback-Exhausted header, got headers: ${JSON.stringify(r.headers)}`,
);
} finally {
__resetFallbackConfig();
__resetSpawnImpl();
codexResetSpawnImpl();
}
});
});