feat(fallback)+chore(cache): D28 — per-hop log observability fields (round-3 F2)

cold-audit catch from 2026-05-24 (round 3)

Round-3 cold-audit Finding 2 (P2 observability gap). ADR 0004 §
Observability headers requires per-hop structured log events to carry
7 fields: timestamp, chain id, hop index, failed provider, trigger type,
IR request hash, and downstream provider that was tried next. Pre-D28
events only carried timestamp/hop/provider/model/error/code — operators
could not:
- correlate hops across one logical request (no chain_id)
- pivot per-prompt via fingerprint (no ir_request_hash)
- tell from one log line which bucket classified the error (no
  trigger_type)
- see lookahead routing (no next_provider)

Changes (3 files, +431 / -3):

1. lib/cache/keys.mjs (+35):
   - New export `computeIRRequestHash(ir)` returning a 16-char SHA-256
     prefix over the IR fields that define request semantics
     (messages-normalized, tools, temperature, max_tokens, top_p, stop,
     tool_choice, response_format). Excludes provider/model/cache_control
     so the hash is provider-agnostic — identical prompts across a
     fallback chain produce identical fingerprints, enabling log
     correlation. 16-char prefix gives 2^64 collision resistance —
     astronomically safe for the use case while keeping log lines compact.

2. lib/fallback/engine.mjs (+86 / -3):
   - New helper `classifyTrigger(error)` returning one of:
     'hard' / 'auth_missing' / 'client_error' / 'non_trigger' / null.
     ProviderError with HARD_TRIGGER_CODES → 'hard'; AUTH_MISSING → its
     own bucket; HTTP 400/401/403/404/422 → 'client_error' (per ADR 0004
     § "No fallback for client-side errors"); HTTP 5xx or non-client 4xx
     (e.g. 429/529) → 'hard'; other → 'non_trigger'. 'soft' is reserved
     in JSDoc for forward-compat per D22 Amendment 2's deferral of soft
     triggers to v1.x.
   - `executeWithFallback` now generates `chainId` (crypto.randomBytes(8)
     .toString('hex')) and `irRequestHash` (computeIRRequestHash(ir))
     ONCE at entry — both stay constant across all hops in the chain.
   - All 8 logEvent call sites augmented with the 4 new fields:
     fallback_soft_trigger, fallback_hop_success, fallback_hop_error,
     fallback_client_error_no_fallback, fallback_auth_missing_no_fallback,
     fallback_hard_trigger, fallback_non_trigger_error,
     fallback_chain_exhausted.
   - `next_provider` is `chain[i + 1]?.provider ?? null` on advancement
     events; null on terminal events (success / client_error /
     auth_missing / chain_exhausted).
   - chain_id NOT reused from generateRequestId() since that produces
     OpenAI-response-scoped `chatcmpl-<base64url>` IDs; chain_id is
     internal log correlation and uses a clean 16-char hex token.

3. test-features.mjs (+313): 24 new tests covering:
   - computeIRRequestHash determinism + 16-char hex shape + 8 per-field
     sensitivity tests (one per IR input field)
   - Provider-agnosticism (same hash across different provider/model)
   - chain_id uniqueness across calls + consistency across hops
   - trigger_type for each branch (hard / auth_missing / client_error /
     non_trigger / null)
   - next_provider correctness (chain[i+1] on advance; null at
     exhaustion)

Tests: 376 → 400 (+24). All pass on Node 20.

Pure observation-only change: no fallback decision logic touched.
`evaluateHardTriggers`, `evaluateSoftTriggers`, `HARD_TRIGGER_CODES`,
`CLIENT_ERROR_STATUSES` are only READ by `classifyTrigger`; the existing
decision branches are textually unchanged — only the trailing logEvent
object literals are expanded.

Authority:
- ADR 0004 § Observability headers (the 7-field requirement)
  https://github.com/dtzp555-max/olp/blob/main/docs/adr/0004-fallback-engine.md
- ADR 0004 § "No fallback for client-side errors" — 401/403/404/422
  bucket (used by classifyTrigger's CLIENT_ERROR_STATUSES branch)
- ADR 0005 Amendment 2 (D15) — cache key composition fields, mirrored
  in computeIRRequestHash minus provider/model/cache_control
- D22 Amendment 2 (ADR 0004) — soft triggers deferred to v1.x;
  'soft' reserved in JSDoc but unreturnable
- CC 开发铁律 v1.6 § 10.x — Round-3 Cold Audit caught this

Reviewer (Iron Rule v1.6 § 10.x Mode A, fresh-context opus, independent
of drafter): APPROVE. Traced all 8 classifyTrigger branches against
ADR 0004 wording; verified 401/403/404/422 correctly classified as
'client_error' (never 'hard') per the ordering of CLIENT_ERROR_STATUSES
check before the generic status>=400 fallthrough; confirmed observation-
only via diff inspection (only logEvent object literals expanded, no
control-flow changes); independently ran git-stash to verify pre-D28
baseline 376 + 24 = 400.

3 non-blocking suggestions noted (JSDoc 'soft' union vs unreachable
return; future cross-correlation of chain_id with chatcmpl response IDs;
log-line redundancy between fallback_hop_error + fallback_hard_trigger
for same hop) — all cosmetic, not folded.

Note on D27 CI: D27 first CI run failed 1/376 on a Suite 17 port-
collision flake (overlapping random port ranges across Suite 17 tests +
Linux TIME_WAIT timing). Re-run was green. The flake is pre-existing
from D18-era test design; tracked as D29 follow-up to switch Suite 17
to OS-assigned port 0. D28 doesn't touch Suite 17 and is unaffected.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-24 16:40:30 +10:00
co-authored by Claude Opus 4.7
parent c3ba751a8f
commit 4a238c9cc4
3 changed files with 431 additions and 3 deletions
+83 -3
View File
@@ -13,12 +13,14 @@
* Sealed at D9. Do NOT modify without an ADR 0004 amendment.
*/
import { randomBytes } from 'node:crypto';
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { join } from 'node:path';
import { ProviderError } from '../providers/base.mjs';
import { getProviderForModel } from '../providers/index.mjs';
import { computeIRRequestHash } from '../cache/keys.mjs';
// ── Hard-trigger evaluation ────────────────────────────────────────────────
@@ -53,6 +55,35 @@ const HARD_TRIGGER_CODES = {
*/
const CLIENT_ERROR_STATUSES = new Set([400, 401, 403, 404, 422]);
/**
* Classifies a thrown error into one of the trigger taxonomy categories
* per ADR 0004 § Trigger taxonomy (D28 round-3 F2). Used in log events
* so operators can pivot on the classification bucket.
*
* Note: 'soft' is not currently returnable from this function. Soft triggers
* are evaluated before spawn via evaluateSoftTriggers() and are deferred to
* v1.x (D22 Amendment 2). The 'soft' variant is listed in the JSDoc return
* type for forward-compatibility only.
*
* @param {Error|null|undefined} error
* @returns {'hard' | 'soft' | 'auth_missing' | 'client_error' | 'non_trigger' | null}
*/
function classifyTrigger(error) {
if (!error) return null;
if (error instanceof ProviderError && error.code) {
if (error.code === 'AUTH_MISSING') return 'auth_missing';
if (HARD_TRIGGER_CODES[error.code] === true) return 'hard';
return 'non_trigger';
}
const status = error.statusCode ?? error.status ?? null;
if (status !== null && typeof status === 'number') {
if (CLIENT_ERROR_STATUSES.has(status)) return 'client_error';
if (status >= 500) return 'hard';
if (status >= 400) return 'hard'; // quota-exhaustion 4xx (e.g. 429, 529)
}
return 'non_trigger';
}
/**
* Returns true if `status` is a client-side error that must NOT trigger fallback.
* Exported for tests.
@@ -220,6 +251,12 @@ export async function executeWithFallback(chain, irRequest, executeHopFn, option
throw new Error('executeWithFallback: chain must be a non-empty array');
}
// D28 round-3 F2: per-request fingerprints for log correlation.
// chain_id is unique per executeWithFallback invocation; ir_request_hash
// is provider-agnostic so it groups all hops from the same logical request.
const chainId = randomBytes(8).toString('hex'); // 16-char hex
const irRequestHash = computeIRRequestHash(irRequest);
const triedProviders = [];
let originalError = null; // Per ADR 0004: first-hop error is the canonical signal
let firstErrorRecorded = false;
@@ -243,9 +280,13 @@ export async function executeWithFallback(chain, irRequest, executeHopFn, option
// visibility; D9 keeps them unified per ADR-0004 spec.
if (evaluateSoftTriggers(softTriggers, quotaSnapshot)) {
logEvent('info', 'fallback_soft_trigger', {
chain_id: chainId,
hop: i,
provider,
model,
trigger_type: 'soft',
ir_request_hash: irRequestHash,
next_provider: chain[i + 1]?.provider ?? null,
reason: 'soft_trigger_fired',
});
triedProviders.push(provider);
@@ -276,7 +317,15 @@ export async function executeWithFallback(chain, irRequest, executeHopFn, option
// executeHopFn returned successfully = chunks buffered = no client writes
// blocked = safe to return. Any retry attempt after this point would be
// post-first-chunk and is therefore forbidden. We return immediately.
logEvent('info', 'fallback_hop_success', { hop: i, provider, model });
logEvent('info', 'fallback_hop_success', {
chain_id: chainId,
hop: i,
provider,
model,
trigger_type: null,
ir_request_hash: irRequestHash,
next_provider: null,
});
return {
chunks,
providerUsed: provider,
@@ -294,18 +343,30 @@ export async function executeWithFallback(chain, irRequest, executeHopFn, option
}
logEvent('warn', 'fallback_hop_error', {
chain_id: chainId,
hop: i,
provider,
model,
error: err.message,
code: err.code ?? null,
trigger_type: classifyTrigger(err),
ir_request_hash: irRequestHash,
next_provider: chain[i + 1]?.provider ?? null,
});
// ── Client error: stop immediately ─────────────────────────────
// Per ADR 0004 § "No fallback for client-side errors"
const status = err.statusCode ?? err.status ?? null;
if (status !== null && CLIENT_ERROR_STATUSES.has(status)) {
logEvent('warn', 'fallback_client_error_no_fallback', { hop: i, provider, status });
logEvent('warn', 'fallback_client_error_no_fallback', {
chain_id: chainId,
hop: i,
provider,
status,
trigger_type: 'client_error',
ir_request_hash: irRequestHash,
next_provider: null,
});
return {
chunks: null,
providerUsed: provider,
@@ -321,7 +382,14 @@ export async function executeWithFallback(chain, irRequest, executeHopFn, option
// Silently masking it by trying next provider prevents user from discovering
// the misconfiguration.
if (err instanceof ProviderError && err.code === 'AUTH_MISSING') {
logEvent('warn', 'fallback_auth_missing_no_fallback', { hop: i, provider });
logEvent('warn', 'fallback_auth_missing_no_fallback', {
chain_id: chainId,
hop: i,
provider,
trigger_type: 'auth_missing',
ir_request_hash: irRequestHash,
next_provider: null,
});
return {
chunks: null,
providerUsed: provider,
@@ -338,12 +406,16 @@ export async function executeWithFallback(chain, irRequest, executeHopFn, option
// Fallback is eligible if the error is a hard trigger.
if (evaluateHardTriggers(err)) {
logEvent('info', 'fallback_hard_trigger', {
chain_id: chainId,
hop: i,
provider,
model,
error: err.message,
code: err.code ?? null,
advance_to_hop: i + 1,
trigger_type: classifyTrigger(err),
ir_request_hash: irRequestHash,
next_provider: chain[i + 1]?.provider ?? null,
});
continue; // advance to next hop
}
@@ -351,10 +423,14 @@ export async function executeWithFallback(chain, irRequest, executeHopFn, option
// ── Non-trigger error: surface immediately ──────────────────────
// Per ADR 0004 § Alternatives considered (a): "any error" fallback rejected.
logEvent('warn', 'fallback_non_trigger_error', {
chain_id: chainId,
hop: i,
provider,
error: err.message,
code: err.code ?? null,
trigger_type: classifyTrigger(err),
ir_request_hash: irRequestHash,
next_provider: null,
});
return {
chunks: null,
@@ -371,8 +447,12 @@ export async function executeWithFallback(chain, irRequest, executeHopFn, option
// Per ADR 0004 § Decision § Chain advancement step 4:
// "return A's original error (not B's, not C's)"
logEvent('error', 'fallback_chain_exhausted', {
chain_id: chainId,
chain: chain.map(h => h.provider),
triedProviders,
trigger_type: null,
ir_request_hash: irRequestHash,
next_provider: null,
});
return {