mirror of
https://github.com/dtzp555-max/olp.git
synced 2026-07-19 09:45:07 +00:00
feat+test+docs: D49 — lib/audit-query.mjs (Phase 3 audit aggregate query layer) (#26)
Second Phase 3 D-day. Implements ADR 0008 § 4 query API. Pure in-memory
ndjson scan; cross-file walk over audit.ndjson (live) +
audit-YYYY-MM-DD.ndjson (rotated). No server.mjs integration in this
D-day (D50 wires the consuming endpoints).
NEW lib/audit-query.mjs (~370 lines): 5 public API functions per
ADR 0008 § 4.1:
- discoverAuditFiles({ olpHome }): filesystem scan; returns
Map<date|'live', path>.
- readAuditWindow({ startMs, endMs, olpHome, logEvent }): generator
over events in half-open window [startMs, endMs). Walks rotated
date files + live file. Skips malformed lines + logs warn.
- aggregateRequests({ windowMs, olpHome }): counts + status buckets
+ by_provider + by_owner_tier + by_path + median/p95 latency over
rolling window.
- topFallbackChains({ windowMs, limit, olpHome }): top-N chains by
trigger count from events with fallback_hops > 0. Tied-count
tiebreak: ascending first_seen.
- spendTrendDaily({ days, olpHome }): daily series ending today
with sparse-fill for zero-request days. Per-day request_count +
median latency + by_provider breakdown.
- cacheHitRateWindow({ windowMs, olpHome }): audit-derived cache
hit rate (bypass excluded from denominator); per-provider + overall.
PII discipline (ADR 0008 § 4.3): every aggregate function relays only
schema fields; never message content. Suite 23g actively asserts the
absence of content/message/messages/prompt/response/body keys in every
aggregate output.
Cross-file walk semantics (ADR 0008 § 4.2): half-open window
[startMs, endMs); date-range computed once from window bounds; each
rotated date file checked; live audit.ndjson always checked (it
covers today regardless of whether the window endpoint is past
midnight).
spendTrendDaily calendar-date semantics:
days: N returns "last N calendar UTC dates ending today" — NOT
"events within a rolling N*86400-ms window" (which would span N+1
distinct UTC dates and produce off-by-one buckets at non-midnight
call times). Computed via:
for (let i = days-1; i >= 0; i--)
dates.push(_utcDateFromMs(now - i*86400*1000));
cacheHitRateWindow denominator: hit_rate = hit / (hit + miss).
Bypass is intentional non-cacheable (Anthropic cache_control marker),
NOT a cache miss; excluding it from the denominator gives a clean
cache-effectiveness signal.
TESTS — Suite 23, +27 (544 → 571):
23a-1..4: discoverAuditFiles (empty dir / live only / live+rotated /
non-audit files ignored)
23b-1..6: readAuditWindow (all-coverage / single-day / half-open
exclusivity / empty window / missing files / malformed-skip with
warn)
23c-1..4: aggregateRequests (counts + status buckets + by_provider;
by_owner_tier; median+p95 latency over realistic distribution;
invalid windowMs rejection)
23d-1..4: topFallbackChains (sort desc by count; limit truncation;
fallback_hops=0 excluded; first_seen/last_seen carried)
23e-1..3: spendTrendDaily (N-day range correctness — caught off-by-
one during local run; populated day breakdown; empty day sparse-
fill)
23f-1..3: cacheHitRateWindow (overall + per-provider hit_rate;
bypass not in denominator; cache_status=null events excluded)
23g-1..3: PII guard for aggregateRequests / spendTrendDaily /
topFallbackChains + cacheHitRateWindow — every output JSON-
stringified + scanned for forbidden PII keys
DOCUMENTATION:
- AGENTS.md: lib/audit-query.mjs new entry; lib/audit.mjs note added
that D52 extends with daily rotation.
NOT IN D49 scope:
- server.mjs endpoints consuming these queries (D50)
- dashboard.html (D51)
- lib/audit.mjs rotation extension + bin/olp-audit-rotate.mjs (D52)
- tried_providers schema fix (D53; D45 P2 deferral)
- Phase 3 close → v0.3.0 (D55; maintainer-triggered)
Test count: 544 → 571 (+27). Verified locally via npm test.
AUTHORITY:
- ADR 0008 § 4 (query API surface) + § 5 (rotation file naming
pattern) + § 3 (storage layout).
- ADR 0007 § 8 (audit ndjson event schema — input data).
- CLAUDE.md release_kit overlay phase_rolling_mode — under
Unreleased.
- Standing autopilot grant.
ALIGNMENT.md scope check: this PR adds a new lib/ module. No provider
plugin / entry surface / IR change. Rule 5 commit-citation requirements
for those scopes do not apply.
Co-authored-by: dtzp555 <dtzp555@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
dtzp555-max
@@ -39,7 +39,8 @@ Runtime: Node.js (ESM, `.mjs` throughout). No build step. No bundler. `server.mj
|
||||
- `lib/fallback/` — fallback engine (trigger detection, chain advancement, idempotent-failure safety, header annotation). Governed by ADR 0004.
|
||||
- `lib/keys.mjs` — multi-key auth, per-key namespacing, identity layer. Carries OCP's per-key isolation model into OLP. **✅ Phase 2 — D44 core + D45 server integration + D46 owner gating shipped (validateKey on every /v1/* + /health; chain filtered by providers_enabled; touchLastUsed fires post-response; /health payload trimmed for non-owner; X-OLP-Fallback-Detail gated by fallback_detail_header_policy).**
|
||||
- `bin/olp-keys.mjs` — keygen CLI bootstrap surface per ADR 0007 § 9.1. **✅ Shipped at D47.** Subcommands: `keygen [--owner|--name=X|--providers=csv|--force]`, `list [--owner-only|--include-revoked]`, `revoke --id=X`. Plaintext token printed once on keygen. Installed via `package.json bin` so `npx olp-keys ...` works (also `npm run olp-keys ...`).
|
||||
- `lib/audit.mjs` — append-only ndjson audit per ADR 0007 § 6.2 + § 8. **🟡 D45 — appendAuditEvent + getAuditDropCount shipped. Fires per /v1/chat/completions + /v1/models request including 401/403/5xx paths. Warn+1-retry on append failure; no memory buffer at Phase 2 (forward path).**
|
||||
- `lib/audit.mjs` — append-only ndjson audit per ADR 0007 § 6.2 + § 8. **🟡 D45 — appendAuditEvent + getAuditDropCount shipped. Fires per /v1/chat/completions + /v1/models request including 401/403/5xx paths. Warn+1-retry on append failure; no memory buffer at Phase 2 (forward path).** **D52 will extend with daily rotation per ADR 0008 § 5.**
|
||||
- `lib/audit-query.mjs` — audit ndjson aggregate query layer per ADR 0008 § 4. **🟡 D49 — discoverAuditFiles + readAuditWindow + aggregateRequests + topFallbackChains + spendTrendDaily + cacheHitRateWindow shipped. Cross-file walk over `audit.ndjson` (live) + `audit-YYYY-MM-DD.ndjson` (rotated). PII guard: aggregate shapes never include message content. In-memory scan per request (ADR 0008 Lane 2 = A; SQLite hybrid deferred to ADR 0007 § 13 trigger).**
|
||||
- `dashboard.html` — owner-only multi-provider dashboard (quota panels, fallback rate, cache hit rate). **📋 Planned (Phase 6) — not yet authored.**
|
||||
- `models-registry.json` — single source of truth for `(provider, model) → metadata`. SPOT.
|
||||
- `ALIGNMENT.md` — the constitution. Binding for any plugin / entry-surface / IR change.
|
||||
|
||||
@@ -4,6 +4,33 @@ All notable changes to OLP land here. Per `CLAUDE.md` release_kit overlay, this
|
||||
|
||||
## Unreleased
|
||||
|
||||
### D49 — `lib/audit-query.mjs` audit aggregate query layer (Phase 3)
|
||||
|
||||
Second Phase 3 D-day. Implements ADR 0008 § 4 query API. Pure in-memory ndjson scan; cross-file walk over `audit.ndjson` (live) + `audit-YYYY-MM-DD.ndjson` (rotated). No server.mjs integration in this D-day (D50 wires the consuming endpoints).
|
||||
|
||||
- **New file `lib/audit-query.mjs`** (~370 lines): 5 public API functions per ADR 0008 § 4.1:
|
||||
- `discoverAuditFiles({ olpHome })` — filesystem scan; returns `Map<date|'live', path>`.
|
||||
- `readAuditWindow({ startMs, endMs, olpHome, logEvent })` — generator over events in half-open window [startMs, endMs). Walks rotated date files + live file. Skips malformed lines + logs warn.
|
||||
- `aggregateRequests({ windowMs, olpHome })` — counts + status buckets + by_provider + by_owner_tier + by_path + median/p95 latency over rolling window.
|
||||
- `topFallbackChains({ windowMs, limit, olpHome })` — top-N chains by trigger count from events with `fallback_hops > 0`. Tied-count tiebreak: ascending first_seen.
|
||||
- `spendTrendDaily({ days, olpHome })` — daily series ending today with sparse-fill for zero-request days. Per-day request_count + median latency + by_provider breakdown.
|
||||
- `cacheHitRateWindow({ windowMs, olpHome })` — audit-derived cache hit rate (bypass excluded from denominator); per-provider + overall.
|
||||
- **PII discipline** (ADR 0008 § 4.3): every aggregate function relays only schema fields; never message content. Suite 23g actively asserts the absence of `content`/`message`/`messages`/`prompt`/`response`/`body` keys in every aggregate output.
|
||||
- **Cross-file walk semantics** (ADR 0008 § 4.2): half-open window [startMs, endMs); date-range computed once from window bounds; each rotated date file checked; live `audit.ndjson` always checked (it covers today regardless of whether the window endpoint is past midnight).
|
||||
- **`spendTrendDaily` calendar-date semantics**: `days: N` returns "last N calendar UTC dates ending today" — NOT "events within a rolling N×86400-ms window" (which would span N+1 distinct UTC dates and produce off-by-one buckets at non-midnight call times). Computed via `for (let i = days-1; i >= 0; i--) dates.push(_utcDateFromMs(now - i*86400*1000));`.
|
||||
- **`cacheHitRateWindow` denominator**: hit_rate = hit / (hit + miss). Bypass is intentional non-cacheable (Anthropic cache_control marker), NOT a cache miss; excluding it from the denominator gives a clean cache-effectiveness signal.
|
||||
- **Test surface (Suite 23, +27 tests — 544 → 571):**
|
||||
- 23a-1..4: `discoverAuditFiles` (empty dir / live only / live+rotated / non-audit files ignored)
|
||||
- 23b-1..6: `readAuditWindow` (all-coverage / single-day / half-open exclusivity / empty window / missing files / malformed-skip with warn)
|
||||
- 23c-1..4: `aggregateRequests` (counts + status buckets + by_provider; by_owner_tier; median+p95 latency over realistic distribution; invalid windowMs rejection)
|
||||
- 23d-1..4: `topFallbackChains` (sort desc by count; limit truncation; fallback_hops=0 excluded; first_seen/last_seen carried)
|
||||
- 23e-1..3: `spendTrendDaily` (N-day range correctness; populated day breakdown; empty day sparse-fill)
|
||||
- 23f-1..3: `cacheHitRateWindow` (overall + per-provider hit_rate; bypass not in denominator; cache_status=null events excluded)
|
||||
- 23g-1..3: PII guard for `aggregateRequests` / `spendTrendDaily` / `topFallbackChains` + `cacheHitRateWindow` — every output JSON-stringified + scanned for forbidden PII keys
|
||||
- **Documentation:** AGENTS.md `lib/audit-query.mjs` new entry; `lib/audit.mjs` note added that D52 extends with daily rotation.
|
||||
- **Test count:** 544 → 571 (+27 D49 tests).
|
||||
- **Authority:** ADR 0008 § 4 (query API surface) + § 5 (rotation file naming pattern) + § 3 (storage layout); ADR 0007 § 8 (audit ndjson event schema — input data); CLAUDE.md `release_kit overlay phase_rolling_mode` — under Unreleased; standing autopilot grant.
|
||||
|
||||
### D48 — ADR 0008 Phase 3 design draft (Dashboard + audit query layer)
|
||||
|
||||
First Phase 3 D-day. Design-only. Ratifies the storage / query model / rotation / dashboard / refresh / scope decisions ahead of D49+ implementation D-days. Opens ADR 0007 § 12 deferral for Dashboard + audit query layer + rotation.
|
||||
|
||||
@@ -0,0 +1,489 @@
|
||||
/**
|
||||
* lib/audit-query.mjs — OLP audit ndjson aggregate query layer (Phase 3 / D49)
|
||||
*
|
||||
* Authority: ADR 0008 § 4 (query API surface) + § 5 (rotation file naming) +
|
||||
* § 3 (storage layout). Reads `~/.olp/logs/audit.ndjson` (live) +
|
||||
* `audit-YYYY-MM-DD.ndjson` (rotated dailies) and returns aggregate
|
||||
* summaries shaped for the Dashboard endpoints (D50).
|
||||
*
|
||||
* Query model (ADR 0008 Lane 2 = A): in-memory scan per request. O(N) where
|
||||
* N = total lines in the date range. Family-scale acceptable; SQLite hybrid
|
||||
* (ADR 0007 § 13) is the documented forward path when N+queries get slow.
|
||||
*
|
||||
* PII discipline (ADR 0007 § 8 + ADR 0008 § 4.3): event shape is hash + shape
|
||||
* only — no message content, no response content, no raw tokens. This module
|
||||
* MUST NOT introduce derived fields that reveal content. Every aggregate
|
||||
* function asserts the input event has the expected shape but does NOT inspect
|
||||
* or relay message bodies.
|
||||
*
|
||||
* What is NOT in this module (intentional split):
|
||||
* - Daily rotation trigger (D52, lib/audit.mjs extension)
|
||||
* - Server endpoints that consume these queries (D50, server.mjs)
|
||||
* - Dashboard HTML / DOM render (D51, dashboard.html)
|
||||
*/
|
||||
|
||||
import { readFileSync, readdirSync, existsSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { homedir } from 'node:os';
|
||||
|
||||
// ── Constants ─────────────────────────────────────────────────────────────
|
||||
|
||||
const DEFAULT_OLP_HOME = join(homedir(), '.olp');
|
||||
const OLP_HOME_ENV = 'OLP_HOME';
|
||||
const LIVE_AUDIT_FILE = 'audit.ndjson';
|
||||
const ROTATED_FILE_PATTERN = /^audit-(\d{4}-\d{2}-\d{2})\.ndjson$/;
|
||||
|
||||
// ── Path helpers ──────────────────────────────────────────────────────────
|
||||
|
||||
function _resolveOlpHome(opts) {
|
||||
if (opts?.olpHome) return opts.olpHome;
|
||||
if (process.env[OLP_HOME_ENV]) return process.env[OLP_HOME_ENV];
|
||||
return DEFAULT_OLP_HOME;
|
||||
}
|
||||
|
||||
function _logsDir(opts) {
|
||||
return join(_resolveOlpHome(opts), 'logs');
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the UTC-date string (YYYY-MM-DD) for an ISO-8601 timestamp.
|
||||
*/
|
||||
function _utcDateString(isoTs) {
|
||||
if (typeof isoTs !== 'string' || isoTs.length < 10) return null;
|
||||
return isoTs.slice(0, 10);
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the UTC-date string for an epoch-ms.
|
||||
*/
|
||||
function _utcDateFromMs(ms) {
|
||||
return new Date(ms).toISOString().slice(0, 10);
|
||||
}
|
||||
|
||||
/**
|
||||
* Inclusive range of UTC date strings from startDate to endDate (both
|
||||
* YYYY-MM-DD). Returns the list in ascending order. Safe for spans up to
|
||||
* several years (no upper bound enforced — caller's responsibility).
|
||||
*/
|
||||
function _dateRange(startDate, endDate) {
|
||||
const dates = [];
|
||||
const cur = new Date(`${startDate}T00:00:00Z`);
|
||||
const end = new Date(`${endDate}T00:00:00Z`);
|
||||
while (cur <= end) {
|
||||
dates.push(cur.toISOString().slice(0, 10));
|
||||
cur.setUTCDate(cur.getUTCDate() + 1);
|
||||
}
|
||||
return dates;
|
||||
}
|
||||
|
||||
// ── File enumeration ──────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Discover audit files in the logs directory. Returns a Map from
|
||||
* date-string ('YYYY-MM-DD' or 'live' for the un-rotated file) to absolute
|
||||
* file path. The 'live' entry is `audit.ndjson` if present; date-string
|
||||
* entries are the rotated daily files matching `audit-YYYY-MM-DD.ndjson`.
|
||||
*
|
||||
* Returns an empty Map if the logs directory does not exist or is empty.
|
||||
* Caller responsible for date filtering.
|
||||
*
|
||||
* @param {object} [opts] - { olpHome }
|
||||
* @returns {Map<string, string>} date-string → absolute file path
|
||||
*/
|
||||
export function discoverAuditFiles(opts = {}) {
|
||||
const dir = _logsDir(opts);
|
||||
const out = new Map();
|
||||
if (!existsSync(dir)) return out;
|
||||
let entries;
|
||||
try { entries = readdirSync(dir); } catch { return out; }
|
||||
for (const name of entries) {
|
||||
if (name === LIVE_AUDIT_FILE) {
|
||||
out.set('live', join(dir, name));
|
||||
continue;
|
||||
}
|
||||
const m = ROTATED_FILE_PATTERN.exec(name);
|
||||
if (m) {
|
||||
out.set(m[1], join(dir, name));
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ── Line-level read + parse ───────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Parse a single ndjson line. Returns the event object on success, or
|
||||
* null on parse error. Caller logs warn for null returns.
|
||||
*/
|
||||
function _parseLine(line) {
|
||||
if (!line) return null;
|
||||
try {
|
||||
const obj = JSON.parse(line);
|
||||
if (typeof obj !== 'object' || obj === null) return null;
|
||||
return obj;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read all events from a single file, skipping malformed lines.
|
||||
* Logs warn (via logEvent override or console) for each malformed line so
|
||||
* a corrupted day doesn't kill the query.
|
||||
*
|
||||
* @param {string} path
|
||||
* @param {(level: string, event: string, data?: object) => void} [logEvent]
|
||||
* @returns {Array<object>} parsed events
|
||||
*/
|
||||
function _readFileEvents(path, logEvent) {
|
||||
let raw;
|
||||
try {
|
||||
raw = readFileSync(path, 'utf-8');
|
||||
} catch (err) {
|
||||
// Re-throw read errors (EACCES, ENOENT during race) so the dashboard
|
||||
// endpoint surfaces 500 with diagnostic per ADR 0008 § 4.4.
|
||||
throw new Error(`audit_query_read_failed: ${path}: ${err?.message ?? err}`);
|
||||
}
|
||||
const lines = raw.split('\n');
|
||||
const events = [];
|
||||
let skipped = 0;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i].trim();
|
||||
if (!line) continue;
|
||||
const ev = _parseLine(line);
|
||||
if (ev === null) {
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
events.push(ev);
|
||||
}
|
||||
if (skipped > 0 && logEvent) {
|
||||
logEvent('warn', 'audit_query_skip_malformed', { path, skipped });
|
||||
}
|
||||
return events;
|
||||
}
|
||||
|
||||
// ── Public API ────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Iterate all audit events in [startMs, endMs). Walks the rotated daily
|
||||
* files whose date overlaps the range + today's live audit.ndjson. Within
|
||||
* each file, includes only events whose `ts` falls in the window.
|
||||
*
|
||||
* Per ADR 0008 § 4.2: window semantics are half-open [start, end).
|
||||
*
|
||||
* @param {object} args
|
||||
* @param {number} args.startMs - epoch-ms inclusive lower bound
|
||||
* @param {number} args.endMs - epoch-ms exclusive upper bound
|
||||
* @param {string} [args.olpHome]
|
||||
* @param {(level: string, event: string, data?: object) => void} [args.logEvent]
|
||||
* @yields {object} parsed audit event
|
||||
*/
|
||||
export function* readAuditWindow({ startMs, endMs, olpHome, logEvent } = {}) {
|
||||
if (typeof startMs !== 'number' || typeof endMs !== 'number') {
|
||||
throw new Error('readAuditWindow: startMs and endMs (numbers) are required');
|
||||
}
|
||||
if (endMs <= startMs) return; // empty window
|
||||
|
||||
const files = discoverAuditFiles({ olpHome });
|
||||
if (files.size === 0) return;
|
||||
|
||||
// Walk all dates in [startMs, endMs) plus the live file (today).
|
||||
const startDate = _utcDateFromMs(startMs);
|
||||
const endDate = _utcDateFromMs(endMs - 1); // endMs is exclusive
|
||||
const dateList = _dateRange(startDate, endDate);
|
||||
|
||||
for (const date of dateList) {
|
||||
const path = files.get(date);
|
||||
if (!path) continue;
|
||||
const events = _readFileEvents(path, logEvent);
|
||||
for (const ev of events) {
|
||||
const tsStr = ev.ts;
|
||||
if (typeof tsStr !== 'string') continue;
|
||||
const tsMs = Date.parse(tsStr);
|
||||
if (Number.isNaN(tsMs)) continue;
|
||||
if (tsMs >= startMs && tsMs < endMs) yield ev;
|
||||
}
|
||||
}
|
||||
|
||||
// Live file (today) — always check; date may overlap window's end.
|
||||
const livePath = files.get('live');
|
||||
if (livePath) {
|
||||
const events = _readFileEvents(livePath, logEvent);
|
||||
for (const ev of events) {
|
||||
const tsStr = ev.ts;
|
||||
if (typeof tsStr !== 'string') continue;
|
||||
const tsMs = Date.parse(tsStr);
|
||||
if (Number.isNaN(tsMs)) continue;
|
||||
if (tsMs >= startMs && tsMs < endMs) yield ev;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Aggregate request shape over a rolling window ending at "now".
|
||||
*
|
||||
* Returns:
|
||||
* {
|
||||
* window: { startMs, endMs },
|
||||
* request_count, status_2xx, status_4xx, status_5xx,
|
||||
* by_provider: { [providerKey]: { count, cache_hit, cache_miss, cache_bypass, fallback_count } },
|
||||
* by_owner_tier: { owner: N, guest: N, anonymous: N },
|
||||
* by_path: { '/v1/chat/completions': N, '/v1/models': N, ... },
|
||||
* median_latency_ms, p95_latency_ms,
|
||||
* }
|
||||
*
|
||||
* Per ADR 0008 § 4.1 + § 4.3 PII discipline: aggregates count + categorical
|
||||
* breakdowns only, NEVER message content.
|
||||
*
|
||||
* @param {object} args
|
||||
* @param {number} args.windowMs - duration in ms; window = [now - windowMs, now)
|
||||
* @param {string} [args.olpHome]
|
||||
* @param {(level, event, data?) => void} [args.logEvent]
|
||||
* @param {() => number} [args._nowFn] - injectable for testing
|
||||
*/
|
||||
export function aggregateRequests({ windowMs, olpHome, logEvent, _nowFn } = {}) {
|
||||
if (typeof windowMs !== 'number' || windowMs <= 0) {
|
||||
throw new Error('aggregateRequests: windowMs (positive number) is required');
|
||||
}
|
||||
const now = (_nowFn ?? Date.now)();
|
||||
const startMs = now - windowMs;
|
||||
const endMs = now;
|
||||
|
||||
const result = {
|
||||
window: { startMs, endMs },
|
||||
request_count: 0,
|
||||
status_2xx: 0,
|
||||
status_4xx: 0,
|
||||
status_5xx: 0,
|
||||
by_provider: {},
|
||||
by_owner_tier: { owner: 0, guest: 0, anonymous: 0 },
|
||||
by_path: {},
|
||||
median_latency_ms: 0,
|
||||
p95_latency_ms: 0,
|
||||
};
|
||||
const latencies = [];
|
||||
|
||||
for (const ev of readAuditWindow({ startMs, endMs, olpHome, logEvent })) {
|
||||
result.request_count++;
|
||||
|
||||
// Status code bucket
|
||||
const sc = typeof ev.status_code === 'number' ? ev.status_code : 0;
|
||||
if (sc >= 200 && sc < 300) result.status_2xx++;
|
||||
else if (sc >= 400 && sc < 500) result.status_4xx++;
|
||||
else if (sc >= 500) result.status_5xx++;
|
||||
|
||||
// By provider
|
||||
if (typeof ev.provider === 'string' && ev.provider.length > 0) {
|
||||
const p = result.by_provider[ev.provider] ??= {
|
||||
count: 0, cache_hit: 0, cache_miss: 0, cache_bypass: 0, fallback_count: 0,
|
||||
};
|
||||
p.count++;
|
||||
if (ev.cache_status === 'hit') p.cache_hit++;
|
||||
else if (ev.cache_status === 'miss') p.cache_miss++;
|
||||
else if (ev.cache_status === 'bypass') p.cache_bypass++;
|
||||
if (typeof ev.fallback_hops === 'number' && ev.fallback_hops > 0) p.fallback_count++;
|
||||
}
|
||||
|
||||
// By owner tier
|
||||
if (ev.owner_tier === 'owner') result.by_owner_tier.owner++;
|
||||
else if (ev.owner_tier === 'guest') result.by_owner_tier.guest++;
|
||||
else result.by_owner_tier.anonymous++;
|
||||
|
||||
// By path
|
||||
if (typeof ev.path === 'string' && ev.path.length > 0) {
|
||||
result.by_path[ev.path] = (result.by_path[ev.path] ?? 0) + 1;
|
||||
}
|
||||
|
||||
// Latency
|
||||
if (typeof ev.latency_ms === 'number' && ev.latency_ms >= 0) {
|
||||
latencies.push(ev.latency_ms);
|
||||
}
|
||||
}
|
||||
|
||||
// Median + p95 over sorted latencies
|
||||
if (latencies.length > 0) {
|
||||
latencies.sort((a, b) => a - b);
|
||||
const midIdx = Math.floor(latencies.length / 2);
|
||||
result.median_latency_ms = latencies.length % 2 === 0
|
||||
? Math.round((latencies[midIdx - 1] + latencies[midIdx]) / 2)
|
||||
: latencies[midIdx];
|
||||
const p95Idx = Math.min(latencies.length - 1, Math.floor(latencies.length * 0.95));
|
||||
result.p95_latency_ms = latencies[p95Idx];
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Top-N fallback chains by trigger count in window. A "chain" is the
|
||||
* `tried_providers` array from an event with fallback_hops > 0. Returns
|
||||
* sorted array descending by count; ties broken by earliest first_seen.
|
||||
*
|
||||
* [{ chain: ['anthropic', 'openai'], count: 42, first_seen, last_seen }, ...]
|
||||
*
|
||||
* @param {object} args
|
||||
* @param {number} args.windowMs
|
||||
* @param {number} [args.limit=10]
|
||||
* @param {string} [args.olpHome]
|
||||
* @param {(level, event, data?) => void} [args.logEvent]
|
||||
* @param {() => number} [args._nowFn]
|
||||
*/
|
||||
export function topFallbackChains({ windowMs, limit = 10, olpHome, logEvent, _nowFn } = {}) {
|
||||
if (typeof windowMs !== 'number' || windowMs <= 0) {
|
||||
throw new Error('topFallbackChains: windowMs (positive number) is required');
|
||||
}
|
||||
const now = (_nowFn ?? Date.now)();
|
||||
const startMs = now - windowMs;
|
||||
const endMs = now;
|
||||
|
||||
// Map chain-key (joined string) → aggregate
|
||||
const chains = new Map();
|
||||
for (const ev of readAuditWindow({ startMs, endMs, olpHome, logEvent })) {
|
||||
if (typeof ev.fallback_hops !== 'number' || ev.fallback_hops <= 0) continue;
|
||||
if (!Array.isArray(ev.tried_providers) || ev.tried_providers.length < 2) continue;
|
||||
const key = ev.tried_providers.join('→');
|
||||
const entry = chains.get(key);
|
||||
const ts = typeof ev.ts === 'string' ? ev.ts : null;
|
||||
if (entry === undefined) {
|
||||
chains.set(key, {
|
||||
chain: [...ev.tried_providers],
|
||||
count: 1,
|
||||
first_seen: ts,
|
||||
last_seen: ts,
|
||||
});
|
||||
} else {
|
||||
entry.count++;
|
||||
if (ts && (!entry.first_seen || ts < entry.first_seen)) entry.first_seen = ts;
|
||||
if (ts && (!entry.last_seen || ts > entry.last_seen)) entry.last_seen = ts;
|
||||
}
|
||||
}
|
||||
|
||||
// Sort desc by count, ascending by first_seen on ties
|
||||
const arr = [...chains.values()];
|
||||
arr.sort((a, b) => {
|
||||
if (b.count !== a.count) return b.count - a.count;
|
||||
if (a.first_seen && b.first_seen) return a.first_seen < b.first_seen ? -1 : a.first_seen > b.first_seen ? 1 : 0;
|
||||
return 0;
|
||||
});
|
||||
return arr.slice(0, limit);
|
||||
}
|
||||
|
||||
/**
|
||||
* Daily series of request_count + median latency_ms + by_provider over N
|
||||
* UTC days ending today. Sparse-fills zero-request days. Returns ascending
|
||||
* by date:
|
||||
*
|
||||
* [{ date: '2026-05-22', request_count, median_latency_ms, by_provider }, ...]
|
||||
*
|
||||
* by_provider is { [providerKey]: count } per day.
|
||||
*
|
||||
* @param {object} args
|
||||
* @param {number} args.days
|
||||
* @param {string} [args.olpHome]
|
||||
* @param {(level, event, data?) => void} [args.logEvent]
|
||||
* @param {() => number} [args._nowFn]
|
||||
*/
|
||||
export function spendTrendDaily({ days, olpHome, logEvent, _nowFn } = {}) {
|
||||
if (typeof days !== 'number' || days <= 0) {
|
||||
throw new Error('spendTrendDaily: days (positive number) is required');
|
||||
}
|
||||
const now = (_nowFn ?? Date.now)();
|
||||
|
||||
// Compute the N UTC dates ending today (inclusive). Semantics: "last N
|
||||
// calendar dates ending today" — NOT "events within a rolling N*86400-ms
|
||||
// window ago" (the latter would span N+1 distinct UTC dates and produce
|
||||
// off-by-one buckets at non-midnight call times).
|
||||
const dates = [];
|
||||
for (let i = days - 1; i >= 0; i--) {
|
||||
dates.push(_utcDateFromMs(now - i * 86400 * 1000));
|
||||
}
|
||||
// Window covers the start of the first date through "now" so readAuditWindow
|
||||
// sees every event whose ts falls in any of the N dates' UTC days.
|
||||
const startMs = Date.parse(`${dates[0]}T00:00:00Z`);
|
||||
const endMs = now;
|
||||
|
||||
// Bucket by UTC date
|
||||
const buckets = new Map();
|
||||
for (const ev of readAuditWindow({ startMs, endMs, olpHome, logEvent })) {
|
||||
const date = _utcDateString(ev.ts);
|
||||
if (!date) continue;
|
||||
const b = buckets.get(date) ?? { request_count: 0, latencies: [], by_provider: {} };
|
||||
b.request_count++;
|
||||
if (typeof ev.latency_ms === 'number') b.latencies.push(ev.latency_ms);
|
||||
if (typeof ev.provider === 'string' && ev.provider.length > 0) {
|
||||
b.by_provider[ev.provider] = (b.by_provider[ev.provider] ?? 0) + 1;
|
||||
}
|
||||
buckets.set(date, b);
|
||||
}
|
||||
|
||||
// Sparse-fill using the precomputed dates list (preserves ascending order)
|
||||
return dates.map(date => {
|
||||
const b = buckets.get(date);
|
||||
if (b) {
|
||||
b.latencies.sort((a, b) => a - b);
|
||||
const midIdx = Math.floor(b.latencies.length / 2);
|
||||
const median = b.latencies.length === 0 ? 0
|
||||
: b.latencies.length % 2 === 0
|
||||
? Math.round((b.latencies[midIdx - 1] + b.latencies[midIdx]) / 2)
|
||||
: b.latencies[midIdx];
|
||||
return {
|
||||
date,
|
||||
request_count: b.request_count,
|
||||
median_latency_ms: median,
|
||||
by_provider: b.by_provider,
|
||||
};
|
||||
}
|
||||
return { date, request_count: 0, median_latency_ms: 0, by_provider: {} };
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Audit-derived cache hit rate over the window. Differs from
|
||||
* `cacheStore.stats()` in server.mjs: that is the live in-process counter;
|
||||
* this is the audit-side rate scoped to the rolling window.
|
||||
*
|
||||
* { window: { startMs, endMs }, total, hit, miss, bypass, hit_rate, by_provider }
|
||||
*
|
||||
* @param {object} args
|
||||
* @param {number} args.windowMs
|
||||
* @param {string} [args.olpHome]
|
||||
* @param {(level, event, data?) => void} [args.logEvent]
|
||||
* @param {() => number} [args._nowFn]
|
||||
*/
|
||||
export function cacheHitRateWindow({ windowMs, olpHome, logEvent, _nowFn } = {}) {
|
||||
if (typeof windowMs !== 'number' || windowMs <= 0) {
|
||||
throw new Error('cacheHitRateWindow: windowMs (positive number) is required');
|
||||
}
|
||||
const now = (_nowFn ?? Date.now)();
|
||||
const startMs = now - windowMs;
|
||||
const endMs = now;
|
||||
|
||||
let total = 0, hit = 0, miss = 0, bypass = 0;
|
||||
const by_provider = {};
|
||||
|
||||
for (const ev of readAuditWindow({ startMs, endMs, olpHome, logEvent })) {
|
||||
if (ev.cache_status === null || ev.cache_status === undefined) continue;
|
||||
total++;
|
||||
const p = typeof ev.provider === 'string' && ev.provider.length > 0 ? ev.provider : '__unknown__';
|
||||
const pe = by_provider[p] ??= { total: 0, hit: 0, miss: 0, bypass: 0, hit_rate: 0 };
|
||||
pe.total++;
|
||||
if (ev.cache_status === 'hit') { hit++; pe.hit++; }
|
||||
else if (ev.cache_status === 'miss') { miss++; pe.miss++; }
|
||||
else if (ev.cache_status === 'bypass') { bypass++; pe.bypass++; }
|
||||
}
|
||||
|
||||
// Compute hit_rate per provider + overall (excludes bypass from denominator
|
||||
// since bypass-by-cache_control is intentional non-cacheable, not a cache miss).
|
||||
for (const p of Object.values(by_provider)) {
|
||||
const denom = p.hit + p.miss;
|
||||
p.hit_rate = denom > 0 ? p.hit / denom : 0;
|
||||
}
|
||||
const overallDenom = hit + miss;
|
||||
const hit_rate = overallDenom > 0 ? hit / overallDenom : 0;
|
||||
|
||||
return {
|
||||
window: { startMs, endMs },
|
||||
total, hit, miss, bypass, hit_rate, by_provider,
|
||||
};
|
||||
}
|
||||
@@ -10897,3 +10897,450 @@ describe('Suite 22 — D47 keygen CLI (bin/olp-keys.mjs, ADR 0007 § 9.1)', () =
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// ── Suite 23: D49 lib/audit-query.mjs (Phase 3 audit aggregate query layer) ──
|
||||
//
|
||||
// Unit tests for the in-memory audit query layer per ADR 0008 § 4.
|
||||
// - discoverAuditFiles: filesystem scan + date-suffix recognition
|
||||
// - readAuditWindow: window filtering + cross-file walk + malformed skip
|
||||
// - aggregateRequests: count + median/p95 latency + by_provider + by_owner_tier + by_path
|
||||
// - topFallbackChains: sort + tied-count tiebreak + tried_providers shape
|
||||
// - spendTrendDaily: sparse-fill + UTC day buckets
|
||||
// - cacheHitRateWindow: per-provider + bypass-not-in-denominator
|
||||
// - PII guard: aggregate shapes never include message content
|
||||
|
||||
import {
|
||||
discoverAuditFiles,
|
||||
readAuditWindow,
|
||||
aggregateRequests,
|
||||
topFallbackChains,
|
||||
spendTrendDaily,
|
||||
cacheHitRateWindow,
|
||||
} from './lib/audit-query.mjs';
|
||||
import { mkdirSync, writeFileSync as fsWriteFileSyncForS23 } from 'node:fs';
|
||||
|
||||
describe('Suite 23 — D49 lib/audit-query.mjs (Phase 3 audit aggregate query layer, ADR 0008 § 4)', () => {
|
||||
|
||||
// Helper: write synthetic audit ndjson files to a tmpdir's logs/ subdir.
|
||||
// entries is { 'live': [...events] | 'YYYY-MM-DD': [...events] }
|
||||
function setupAuditFiles(tmp, entries) {
|
||||
const logsDir = _pathJoinForSetup(tmp, 'logs');
|
||||
mkdirSync(logsDir, { recursive: true, mode: 0o700 });
|
||||
for (const [date, events] of Object.entries(entries)) {
|
||||
const filename = date === 'live' ? 'audit.ndjson' : `audit-${date}.ndjson`;
|
||||
const path = _pathJoinForSetup(logsDir, filename);
|
||||
const lines = events.map(ev => JSON.stringify(ev)).join('\n') + '\n';
|
||||
fsWriteFileSyncForS23(path, lines, { mode: 0o600 });
|
||||
}
|
||||
}
|
||||
|
||||
function makeEvent(overrides = {}) {
|
||||
return {
|
||||
ts: new Date().toISOString(),
|
||||
key_id: 'k1',
|
||||
owner_tier: 'owner',
|
||||
method: 'POST',
|
||||
path: '/v1/chat/completions',
|
||||
provider: 'anthropic',
|
||||
model: 'claude-sonnet-4-6',
|
||||
status_code: 200,
|
||||
latency_ms: 100,
|
||||
cache_status: 'miss',
|
||||
fallback_hops: 0,
|
||||
tried_providers: ['anthropic'],
|
||||
error_code: null,
|
||||
ir_request_hash: 'abc123',
|
||||
chain_id: 'c1',
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('23a — discoverAuditFiles', () => {
|
||||
let TMP;
|
||||
before(() => { TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23a-')); });
|
||||
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
||||
|
||||
it('23a-1: empty logs dir → empty Map', () => {
|
||||
const r = discoverAuditFiles({ olpHome: TMP });
|
||||
assert.equal(r.size, 0);
|
||||
});
|
||||
|
||||
it('23a-2: only audit.ndjson → Map with "live" key', () => {
|
||||
setupAuditFiles(TMP, { 'live': [makeEvent()] });
|
||||
const r = discoverAuditFiles({ olpHome: TMP });
|
||||
assert.equal(r.size, 1);
|
||||
assert.ok(r.has('live'));
|
||||
});
|
||||
|
||||
it('23a-3: rotated files + live → all recognized', () => {
|
||||
setupAuditFiles(TMP, {
|
||||
'live': [makeEvent()],
|
||||
'2026-05-24': [makeEvent({ ts: '2026-05-24T12:00:00Z' })],
|
||||
'2026-05-23': [makeEvent({ ts: '2026-05-23T12:00:00Z' })],
|
||||
});
|
||||
const r = discoverAuditFiles({ olpHome: TMP });
|
||||
assert.equal(r.size, 3);
|
||||
assert.ok(r.has('live'));
|
||||
assert.ok(r.has('2026-05-24'));
|
||||
assert.ok(r.has('2026-05-23'));
|
||||
});
|
||||
|
||||
it('23a-4: non-audit files ignored', () => {
|
||||
setupAuditFiles(TMP, { 'live': [makeEvent()] });
|
||||
fsWriteFileSyncForS23(_pathJoinForSetup(TMP, 'logs', 'some-random.log'), 'noise\n', { mode: 0o600 });
|
||||
fsWriteFileSyncForS23(_pathJoinForSetup(TMP, 'logs', 'audit-invalid-date.ndjson'), 'noise\n', { mode: 0o600 });
|
||||
const r = discoverAuditFiles({ olpHome: TMP });
|
||||
// live + 3 prior rotated from 23a-3 (state carries within describe block); ignore random files
|
||||
assert.ok(r.size >= 1);
|
||||
for (const key of r.keys()) {
|
||||
assert.ok(key === 'live' || /^\d{4}-\d{2}-\d{2}$/.test(key));
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('23b — readAuditWindow', () => {
|
||||
let TMP;
|
||||
before(() => {
|
||||
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23b-'));
|
||||
setupAuditFiles(TMP, {
|
||||
'2026-05-23': [
|
||||
makeEvent({ ts: '2026-05-23T10:00:00Z', provider: 'anthropic' }),
|
||||
makeEvent({ ts: '2026-05-23T20:00:00Z', provider: 'openai' }),
|
||||
],
|
||||
'2026-05-24': [
|
||||
makeEvent({ ts: '2026-05-24T10:00:00Z', provider: 'mistral' }),
|
||||
],
|
||||
'live': [
|
||||
makeEvent({ ts: '2026-05-25T10:00:00Z', provider: 'anthropic' }),
|
||||
],
|
||||
});
|
||||
});
|
||||
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
||||
|
||||
it('23b-1: window covering all → all 4 events', () => {
|
||||
const start = Date.parse('2026-05-23T00:00:00Z');
|
||||
const end = Date.parse('2026-05-26T00:00:00Z');
|
||||
const events = [...readAuditWindow({ startMs: start, endMs: end, olpHome: TMP })];
|
||||
assert.equal(events.length, 4);
|
||||
});
|
||||
|
||||
it('23b-2: window covering only 2026-05-24 → 1 event (mistral)', () => {
|
||||
const start = Date.parse('2026-05-24T00:00:00Z');
|
||||
const end = Date.parse('2026-05-25T00:00:00Z');
|
||||
const events = [...readAuditWindow({ startMs: start, endMs: end, olpHome: TMP })];
|
||||
assert.equal(events.length, 1);
|
||||
assert.equal(events[0].provider, 'mistral');
|
||||
});
|
||||
|
||||
it('23b-3: half-open semantics — endMs exclusive', () => {
|
||||
// Event at 2026-05-23T20:00:00Z. Window endMs = exactly that ts → exclude.
|
||||
const start = Date.parse('2026-05-23T00:00:00Z');
|
||||
const end = Date.parse('2026-05-23T20:00:00Z');
|
||||
const events = [...readAuditWindow({ startMs: start, endMs: end, olpHome: TMP })];
|
||||
assert.equal(events.length, 1); // only the 10:00 event
|
||||
assert.equal(events[0].ts, '2026-05-23T10:00:00Z');
|
||||
});
|
||||
|
||||
it('23b-4: empty window (end <= start) → empty', () => {
|
||||
const events = [...readAuditWindow({ startMs: 1000, endMs: 1000, olpHome: TMP })];
|
||||
assert.equal(events.length, 0);
|
||||
const events2 = [...readAuditWindow({ startMs: 2000, endMs: 1000, olpHome: TMP })];
|
||||
assert.equal(events2.length, 0);
|
||||
});
|
||||
|
||||
it('23b-5: missing files → empty iteration (not an error)', () => {
|
||||
const TMP2 = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23b5-'));
|
||||
try {
|
||||
const events = [...readAuditWindow({
|
||||
startMs: Date.parse('2026-05-01T00:00:00Z'),
|
||||
endMs: Date.parse('2026-05-02T00:00:00Z'),
|
||||
olpHome: TMP2,
|
||||
})];
|
||||
assert.equal(events.length, 0);
|
||||
} finally {
|
||||
rmSync(TMP2, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('23b-6: malformed lines skipped + warn fired', () => {
|
||||
const TMP3 = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23b6-'));
|
||||
try {
|
||||
mkdirSync(_pathJoinForSetup(TMP3, 'logs'), { recursive: true, mode: 0o700 });
|
||||
const path = _pathJoinForSetup(TMP3, 'logs', 'audit.ndjson');
|
||||
const lines = [
|
||||
JSON.stringify(makeEvent({ ts: '2026-05-25T10:00:00Z' })),
|
||||
'{ this is not valid json',
|
||||
JSON.stringify(makeEvent({ ts: '2026-05-25T11:00:00Z' })),
|
||||
'',
|
||||
'definitely not json',
|
||||
JSON.stringify(makeEvent({ ts: '2026-05-25T12:00:00Z' })),
|
||||
].join('\n');
|
||||
fsWriteFileSyncForS23(path, lines, { mode: 0o600 });
|
||||
|
||||
let warnFired = false;
|
||||
const events = [...readAuditWindow({
|
||||
startMs: Date.parse('2026-05-25T00:00:00Z'),
|
||||
endMs: Date.parse('2026-05-26T00:00:00Z'),
|
||||
olpHome: TMP3,
|
||||
logEvent: (level, ev) => { if (level === 'warn' && ev === 'audit_query_skip_malformed') warnFired = true; },
|
||||
})];
|
||||
assert.equal(events.length, 3, 'malformed lines skipped, 3 valid events kept');
|
||||
assert.ok(warnFired, 'warn must fire for malformed lines');
|
||||
} finally {
|
||||
rmSync(TMP3, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('23c — aggregateRequests', () => {
|
||||
let TMP;
|
||||
before(() => {
|
||||
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23c-'));
|
||||
const now = Date.now();
|
||||
const t = (offsetMs) => new Date(now - offsetMs).toISOString();
|
||||
setupAuditFiles(TMP, {
|
||||
'live': [
|
||||
makeEvent({ ts: t(60000), provider: 'anthropic', cache_status: 'hit', owner_tier: 'owner', status_code: 200, latency_ms: 50 }),
|
||||
makeEvent({ ts: t(50000), provider: 'anthropic', cache_status: 'miss', owner_tier: 'owner', status_code: 200, latency_ms: 150 }),
|
||||
makeEvent({ ts: t(40000), provider: 'openai', cache_status: 'miss', owner_tier: 'guest', status_code: 200, latency_ms: 200 }),
|
||||
makeEvent({ ts: t(30000), provider: 'mistral', cache_status: 'bypass', owner_tier: 'owner', status_code: 401, latency_ms: 10 }),
|
||||
makeEvent({ ts: t(20000), provider: 'anthropic', cache_status: 'miss', owner_tier: 'owner', status_code: 503, latency_ms: 5000, fallback_hops: 1, tried_providers: ['anthropic', 'openai'] }),
|
||||
],
|
||||
});
|
||||
});
|
||||
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
||||
|
||||
it('23c-1: aggregateRequests counts requests + status buckets + by_provider', () => {
|
||||
const r = aggregateRequests({ windowMs: 86400 * 1000, olpHome: TMP });
|
||||
assert.equal(r.request_count, 5);
|
||||
assert.equal(r.status_2xx, 3); // 200x3
|
||||
assert.equal(r.status_4xx, 1); // 401
|
||||
assert.equal(r.status_5xx, 1); // 503
|
||||
assert.equal(r.by_provider.anthropic.count, 3);
|
||||
assert.equal(r.by_provider.anthropic.cache_hit, 1);
|
||||
assert.equal(r.by_provider.anthropic.cache_miss, 2);
|
||||
assert.equal(r.by_provider.anthropic.fallback_count, 1);
|
||||
assert.equal(r.by_provider.openai.count, 1);
|
||||
assert.equal(r.by_provider.mistral.cache_bypass, 1);
|
||||
});
|
||||
|
||||
it('23c-2: by_owner_tier breakdown', () => {
|
||||
const r = aggregateRequests({ windowMs: 86400 * 1000, olpHome: TMP });
|
||||
assert.equal(r.by_owner_tier.owner, 4);
|
||||
assert.equal(r.by_owner_tier.guest, 1);
|
||||
assert.equal(r.by_owner_tier.anonymous, 0);
|
||||
});
|
||||
|
||||
it('23c-3: median + p95 latency over [5, 50, 150, 200, 5000]', () => {
|
||||
const r = aggregateRequests({ windowMs: 86400 * 1000, olpHome: TMP });
|
||||
// sorted: 5, 10, 50, 150, 200; we have 5 events with latencies [50, 150, 200, 10, 5000]
|
||||
// sorted: [10, 50, 150, 200, 5000]; median (index 2) = 150
|
||||
assert.equal(r.median_latency_ms, 150);
|
||||
// p95 index = floor(5 * 0.95) = 4 → 5000
|
||||
assert.equal(r.p95_latency_ms, 5000);
|
||||
});
|
||||
|
||||
it('23c-4: throws on invalid windowMs', () => {
|
||||
assert.throws(() => aggregateRequests({ olpHome: TMP }), /windowMs.*required/);
|
||||
assert.throws(() => aggregateRequests({ windowMs: 0, olpHome: TMP }), /windowMs.*required/);
|
||||
assert.throws(() => aggregateRequests({ windowMs: -1, olpHome: TMP }), /windowMs.*required/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('23d — topFallbackChains', () => {
|
||||
let TMP;
|
||||
before(() => {
|
||||
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23d-'));
|
||||
const now = Date.now();
|
||||
const t = (offsetMs) => new Date(now - offsetMs).toISOString();
|
||||
setupAuditFiles(TMP, {
|
||||
'live': [
|
||||
// 3x anthropic→openai
|
||||
makeEvent({ ts: t(5000), fallback_hops: 1, tried_providers: ['anthropic', 'openai'] }),
|
||||
makeEvent({ ts: t(4000), fallback_hops: 1, tried_providers: ['anthropic', 'openai'] }),
|
||||
makeEvent({ ts: t(3000), fallback_hops: 1, tried_providers: ['anthropic', 'openai'] }),
|
||||
// 2x anthropic→mistral
|
||||
makeEvent({ ts: t(2500), fallback_hops: 1, tried_providers: ['anthropic', 'mistral'] }),
|
||||
makeEvent({ ts: t(2000), fallback_hops: 1, tried_providers: ['anthropic', 'mistral'] }),
|
||||
// 1x openai→mistral (single chain)
|
||||
makeEvent({ ts: t(1000), fallback_hops: 1, tried_providers: ['openai', 'mistral'] }),
|
||||
// events with fallback_hops:0 are excluded
|
||||
makeEvent({ ts: t(500), fallback_hops: 0, tried_providers: ['anthropic'] }),
|
||||
],
|
||||
});
|
||||
});
|
||||
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
||||
|
||||
it('23d-1: sorted desc by count', () => {
|
||||
const r = topFallbackChains({ windowMs: 86400 * 1000, olpHome: TMP });
|
||||
assert.equal(r.length, 3);
|
||||
assert.equal(r[0].count, 3);
|
||||
assert.deepEqual(r[0].chain, ['anthropic', 'openai']);
|
||||
assert.equal(r[1].count, 2);
|
||||
assert.deepEqual(r[1].chain, ['anthropic', 'mistral']);
|
||||
assert.equal(r[2].count, 1);
|
||||
assert.deepEqual(r[2].chain, ['openai', 'mistral']);
|
||||
});
|
||||
|
||||
it('23d-2: limit argument truncates result', () => {
|
||||
const r = topFallbackChains({ windowMs: 86400 * 1000, limit: 2, olpHome: TMP });
|
||||
assert.equal(r.length, 2);
|
||||
});
|
||||
|
||||
it('23d-3: events with fallback_hops=0 excluded from chains', () => {
|
||||
const r = topFallbackChains({ windowMs: 86400 * 1000, olpHome: TMP });
|
||||
const allCounts = r.reduce((s, c) => s + c.count, 0);
|
||||
assert.equal(allCounts, 6); // 3+2+1 — fallback_hops:0 event not counted
|
||||
});
|
||||
|
||||
it('23d-4: chain entries carry first_seen + last_seen', () => {
|
||||
const r = topFallbackChains({ windowMs: 86400 * 1000, olpHome: TMP });
|
||||
for (const c of r) {
|
||||
assert.ok(typeof c.first_seen === 'string');
|
||||
assert.ok(typeof c.last_seen === 'string');
|
||||
assert.ok(c.first_seen <= c.last_seen);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('23e — spendTrendDaily', () => {
|
||||
let TMP;
|
||||
before(() => {
|
||||
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23e-'));
|
||||
// 3 events on day-2, 1 event on day-0; day-1 has none (sparse fill test)
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const day1 = new Date(Date.now() - 86400 * 1000).toISOString().slice(0, 10);
|
||||
const day2 = new Date(Date.now() - 2 * 86400 * 1000).toISOString().slice(0, 10);
|
||||
setupAuditFiles(TMP, {
|
||||
[day2]: [
|
||||
makeEvent({ ts: `${day2}T10:00:00Z`, provider: 'anthropic', latency_ms: 100 }),
|
||||
makeEvent({ ts: `${day2}T11:00:00Z`, provider: 'openai', latency_ms: 200 }),
|
||||
makeEvent({ ts: `${day2}T12:00:00Z`, provider: 'anthropic', latency_ms: 50 }),
|
||||
],
|
||||
'live': [
|
||||
makeEvent({ ts: new Date().toISOString(), provider: 'mistral', latency_ms: 75 }),
|
||||
],
|
||||
});
|
||||
});
|
||||
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
||||
|
||||
it('23e-1: 3-day window returns 3 entries (sparse-fills empty days)', () => {
|
||||
const r = spendTrendDaily({ days: 3, olpHome: TMP });
|
||||
assert.equal(r.length, 3);
|
||||
// Each entry has a date + request_count (>=0)
|
||||
for (const e of r) {
|
||||
assert.ok(typeof e.date === 'string');
|
||||
assert.equal(e.date.length, 10);
|
||||
assert.ok(typeof e.request_count === 'number');
|
||||
assert.ok(typeof e.median_latency_ms === 'number');
|
||||
assert.ok(typeof e.by_provider === 'object');
|
||||
}
|
||||
});
|
||||
|
||||
it('23e-2: day with data has correct breakdown', () => {
|
||||
const r = spendTrendDaily({ days: 5, olpHome: TMP });
|
||||
const day2Date = new Date(Date.now() - 2 * 86400 * 1000).toISOString().slice(0, 10);
|
||||
const day2 = r.find(e => e.date === day2Date);
|
||||
assert.ok(day2);
|
||||
assert.equal(day2.request_count, 3);
|
||||
assert.equal(day2.by_provider.anthropic, 2);
|
||||
assert.equal(day2.by_provider.openai, 1);
|
||||
// median of [50, 100, 200] = 100
|
||||
assert.equal(day2.median_latency_ms, 100);
|
||||
});
|
||||
|
||||
it('23e-3: empty day has request_count=0 and empty by_provider', () => {
|
||||
const r = spendTrendDaily({ days: 3, olpHome: TMP });
|
||||
const day1Date = new Date(Date.now() - 86400 * 1000).toISOString().slice(0, 10);
|
||||
const day1 = r.find(e => e.date === day1Date);
|
||||
assert.ok(day1);
|
||||
assert.equal(day1.request_count, 0);
|
||||
assert.deepEqual(day1.by_provider, {});
|
||||
assert.equal(day1.median_latency_ms, 0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('23f — cacheHitRateWindow', () => {
|
||||
let TMP;
|
||||
before(() => {
|
||||
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23f-'));
|
||||
const now = Date.now();
|
||||
const t = (offsetMs) => new Date(now - offsetMs).toISOString();
|
||||
setupAuditFiles(TMP, {
|
||||
'live': [
|
||||
makeEvent({ ts: t(5000), provider: 'anthropic', cache_status: 'hit' }),
|
||||
makeEvent({ ts: t(4000), provider: 'anthropic', cache_status: 'hit' }),
|
||||
makeEvent({ ts: t(3000), provider: 'anthropic', cache_status: 'miss' }),
|
||||
makeEvent({ ts: t(2000), provider: 'openai', cache_status: 'miss' }),
|
||||
makeEvent({ ts: t(1500), provider: 'openai', cache_status: 'bypass' }),
|
||||
// events with cache_status null (e.g., 401 paths) excluded
|
||||
makeEvent({ ts: t(1000), provider: null, cache_status: null }),
|
||||
],
|
||||
});
|
||||
});
|
||||
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
||||
|
||||
it('23f-1: hit_rate excludes bypass from denominator', () => {
|
||||
const r = cacheHitRateWindow({ windowMs: 86400 * 1000, olpHome: TMP });
|
||||
// overall: hit=2, miss=2, bypass=1 → hit_rate = 2/(2+2) = 0.5
|
||||
assert.equal(r.hit, 2);
|
||||
assert.equal(r.miss, 2);
|
||||
assert.equal(r.bypass, 1);
|
||||
assert.equal(r.hit_rate, 0.5);
|
||||
});
|
||||
|
||||
it('23f-2: per-provider hit_rate', () => {
|
||||
const r = cacheHitRateWindow({ windowMs: 86400 * 1000, olpHome: TMP });
|
||||
// anthropic: 2 hit + 1 miss → rate 2/3
|
||||
assert.equal(r.by_provider.anthropic.hit, 2);
|
||||
assert.equal(r.by_provider.anthropic.miss, 1);
|
||||
assert.equal(r.by_provider.anthropic.hit_rate, 2 / 3);
|
||||
// openai: 1 miss + 1 bypass → rate 0/1 = 0
|
||||
assert.equal(r.by_provider.openai.miss, 1);
|
||||
assert.equal(r.by_provider.openai.bypass, 1);
|
||||
assert.equal(r.by_provider.openai.hit_rate, 0);
|
||||
});
|
||||
|
||||
it('23f-3: events with cache_status null excluded from total', () => {
|
||||
const r = cacheHitRateWindow({ windowMs: 86400 * 1000, olpHome: TMP });
|
||||
assert.equal(r.total, 5); // 6 events but 1 has cache_status:null
|
||||
});
|
||||
});
|
||||
|
||||
describe('23g — PII guard (ADR 0008 § 4.3)', () => {
|
||||
let TMP;
|
||||
before(() => {
|
||||
TMP = _mkdtempSyncForSetup(_pathJoinForSetup(_tmpdirForSetup(), 'olp-test-23g-'));
|
||||
setupAuditFiles(TMP, {
|
||||
'live': [makeEvent({ provider: 'anthropic', latency_ms: 100 })],
|
||||
});
|
||||
});
|
||||
after(() => { rmSync(TMP, { recursive: true, force: true }); });
|
||||
|
||||
it('23g-1: aggregateRequests returns NO message-content fields', () => {
|
||||
const r = aggregateRequests({ windowMs: 86400 * 1000, olpHome: TMP });
|
||||
const json = JSON.stringify(r);
|
||||
for (const piiKey of ['content', 'message', 'messages', 'prompt', 'response', 'body']) {
|
||||
assert.ok(!json.includes(`"${piiKey}"`),
|
||||
`aggregateRequests result MUST NOT contain field "${piiKey}" (PII guard)`);
|
||||
}
|
||||
});
|
||||
|
||||
it('23g-2: spendTrendDaily returns NO message-content fields', () => {
|
||||
const r = spendTrendDaily({ days: 1, olpHome: TMP });
|
||||
const json = JSON.stringify(r);
|
||||
for (const piiKey of ['content', 'message', 'messages', 'prompt', 'response', 'body']) {
|
||||
assert.ok(!json.includes(`"${piiKey}"`));
|
||||
}
|
||||
});
|
||||
|
||||
it('23g-3: topFallbackChains + cacheHitRateWindow contain no message fields', () => {
|
||||
const j1 = JSON.stringify(topFallbackChains({ windowMs: 86400 * 1000, olpHome: TMP }));
|
||||
const j2 = JSON.stringify(cacheHitRateWindow({ windowMs: 86400 * 1000, olpHome: TMP }));
|
||||
for (const piiKey of ['content', 'message', 'messages', 'prompt', 'response', 'body']) {
|
||||
assert.ok(!j1.includes(`"${piiKey}"`));
|
||||
assert.ok(!j2.includes(`"${piiKey}"`));
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user