feat+test+docs: D49 — lib/audit-query.mjs (Phase 3 audit aggregate query layer) (#26)

Second Phase 3 D-day. Implements ADR 0008 § 4 query API. Pure in-memory
ndjson scan; cross-file walk over audit.ndjson (live) +
audit-YYYY-MM-DD.ndjson (rotated). No server.mjs integration in this
D-day (D50 wires the consuming endpoints).

NEW lib/audit-query.mjs (~370 lines): 5 public API functions per
ADR 0008 § 4.1:

  - discoverAuditFiles({ olpHome }): filesystem scan; returns
    Map<date|'live', path>.
  - readAuditWindow({ startMs, endMs, olpHome, logEvent }): generator
    over events in half-open window [startMs, endMs). Walks rotated
    date files + live file. Skips malformed lines + logs warn.
  - aggregateRequests({ windowMs, olpHome }): counts + status buckets
    + by_provider + by_owner_tier + by_path + median/p95 latency over
    rolling window.
  - topFallbackChains({ windowMs, limit, olpHome }): top-N chains by
    trigger count from events with fallback_hops > 0. Tied-count
    tiebreak: ascending first_seen.
  - spendTrendDaily({ days, olpHome }): daily series ending today
    with sparse-fill for zero-request days. Per-day request_count +
    median latency + by_provider breakdown.
  - cacheHitRateWindow({ windowMs, olpHome }): audit-derived cache
    hit rate (bypass excluded from denominator); per-provider + overall.

PII discipline (ADR 0008 § 4.3): every aggregate function relays only
schema fields; never message content. Suite 23g actively asserts the
absence of content/message/messages/prompt/response/body keys in every
aggregate output.

Cross-file walk semantics (ADR 0008 § 4.2): half-open window
[startMs, endMs); date-range computed once from window bounds; each
rotated date file checked; live audit.ndjson always checked (it
covers today regardless of whether the window endpoint is past
midnight).

spendTrendDaily calendar-date semantics:
  days: N returns "last N calendar UTC dates ending today" — NOT
  "events within a rolling N*86400-ms window" (which would span N+1
  distinct UTC dates and produce off-by-one buckets at non-midnight
  call times). Computed via:
    for (let i = days-1; i >= 0; i--)
      dates.push(_utcDateFromMs(now - i*86400*1000));

cacheHitRateWindow denominator: hit_rate = hit / (hit + miss).
Bypass is intentional non-cacheable (Anthropic cache_control marker),
NOT a cache miss; excluding it from the denominator gives a clean
cache-effectiveness signal.

TESTS — Suite 23, +27 (544 → 571):

  23a-1..4: discoverAuditFiles (empty dir / live only / live+rotated /
    non-audit files ignored)
  23b-1..6: readAuditWindow (all-coverage / single-day / half-open
    exclusivity / empty window / missing files / malformed-skip with
    warn)
  23c-1..4: aggregateRequests (counts + status buckets + by_provider;
    by_owner_tier; median+p95 latency over realistic distribution;
    invalid windowMs rejection)
  23d-1..4: topFallbackChains (sort desc by count; limit truncation;
    fallback_hops=0 excluded; first_seen/last_seen carried)
  23e-1..3: spendTrendDaily (N-day range correctness — caught off-by-
    one during local run; populated day breakdown; empty day sparse-
    fill)
  23f-1..3: cacheHitRateWindow (overall + per-provider hit_rate;
    bypass not in denominator; cache_status=null events excluded)
  23g-1..3: PII guard for aggregateRequests / spendTrendDaily /
    topFallbackChains + cacheHitRateWindow — every output JSON-
    stringified + scanned for forbidden PII keys

DOCUMENTATION:

  - AGENTS.md: lib/audit-query.mjs new entry; lib/audit.mjs note added
    that D52 extends with daily rotation.

NOT IN D49 scope:

  - server.mjs endpoints consuming these queries (D50)
  - dashboard.html (D51)
  - lib/audit.mjs rotation extension + bin/olp-audit-rotate.mjs (D52)
  - tried_providers schema fix (D53; D45 P2 deferral)
  - Phase 3 close → v0.3.0 (D55; maintainer-triggered)

Test count: 544 → 571 (+27). Verified locally via npm test.

AUTHORITY:

  - ADR 0008 § 4 (query API surface) + § 5 (rotation file naming
    pattern) + § 3 (storage layout).
  - ADR 0007 § 8 (audit ndjson event schema — input data).
  - CLAUDE.md release_kit overlay phase_rolling_mode — under
    Unreleased.
  - Standing autopilot grant.

ALIGNMENT.md scope check: this PR adds a new lib/ module. No provider
plugin / entry surface / IR change. Rule 5 commit-citation requirements
for those scopes do not apply.

Co-authored-by: dtzp555 <dtzp555@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
dtzp555-max
2026-05-25 15:51:57 +10:00
committed by GitHub
co-authored by taodeng Claude Opus 4.7
parent c0b696984f
commit 686794e316
4 changed files with 965 additions and 1 deletions
+489
View File
@@ -0,0 +1,489 @@
/**
* lib/audit-query.mjs — OLP audit ndjson aggregate query layer (Phase 3 / D49)
*
* Authority: ADR 0008 § 4 (query API surface) + § 5 (rotation file naming) +
* § 3 (storage layout). Reads `~/.olp/logs/audit.ndjson` (live) +
* `audit-YYYY-MM-DD.ndjson` (rotated dailies) and returns aggregate
* summaries shaped for the Dashboard endpoints (D50).
*
* Query model (ADR 0008 Lane 2 = A): in-memory scan per request. O(N) where
* N = total lines in the date range. Family-scale acceptable; SQLite hybrid
* (ADR 0007 § 13) is the documented forward path when N+queries get slow.
*
* PII discipline (ADR 0007 § 8 + ADR 0008 § 4.3): event shape is hash + shape
* only — no message content, no response content, no raw tokens. This module
* MUST NOT introduce derived fields that reveal content. Every aggregate
* function asserts the input event has the expected shape but does NOT inspect
* or relay message bodies.
*
* What is NOT in this module (intentional split):
* - Daily rotation trigger (D52, lib/audit.mjs extension)
* - Server endpoints that consume these queries (D50, server.mjs)
* - Dashboard HTML / DOM render (D51, dashboard.html)
*/
import { readFileSync, readdirSync, existsSync } from 'node:fs';
import { join } from 'node:path';
import { homedir } from 'node:os';
// ── Constants ─────────────────────────────────────────────────────────────
const DEFAULT_OLP_HOME = join(homedir(), '.olp');
const OLP_HOME_ENV = 'OLP_HOME';
const LIVE_AUDIT_FILE = 'audit.ndjson';
const ROTATED_FILE_PATTERN = /^audit-(\d{4}-\d{2}-\d{2})\.ndjson$/;
// ── Path helpers ──────────────────────────────────────────────────────────
function _resolveOlpHome(opts) {
if (opts?.olpHome) return opts.olpHome;
if (process.env[OLP_HOME_ENV]) return process.env[OLP_HOME_ENV];
return DEFAULT_OLP_HOME;
}
function _logsDir(opts) {
return join(_resolveOlpHome(opts), 'logs');
}
/**
* Returns the UTC-date string (YYYY-MM-DD) for an ISO-8601 timestamp.
*/
function _utcDateString(isoTs) {
if (typeof isoTs !== 'string' || isoTs.length < 10) return null;
return isoTs.slice(0, 10);
}
/**
* Returns the UTC-date string for an epoch-ms.
*/
function _utcDateFromMs(ms) {
return new Date(ms).toISOString().slice(0, 10);
}
/**
* Inclusive range of UTC date strings from startDate to endDate (both
* YYYY-MM-DD). Returns the list in ascending order. Safe for spans up to
* several years (no upper bound enforced — caller's responsibility).
*/
function _dateRange(startDate, endDate) {
const dates = [];
const cur = new Date(`${startDate}T00:00:00Z`);
const end = new Date(`${endDate}T00:00:00Z`);
while (cur <= end) {
dates.push(cur.toISOString().slice(0, 10));
cur.setUTCDate(cur.getUTCDate() + 1);
}
return dates;
}
// ── File enumeration ──────────────────────────────────────────────────────
/**
* Discover audit files in the logs directory. Returns a Map from
* date-string ('YYYY-MM-DD' or 'live' for the un-rotated file) to absolute
* file path. The 'live' entry is `audit.ndjson` if present; date-string
* entries are the rotated daily files matching `audit-YYYY-MM-DD.ndjson`.
*
* Returns an empty Map if the logs directory does not exist or is empty.
* Caller responsible for date filtering.
*
* @param {object} [opts] - { olpHome }
* @returns {Map<string, string>} date-string → absolute file path
*/
export function discoverAuditFiles(opts = {}) {
const dir = _logsDir(opts);
const out = new Map();
if (!existsSync(dir)) return out;
let entries;
try { entries = readdirSync(dir); } catch { return out; }
for (const name of entries) {
if (name === LIVE_AUDIT_FILE) {
out.set('live', join(dir, name));
continue;
}
const m = ROTATED_FILE_PATTERN.exec(name);
if (m) {
out.set(m[1], join(dir, name));
}
}
return out;
}
// ── Line-level read + parse ───────────────────────────────────────────────
/**
* Parse a single ndjson line. Returns the event object on success, or
* null on parse error. Caller logs warn for null returns.
*/
function _parseLine(line) {
if (!line) return null;
try {
const obj = JSON.parse(line);
if (typeof obj !== 'object' || obj === null) return null;
return obj;
} catch {
return null;
}
}
/**
* Read all events from a single file, skipping malformed lines.
* Logs warn (via logEvent override or console) for each malformed line so
* a corrupted day doesn't kill the query.
*
* @param {string} path
* @param {(level: string, event: string, data?: object) => void} [logEvent]
* @returns {Array<object>} parsed events
*/
function _readFileEvents(path, logEvent) {
let raw;
try {
raw = readFileSync(path, 'utf-8');
} catch (err) {
// Re-throw read errors (EACCES, ENOENT during race) so the dashboard
// endpoint surfaces 500 with diagnostic per ADR 0008 § 4.4.
throw new Error(`audit_query_read_failed: ${path}: ${err?.message ?? err}`);
}
const lines = raw.split('\n');
const events = [];
let skipped = 0;
for (let i = 0; i < lines.length; i++) {
const line = lines[i].trim();
if (!line) continue;
const ev = _parseLine(line);
if (ev === null) {
skipped++;
continue;
}
events.push(ev);
}
if (skipped > 0 && logEvent) {
logEvent('warn', 'audit_query_skip_malformed', { path, skipped });
}
return events;
}
// ── Public API ────────────────────────────────────────────────────────────
/**
* Iterate all audit events in [startMs, endMs). Walks the rotated daily
* files whose date overlaps the range + today's live audit.ndjson. Within
* each file, includes only events whose `ts` falls in the window.
*
* Per ADR 0008 § 4.2: window semantics are half-open [start, end).
*
* @param {object} args
* @param {number} args.startMs - epoch-ms inclusive lower bound
* @param {number} args.endMs - epoch-ms exclusive upper bound
* @param {string} [args.olpHome]
* @param {(level: string, event: string, data?: object) => void} [args.logEvent]
* @yields {object} parsed audit event
*/
export function* readAuditWindow({ startMs, endMs, olpHome, logEvent } = {}) {
if (typeof startMs !== 'number' || typeof endMs !== 'number') {
throw new Error('readAuditWindow: startMs and endMs (numbers) are required');
}
if (endMs <= startMs) return; // empty window
const files = discoverAuditFiles({ olpHome });
if (files.size === 0) return;
// Walk all dates in [startMs, endMs) plus the live file (today).
const startDate = _utcDateFromMs(startMs);
const endDate = _utcDateFromMs(endMs - 1); // endMs is exclusive
const dateList = _dateRange(startDate, endDate);
for (const date of dateList) {
const path = files.get(date);
if (!path) continue;
const events = _readFileEvents(path, logEvent);
for (const ev of events) {
const tsStr = ev.ts;
if (typeof tsStr !== 'string') continue;
const tsMs = Date.parse(tsStr);
if (Number.isNaN(tsMs)) continue;
if (tsMs >= startMs && tsMs < endMs) yield ev;
}
}
// Live file (today) — always check; date may overlap window's end.
const livePath = files.get('live');
if (livePath) {
const events = _readFileEvents(livePath, logEvent);
for (const ev of events) {
const tsStr = ev.ts;
if (typeof tsStr !== 'string') continue;
const tsMs = Date.parse(tsStr);
if (Number.isNaN(tsMs)) continue;
if (tsMs >= startMs && tsMs < endMs) yield ev;
}
}
}
/**
* Aggregate request shape over a rolling window ending at "now".
*
* Returns:
* {
* window: { startMs, endMs },
* request_count, status_2xx, status_4xx, status_5xx,
* by_provider: { [providerKey]: { count, cache_hit, cache_miss, cache_bypass, fallback_count } },
* by_owner_tier: { owner: N, guest: N, anonymous: N },
* by_path: { '/v1/chat/completions': N, '/v1/models': N, ... },
* median_latency_ms, p95_latency_ms,
* }
*
* Per ADR 0008 § 4.1 + § 4.3 PII discipline: aggregates count + categorical
* breakdowns only, NEVER message content.
*
* @param {object} args
* @param {number} args.windowMs - duration in ms; window = [now - windowMs, now)
* @param {string} [args.olpHome]
* @param {(level, event, data?) => void} [args.logEvent]
* @param {() => number} [args._nowFn] - injectable for testing
*/
export function aggregateRequests({ windowMs, olpHome, logEvent, _nowFn } = {}) {
if (typeof windowMs !== 'number' || windowMs <= 0) {
throw new Error('aggregateRequests: windowMs (positive number) is required');
}
const now = (_nowFn ?? Date.now)();
const startMs = now - windowMs;
const endMs = now;
const result = {
window: { startMs, endMs },
request_count: 0,
status_2xx: 0,
status_4xx: 0,
status_5xx: 0,
by_provider: {},
by_owner_tier: { owner: 0, guest: 0, anonymous: 0 },
by_path: {},
median_latency_ms: 0,
p95_latency_ms: 0,
};
const latencies = [];
for (const ev of readAuditWindow({ startMs, endMs, olpHome, logEvent })) {
result.request_count++;
// Status code bucket
const sc = typeof ev.status_code === 'number' ? ev.status_code : 0;
if (sc >= 200 && sc < 300) result.status_2xx++;
else if (sc >= 400 && sc < 500) result.status_4xx++;
else if (sc >= 500) result.status_5xx++;
// By provider
if (typeof ev.provider === 'string' && ev.provider.length > 0) {
const p = result.by_provider[ev.provider] ??= {
count: 0, cache_hit: 0, cache_miss: 0, cache_bypass: 0, fallback_count: 0,
};
p.count++;
if (ev.cache_status === 'hit') p.cache_hit++;
else if (ev.cache_status === 'miss') p.cache_miss++;
else if (ev.cache_status === 'bypass') p.cache_bypass++;
if (typeof ev.fallback_hops === 'number' && ev.fallback_hops > 0) p.fallback_count++;
}
// By owner tier
if (ev.owner_tier === 'owner') result.by_owner_tier.owner++;
else if (ev.owner_tier === 'guest') result.by_owner_tier.guest++;
else result.by_owner_tier.anonymous++;
// By path
if (typeof ev.path === 'string' && ev.path.length > 0) {
result.by_path[ev.path] = (result.by_path[ev.path] ?? 0) + 1;
}
// Latency
if (typeof ev.latency_ms === 'number' && ev.latency_ms >= 0) {
latencies.push(ev.latency_ms);
}
}
// Median + p95 over sorted latencies
if (latencies.length > 0) {
latencies.sort((a, b) => a - b);
const midIdx = Math.floor(latencies.length / 2);
result.median_latency_ms = latencies.length % 2 === 0
? Math.round((latencies[midIdx - 1] + latencies[midIdx]) / 2)
: latencies[midIdx];
const p95Idx = Math.min(latencies.length - 1, Math.floor(latencies.length * 0.95));
result.p95_latency_ms = latencies[p95Idx];
}
return result;
}
/**
* Top-N fallback chains by trigger count in window. A "chain" is the
* `tried_providers` array from an event with fallback_hops > 0. Returns
* sorted array descending by count; ties broken by earliest first_seen.
*
* [{ chain: ['anthropic', 'openai'], count: 42, first_seen, last_seen }, ...]
*
* @param {object} args
* @param {number} args.windowMs
* @param {number} [args.limit=10]
* @param {string} [args.olpHome]
* @param {(level, event, data?) => void} [args.logEvent]
* @param {() => number} [args._nowFn]
*/
export function topFallbackChains({ windowMs, limit = 10, olpHome, logEvent, _nowFn } = {}) {
if (typeof windowMs !== 'number' || windowMs <= 0) {
throw new Error('topFallbackChains: windowMs (positive number) is required');
}
const now = (_nowFn ?? Date.now)();
const startMs = now - windowMs;
const endMs = now;
// Map chain-key (joined string) → aggregate
const chains = new Map();
for (const ev of readAuditWindow({ startMs, endMs, olpHome, logEvent })) {
if (typeof ev.fallback_hops !== 'number' || ev.fallback_hops <= 0) continue;
if (!Array.isArray(ev.tried_providers) || ev.tried_providers.length < 2) continue;
const key = ev.tried_providers.join('→');
const entry = chains.get(key);
const ts = typeof ev.ts === 'string' ? ev.ts : null;
if (entry === undefined) {
chains.set(key, {
chain: [...ev.tried_providers],
count: 1,
first_seen: ts,
last_seen: ts,
});
} else {
entry.count++;
if (ts && (!entry.first_seen || ts < entry.first_seen)) entry.first_seen = ts;
if (ts && (!entry.last_seen || ts > entry.last_seen)) entry.last_seen = ts;
}
}
// Sort desc by count, ascending by first_seen on ties
const arr = [...chains.values()];
arr.sort((a, b) => {
if (b.count !== a.count) return b.count - a.count;
if (a.first_seen && b.first_seen) return a.first_seen < b.first_seen ? -1 : a.first_seen > b.first_seen ? 1 : 0;
return 0;
});
return arr.slice(0, limit);
}
/**
* Daily series of request_count + median latency_ms + by_provider over N
* UTC days ending today. Sparse-fills zero-request days. Returns ascending
* by date:
*
* [{ date: '2026-05-22', request_count, median_latency_ms, by_provider }, ...]
*
* by_provider is { [providerKey]: count } per day.
*
* @param {object} args
* @param {number} args.days
* @param {string} [args.olpHome]
* @param {(level, event, data?) => void} [args.logEvent]
* @param {() => number} [args._nowFn]
*/
export function spendTrendDaily({ days, olpHome, logEvent, _nowFn } = {}) {
if (typeof days !== 'number' || days <= 0) {
throw new Error('spendTrendDaily: days (positive number) is required');
}
const now = (_nowFn ?? Date.now)();
// Compute the N UTC dates ending today (inclusive). Semantics: "last N
// calendar dates ending today" — NOT "events within a rolling N*86400-ms
// window ago" (the latter would span N+1 distinct UTC dates and produce
// off-by-one buckets at non-midnight call times).
const dates = [];
for (let i = days - 1; i >= 0; i--) {
dates.push(_utcDateFromMs(now - i * 86400 * 1000));
}
// Window covers the start of the first date through "now" so readAuditWindow
// sees every event whose ts falls in any of the N dates' UTC days.
const startMs = Date.parse(`${dates[0]}T00:00:00Z`);
const endMs = now;
// Bucket by UTC date
const buckets = new Map();
for (const ev of readAuditWindow({ startMs, endMs, olpHome, logEvent })) {
const date = _utcDateString(ev.ts);
if (!date) continue;
const b = buckets.get(date) ?? { request_count: 0, latencies: [], by_provider: {} };
b.request_count++;
if (typeof ev.latency_ms === 'number') b.latencies.push(ev.latency_ms);
if (typeof ev.provider === 'string' && ev.provider.length > 0) {
b.by_provider[ev.provider] = (b.by_provider[ev.provider] ?? 0) + 1;
}
buckets.set(date, b);
}
// Sparse-fill using the precomputed dates list (preserves ascending order)
return dates.map(date => {
const b = buckets.get(date);
if (b) {
b.latencies.sort((a, b) => a - b);
const midIdx = Math.floor(b.latencies.length / 2);
const median = b.latencies.length === 0 ? 0
: b.latencies.length % 2 === 0
? Math.round((b.latencies[midIdx - 1] + b.latencies[midIdx]) / 2)
: b.latencies[midIdx];
return {
date,
request_count: b.request_count,
median_latency_ms: median,
by_provider: b.by_provider,
};
}
return { date, request_count: 0, median_latency_ms: 0, by_provider: {} };
});
}
/**
* Audit-derived cache hit rate over the window. Differs from
* `cacheStore.stats()` in server.mjs: that is the live in-process counter;
* this is the audit-side rate scoped to the rolling window.
*
* { window: { startMs, endMs }, total, hit, miss, bypass, hit_rate, by_provider }
*
* @param {object} args
* @param {number} args.windowMs
* @param {string} [args.olpHome]
* @param {(level, event, data?) => void} [args.logEvent]
* @param {() => number} [args._nowFn]
*/
export function cacheHitRateWindow({ windowMs, olpHome, logEvent, _nowFn } = {}) {
if (typeof windowMs !== 'number' || windowMs <= 0) {
throw new Error('cacheHitRateWindow: windowMs (positive number) is required');
}
const now = (_nowFn ?? Date.now)();
const startMs = now - windowMs;
const endMs = now;
let total = 0, hit = 0, miss = 0, bypass = 0;
const by_provider = {};
for (const ev of readAuditWindow({ startMs, endMs, olpHome, logEvent })) {
if (ev.cache_status === null || ev.cache_status === undefined) continue;
total++;
const p = typeof ev.provider === 'string' && ev.provider.length > 0 ? ev.provider : '__unknown__';
const pe = by_provider[p] ??= { total: 0, hit: 0, miss: 0, bypass: 0, hit_rate: 0 };
pe.total++;
if (ev.cache_status === 'hit') { hit++; pe.hit++; }
else if (ev.cache_status === 'miss') { miss++; pe.miss++; }
else if (ev.cache_status === 'bypass') { bypass++; pe.bypass++; }
}
// Compute hit_rate per provider + overall (excludes bypass from denominator
// since bypass-by-cache_control is intentional non-cacheable, not a cache miss).
for (const p of Object.values(by_provider)) {
const denom = p.hit + p.miss;
p.hit_rate = denom > 0 ? p.hit / denom : 0;
}
const overallDenom = hit + miss;
const hit_rate = overallDenom > 0 ? hit / overallDenom : 0;
return {
window: { startMs, endMs },
total, hit, miss, bypass, hit_rate, by_provider,
};
}