mirror of
https://github.com/dtzp555-max/olp.git
synced 2026-07-19 09:45:07 +00:00
release(v0.5.1): hotfix — quota probe cache/backoff/schema-drift correctness (codex review) (#58)
* release(v0.5.1): hotfix — quota probe cache/backoff/schema-drift correctness (codex review) Addresses three production-quality findings from codex's post-v0.5.0 review (codex review on PR #57, reproduced with local mocks): F1 [P1] — Doctor bypass of cache + backoff (ADR 0013 Rule 3) `anthropic.quota_probe_reachable` called `_probeOnce(auth)` directly, bypassing `quotaProbeState.backoffUntil`. Successive `olp doctor` invocations within a backoff window each hit upstream — violating ADR 0013 Rule 3 (backoff is mandatory for ALL consumers). Fix: doctor now routes through `quotaStatus()`. ADR 0013 Rule 3 clarified: "All consumers of `quotaStatus()`, including `olp doctor` checks, MUST route through `quotaStatus()` and MUST NOT call `_probeOnce()` directly." F2 [P2] — 200 with empty ratelimit headers cached as live data (ADR 0013 Rule 5) A 200 OK with zero `anthropic-ratelimit-*` headers was cached as `stale: false` (live). Minimum-viable-schema gate added to `_probeOnce`: requires 5h-utilization + 5h-reset + 7d-utilization + 7d-reset present; absence → `failureKind: 'schema_drift'`, backoff scheduled, result NOT cached. ADR 0013 Rule 5 updated with the gate specification. F3 [P2] — Dashboard-data loses failure detail (ADR 0013 Rule 6) `aggregateProviderQuota()` collapsed all failure modes into `status: 'unavailable'` ("no public quota api or probe disabled") — same as providers with no API at all. Fix: `quotaStatus()` v0.5.1 contract — `null` ONLY for opt-in-off; failures return `{ probe_status: 'unreachable', failure: { kind, message, backoff_until? } }`. New `failure_kind` enum: no_credentials | auth_failed | rate_limited | schema_drift | network | other. Dashboard renders `unreachable` with red border + failure detail. Authority: ADR 0013 Rules 3, 5, 6 (cache + backoff + schema-drift + failure transparency) ADR 0008 Amendment 2 (richer quota_v2 shape; new unreachable status) ADR 0002 Amendment 8 unchanged (constitutional permission for the probe) Codex review findings F1–F3 (codex on PR #57) Changes: - lib/providers/anthropic.mjs: quotaProbeState gains lastError + failureKind; _probeOnce: min-field gate + failureKind population; quotaStatus(): v0.5.1 contract (null=disabled only; probe_status:live/stale/unreachable); doctorChecks routes through quotaStatus(); reset functions updated - lib/audit-query.mjs: _normalizeAnthropicQuota handles probe_status field; aggregateProviderQuota emits failure/failure_kind/backoff_until; unreachable status - dashboard.html: unreachable CSS classes + render path + footer v0.5.1 - test-features.mjs: 38f/j/l updated for v0.5.1 shape; 38r refactored for F1; 38g/k gain probe_status assertions; 38u/v/w new regression tests; 756→759 tests - docs/adr/0008: Amendment 2 (richer ProviderQuotaEntry + quotaStatus contract) - docs/adr/0013: Rule 3 clarification (doctor must use quotaStatus); Rule 5 min-viable-schema gate specification - package.json: 0.5.0 → 0.5.1 - CHANGELOG.md: v0.5.1 hotfix entry promoted from Unreleased - README.md / AGENTS.md: Phase 5 closed at v0.5.1; Phase 6 next Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs+code: PR #58 fold-in — reviewer Nits #1 + #2 (ADR doc-drift + opt_in_off enum) Fresh-context reviewer (PR #58) verdict: APPROVE_WITH_MINOR, 0 blocking, 4 nits. Folding in #1 + #2 (both 1-line cosmetic-but-correct fixes). Deferring #3 (test-only edge case in module-level seam restore) and #4 (pre-existing codex F4 — bin/olp.mjs + olp-plugin still read legacy quota shape; separate PR planned). Nit #1 — ADR 0002 Amendment 8 documentation drift. Amendment 8 at v0.5.0 line 20 said "the function returns `null` rather than throwing", and line 33 said "stale-cache-on-failure (`null` is returned only when no cache entry exists; if a stale entry exists it's returned with a `stale: true` marker)". v0.5.1 refined this contract: `null` is now reserved STRICTLY for opt-in-off, and all failure modes return `{ probe_status: 'unreachable' | 'stale', failure: {...} }`. The substantive idempotent-failure constraint (no throw to caller) is unchanged. The operational description in Amendment 8 was stale — fixed to cross-reference ADR 0008 Amendment 2 + ADR 0013 Rule 6 for the v0.5.1 contract refinement. Also references Suite 38 (38u/38v/38w) as the regression coverage producing the new shape. Nit #2 — `failureKind: 'opt_in_off'` declared but never produced. The enum value was listed in both the code comment (anthropic.mjs:250) and ADR 0008 Amendment 2 (line 30) but never actually assigned — because when opt-in is off, `quotaStatus()` returns the literal `null` BEFORE any state mutation happens. The enum value was dead. Fix: removed `opt_in_off` from both enum declarations + added an inline note explaining that consumers (audit-query, doctor) distinguish opt-in-off by checking `quotaStatus() === null`, not via failureKind. Deferred: - Nit #3 (test-seam restore edge case): if a caller pre-sets `_quotaAuthReadFnForTest` AND passes a non-default `_authReadFn`, the finally block restores to null clobbering pre-set value. Test-only impact, no production risk. Pure hygiene; defer. - Nit #4 (codex F4): bin/olp.mjs cmdUsage and olp-plugin/index.js still read legacy `body.quota` field, never consume `quota_v2`. Reviewer confirmed neither crashes — both gracefully fall through to "no quota api" branch. Out of scope for this hotfix per the hotfix dispatch contract; separate PR will migrate them. Tests: 759/759 still pass post-fold-in. No test changes needed. Authority: PR #58 review thread + ADR 0013 Rule 6 (failure transparency) + ADR 0008 Amendment 2 (ProviderQuotaEntry v0.5.1 shape). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: dtzp555 <dtzp555@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
dtzp555-max
@@ -49,7 +49,7 @@ Runtime: Node.js (ESM, `.mjs` throughout). No build step. No bundler. `server.mj
|
||||
- `.github/workflows/alignment.yml` — CI blacklist grep + per-provider citation soft check; fails the build on known-hallucinated tokens.
|
||||
- `CLAUDE.md` — Claude-Code-specific session instructions + `release_kit` overlay (Iron Rule 5.5).
|
||||
|
||||
**Implementation status note (as of 2026-05-25):** Files marked 📋 above are designed and documented but not yet on disk; files marked 🟡 are partially shipped; files marked ✅ are Phase 2 deliverables. The shipped set as of D47 is: `server.mjs` (with Phase 2 auth middleware + audit wire + owner-vs-non-owner gating), `lib/ir/`, `lib/providers/{anthropic,codex,mistral}.mjs`, `lib/cache/{keys,store}.mjs`, `lib/fallback/engine.mjs`, `lib/keys.mjs` (core + loadAuthConfigSync — D44 + D45), `lib/audit.mjs` (D45), `bin/olp-keys.mjs` (D47), `models-registry.json`, `test-features.mjs` (Suites 19–22). Phase 2 functional scope is complete; remaining is Phase 2 close → v0.2.0 (maintainer-triggered, explicit per CLAUDE.md `release_kit.phase_close_trigger`).
|
||||
**Implementation status note (as of 2026-05-27):** Phase 5 (Quota Probes + Dashboard Enrichment) is closed at v0.5.0 + v0.5.1 hotfix. The shipped set includes all Phases 1–5 deliverables. v0.5.1 hotfix (2026-05-27) fixes three codex review findings: F1 (doctor check bypassed backoff by calling `_probeOnce` directly — now routes through `quotaStatus()`), F2 (200 with empty `anthropic-ratelimit-*` headers was cached as live — minimum-viable-schema gate added), F3 (null collapsed all failure modes — `probe_status:'unreachable'` shape + `failure`/`failure_kind`/`backoff_until` fields added). See ADR 0008 Amendment 2 + ADR 0013 Rule 3/5 clarifications. Phase 6 is next (per CLAUDE.md `release_kit.current_phase`).
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -6,6 +6,46 @@ All notable changes to OLP land here. Per `CLAUDE.md` release_kit overlay, this
|
||||
|
||||
(empty — Phase 6 entries land here once Phase 6 opens)
|
||||
|
||||
## v0.5.1 — 2026-05-27
|
||||
|
||||
**Hotfix — Quota probe cache/backoff/schema-drift correctness (codex review findings F1–F3).** Three production-quality bugs in the v0.5.0 quota probe, reproduced by codex with local mocks, are corrected. 756 → 759 tests (3 new regression tests); 4 existing test assertions updated to reflect the v0.5.1 return-shape contract.
|
||||
|
||||
### Fixes
|
||||
|
||||
- **F1 [P1] — Doctor bypass of cache + backoff (ADR 0013 Rule 3).** `anthropic.quota_probe_reachable` doctor check called `_probeOnce(auth)` directly, bypassing the module-level `quotaProbeState.backoffUntil` check. Successive `olp doctor` invocations within a backoff window each hit upstream — violating ADR 0013 Rule 3 (60s-3600s exponential backoff is mandatory for all consumers). **Fix:** doctor check now routes through `quotaStatus()`, which enforces cache + backoff. ADR 0013 Rule 3 clarification added: "All consumers of `quotaStatus()`, including `olp doctor` checks, MUST route through `quotaStatus()` and MUST NOT call `_probeOnce()` directly."
|
||||
|
||||
- **F2 [P2] — 200 with empty `anthropic-ratelimit-*` headers cached as live data (ADR 0013 Rule 5).** `_probeOnce` treated any 200 OK (regardless of header content) as a successful probe, caching it with `stale: false` even when zero `anthropic-ratelimit-*` headers were present. A proxy stripping headers, a schema change, or a mock returning `{}` would silently appear as "LIVE" on the dashboard with all bars empty. **Fix:** minimum-viable-schema gate requires these 4 fields non-null: `5h-utilization`, `5h-reset`, `7d-utilization`, `7d-reset`. Any absence → `failureKind: 'schema_drift'`, backoff scheduled, result not cached. ADR 0013 Rule 5 updated with the gate specification.
|
||||
|
||||
- **F3 [P2] — Dashboard-data loses failure detail (ADR 0013 Rule 6).** `aggregateProviderQuota()` collapsed all non-null failure modes (no credentials, auth failure, rate limit, schema drift, network error) into `status: 'unavailable', reason: 'no public quota api or probe disabled'` — the same string as providers with no quota API at all. Operator could not tell what to fix. **Fix:** `quotaStatus()` v0.5.1 return contract: `null` reserved for opt-in-off only; probe failures return `{ probe_status: 'unreachable', failure: { kind, message, backoff_until? } }`. `aggregateProviderQuota()` emits new fields `failure_kind`, `failure`, `backoff_until` per row. `status: 'unreachable'` distinguishes "probe failed" from `status: 'unavailable'` ("no API or disabled"). Dashboard renders `unreachable` with a red border + failure.message + backoff countdown.
|
||||
|
||||
### Backwards-compat notes
|
||||
|
||||
- `quotaStatus()`: `stale: false` → now also includes `probe_status: 'live'` (additive). `stale: true` → now also includes `probe_status: 'stale'` + `failure: {...}` (additive). `null` → NOW RESERVED FOR OPT-IN-OFF ONLY (breaking for callers that relied on `null` to detect "no credentials" or "probe failed" — use `probe_status: 'unreachable'` instead).
|
||||
- `ProviderQuotaEntry.status`: gains `'unreachable'` as a new value (additive). Existing `'live'`, `'stale'`, `'unavailable'` semantics unchanged.
|
||||
- `ProviderQuotaEntry` gains new fields `failure`, `failure_kind`, `backoff_until` (additive, null when not applicable).
|
||||
- `dashboard.html`: handles `unreachable` row (no existing row had this status; additive render path).
|
||||
|
||||
### Test changes
|
||||
|
||||
- 38f, 38j, 38l: updated assertions from `null` to `probe_status: 'unreachable'` (F3 shape change).
|
||||
- 38r: refactored to seed cache + manually expire it + set backoff (F1 — doctor now routes through `quotaStatus()`). Added F1-regression assertion: HTTP call counter stays at 1 after two doctor calls within backoff.
|
||||
- 38g, 38k: added `probe_status` + `failure` assertions (verify new fields present on live/stale shapes).
|
||||
- **38u** (new): F1 regression — successive doctor calls within backoff window → HTTP counter stays at 1.
|
||||
- **38v** (new): F2 regression — 200 + empty ratelimit headers → `probe_status: 'unreachable'` + `failure_kind: 'schema_drift'` + cache stays null.
|
||||
- **38w** (new): F3 regression — `lastError` + `failureKind` propagate through `quotaStatus()` shape for all failure modes (rate_limited / auth_failed / schema_drift / no_credentials).
|
||||
|
||||
### ADR changes
|
||||
|
||||
- **ADR 0013 Rule 3** clarification: doctor checks route through `quotaStatus()`, not `_probeOnce()` directly.
|
||||
- **ADR 0013 Rule 5** update: minimum-viable-schema gate specification (4 required fields; absence = schema_drift signal).
|
||||
- **ADR 0008 Amendment 2**: richer `ProviderQuotaEntry` shape with `failure`/`failure_kind`/`backoff_until`; `probe_status` on `quotaStatus()` return; `unreachable` status semantics; `dashboard.html` unreachable rendering.
|
||||
|
||||
### Authority
|
||||
|
||||
ADR 0013 Rules 3, 5, 6 (cache + backoff + schema-drift + failure transparency); ADR 0008 Amendment 2; ADR 0002 Amendment 8 (unchanged); codex review findings F1–F3 (codex PR review on v0.5.0 close PR #57).
|
||||
|
||||
---
|
||||
|
||||
## v0.5.0 — 2026-05-26
|
||||
|
||||
**Phase 5 — Provider Quota Probes + Dashboard Enrichment.** OLP gains live subscription-quota observability for Anthropic Pro/Max subscribers, surfaced through a Claude.ai-style Plan Usage panel on the owner-only dashboard. The probe is opt-in, READ-ONLY, idempotent on failure, and 5-min-cached with 60s→3600s exponential backoff. Six D-days, seven PRs, zero blocking reviewer findings, no flaky tests; 720 → 756 total tests.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
A personal- and family-scale multi-provider LLM proxy. One HTTP endpoint, many subscriptions behind it, automatic routing + fallback + content-addressed caching. Your IDEs and family clients keep working as long as **any** of your subscriptions has quota left.
|
||||
|
||||
> **Status:** v0.4.3 shipped, 714+ tests. Phase 4 (Operator + Client UX) closed; Phase 5 scope is open. Coming from [OCP](https://github.com/dtzp555-max/ocp)? See [§ Migration from OCP](#migration-from-ocp).
|
||||
> **Status:** v0.5.1 shipped, 759+ tests. Phase 5 (Quota Probes + Dashboard Enrichment) closed; Phase 6 next. Coming from [OCP](https://github.com/dtzp555-max/ocp)? See [§ Migration from OCP](#migration-from-ocp).
|
||||
|
||||
---
|
||||
|
||||
@@ -484,7 +484,7 @@ Use a dedicated bot key — not the maintainer's personal owner key — so revoc
|
||||
|
||||
## Implementation status (as of 2026-05-26, post-v0.4.0)
|
||||
|
||||
Phase 1 closed at v0.1.1 (multi-provider proxy core + pre-Phase-2 cleanup). Phase 2 closed at v0.2.0 (multi-key auth + audit + owner gating + keygen CLI; ADR 0007 § 10 all 11 acceptance criteria shipped). Phase 3 closed at v0.3.0 (Dashboard + `lib/audit-query.mjs` + daily audit rotation; ADR 0008 § 10 all 15 acceptance criteria shipped). Phase 4 closed at v0.4.0 (Operator + Client UX per ADR 0010: SSE heartbeat + `recentErrors[20]` + `/v0/management/status` / `olp` Node CLI + `olp doctor` framework + ADR 0002 Amendment 7 / `olp-connect` bash + `/health.anonymousKey` + ADR 0011 / `olp-plugin/` Telegram-Discord + 6-IDE integration docs). Phase 5 scope is open — candidates per ADR 0010 § Out-of-Phase-4-scope. This table reflects what is currently shipped vs. what is designed for later phases.
|
||||
Phase 1 closed at v0.1.1 (multi-provider proxy core + pre-Phase-2 cleanup). Phase 2 closed at v0.2.0 (multi-key auth + audit + owner gating + keygen CLI; ADR 0007 § 10 all 11 acceptance criteria shipped). Phase 3 closed at v0.3.0 (Dashboard + `lib/audit-query.mjs` + daily audit rotation; ADR 0008 § 10 all 15 acceptance criteria shipped). Phase 4 closed at v0.4.0 (Operator + Client UX per ADR 0010: SSE heartbeat + `recentErrors[20]` + `/v0/management/status` / `olp` Node CLI + `olp doctor` framework + ADR 0002 Amendment 7 / `olp-connect` bash + `/health.anonymousKey` + ADR 0011 / `olp-plugin/` Telegram-Discord + 6-IDE integration docs). Phase 5 closed at v0.5.0 (Quota Probes + Dashboard Enrichment — live Anthropic plan-usage probe + Claude.ai-style dashboard + audit-query aggregateProviderQuota); v0.5.1 hotfix (quota probe cache/backoff/schema-drift correctness — codex review findings F1–F3). Phase 6 is next. This table reflects what is currently shipped vs. what is designed for later phases.
|
||||
|
||||
| File / artifact | Status | Notes |
|
||||
|---|---|---|
|
||||
@@ -573,7 +573,8 @@ The original v0.1 spec (in `~/.cc-rules/memory/projects/olp_v0_1_spec.md` on the
|
||||
- **Phase 1** — Multi-provider proxy core: `server.mjs`, IR, three Tier-D provider plugins (Anthropic / OpenAI Codex / Mistral Vibe), cache (D1+D4) + cleanup (D2 bypass / D3 chunked replay / D23 size cap), fallback engine with first-chunk safety + hard triggers + per-hop log observability, IR↔OpenAI translation under Rule 2(b). ✅ Shipped — v0.1.0 (2026-05-24) + v0.1.1 cleanup (2026-05-25, D35–D42).
|
||||
- **Phase 2** — Multi-key auth (`lib/keys.mjs`) per ADR 0007: opaque OLP API keys, per-key cache namespacing, owner-vs-guest tier for header gating, audit ndjson (`lib/audit.mjs`), `/health` payload trimming + `X-OLP-Fallback-Detail` emission gating, `OLP_OWNER_TOKEN` env override, keygen CLI (`bin/olp-keys.mjs`). ✅ Shipped — v0.2.0 (2026-05-25, D43-A → D47). All 11 ADR 0007 § 10 acceptance criteria covered.
|
||||
- **Phase 3** — Dashboard + audit query layer + daily audit rotation per ADR 0008: in-memory ndjson aggregate query layer (`lib/audit-query.mjs`), 4 owner-only_block management endpoints (`/dashboard` + `/v0/management/dashboard-data` + `/v0/management/quota` + `/cache/stats`), multi-panel `dashboard.html` with 30s poll, synchronous daily audit rotation + `bin/olp-audit-rotate.mjs` cron tool, `tried_providers` schema fix (D45 P2 deferral). ✅ Shipped — v0.3.0 (2026-05-25, D48 → D54). All 15 ADR 0008 § 10 acceptance criteria covered.
|
||||
- **Phase 4 (planned)** — Per-key per-provider auth artifact mapping (ADR 0007 § 12 deferral), audit query rotation/retention policies, SQLite hybrid migration (ADR 0007 § 13 trigger), provider-cost weights for spend trend.
|
||||
- **Phase 5** — Live quota probe (Anthropic Pro/Max OAuth plan-usage via `anthropic-ratelimit-unified-*` headers), Claude.ai-style dashboard enrichment (utilization bars + reset countdowns), audit-query `aggregateProviderQuota()`, per-provider quota_v2 shape in dashboard-data. ✅ Shipped — v0.5.0 (2026-05-27). v0.5.1 hotfix (2026-05-27): quota probe cache/backoff/schema-drift correctness (codex review findings F1–F3).
|
||||
- **Phase 6 (planned)** — Per-key per-provider auth artifact mapping (ADR 0007 § 12 deferral), audit query rotation/retention policies, SQLite hybrid migration (ADR 0007 § 13 trigger), provider-cost weights for spend trend.
|
||||
- **Phase 4+ (v1.x roadmap, triggered as needed)** — Full deferred-work tracker: [`docs/v1x-roadmap.md`](./docs/v1x-roadmap.md). Includes streaming-path singleflight ([issue #16](https://github.com/dtzp555-max/olp/issues/16) + ADR 0005 Amendment 8 design ratified), soft-trigger reactivation (ADR 0004 Amendment 2), `/health` activeSpawns integration, provider-level `cacheKeyFields` mask, streaming-path SPAWN_FAILED salvage.
|
||||
- **Phase N (opt-in)** — Tier-2 / Tier-C provider plugins (Grok / Kimi / MiniMax / GLM / Qwen) per [ADR 0006](./docs/adr/0006-provider-inclusion.md); provider-native protocol endpoints; deterministic triggers. Triggered by tier-2 demand, not on the bootstrap path.
|
||||
|
||||
|
||||
+28
-1
@@ -100,6 +100,7 @@
|
||||
.provider-row:last-child { margin-bottom: 0; }
|
||||
.provider-row.unavailable { background: #f9fafb; }
|
||||
.provider-row.stale { border-color: #fcd34d; }
|
||||
.provider-row.unreachable { border-color: #fca5a5; background: #fff5f5; }
|
||||
|
||||
.provider-row-top {
|
||||
display: flex;
|
||||
@@ -134,6 +135,7 @@
|
||||
.status-dot.live { background: #10b981; }
|
||||
.status-dot.stale { background: #f59e0b; }
|
||||
.status-dot.unavailable { background: #9ca3af; }
|
||||
.status-dot.unreachable { background: #ef4444; }
|
||||
|
||||
.status-chip {
|
||||
display: inline-block;
|
||||
@@ -147,6 +149,7 @@
|
||||
.status-chip.live { background: #d1fae5; color: #065f46; }
|
||||
.status-chip.stale { background: #fef3c7; color: #92400e; }
|
||||
.status-chip.unavailable { background: #f3f4f6; color: #6b7280; }
|
||||
.status-chip.unreachable { background: #fee2e2; color: #991b1b; }
|
||||
|
||||
.chip-sm {
|
||||
display: inline-block;
|
||||
@@ -168,6 +171,12 @@
|
||||
font-style: italic;
|
||||
padding: 0.25rem 0 0;
|
||||
}
|
||||
.unreachable-reason {
|
||||
font-size: 0.875rem;
|
||||
color: #b91c1c;
|
||||
font-style: italic;
|
||||
padding: 0.25rem 0 0;
|
||||
}
|
||||
.last-fresh-tag {
|
||||
font-size: 0.7rem;
|
||||
color: #9ca3af;
|
||||
@@ -294,7 +303,7 @@
|
||||
<div id="panel-chains"><div class="panel-loading">Loading…</div></div>
|
||||
</section>
|
||||
</div>
|
||||
<footer>OLP Dashboard · Plan Usage: 60s refresh · other panels: 30s · paused when tab hidden · v0.5.0-phase5</footer>
|
||||
<footer>OLP Dashboard · Plan Usage: 60s refresh · other panels: 30s · paused when tab hidden · v0.5.1</footer>
|
||||
<script>
|
||||
(function () {
|
||||
'use strict';
|
||||
@@ -442,6 +451,24 @@
|
||||
continue;
|
||||
}
|
||||
|
||||
/* ── unreachable (v0.5.1): probe failed + no cache — show failure detail ── */
|
||||
if (status === 'unreachable') {
|
||||
const failure = entry.failure || {};
|
||||
const kind = failure.kind || 'unknown';
|
||||
const msg = failure.message || 'probe failed — no cached data available';
|
||||
const shortText = `${kind}: ${msg}`;
|
||||
rowEl.appendChild(el('div', { class: 'unreachable-reason' }, shortText));
|
||||
if (failure.backoff_until) {
|
||||
const backoffMs = Math.max(0, failure.backoff_until - Date.now());
|
||||
const backoffSec = Math.round(backoffMs / 1000);
|
||||
if (backoffSec > 0) {
|
||||
rowEl.appendChild(el('div', { class: 'unavailable-reason' }, `backoff active: ${backoffSec}s remaining`));
|
||||
}
|
||||
}
|
||||
frag.appendChild(rowEl);
|
||||
continue;
|
||||
}
|
||||
|
||||
/* ── utilization bars (5h + 7d) ── */
|
||||
const util = entry.utilization || {};
|
||||
const reset = entry.reset || {};
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
- `quotaStatus(authContext): { quotaInfo }` is now permitted to call provider HTTP APIs directly, subject to **all three** constraints:
|
||||
1. **READ-ONLY** — the API call must not mutate provider-side state. POST is acceptable when the response is what's needed (Anthropic returns ratelimit headers on `POST /v1/messages`); the request body MUST minimise side-effects (`max_tokens: 1`, dummy `messages`).
|
||||
2. **Subscription-scope reuse** — the credentials used MUST be the same auth artifact the spawn path already reads via `readAuthArtifact()`. No new OAuth grant, no new API-key registration, no separate scopes.
|
||||
3. **Idempotent failure** — if the probe fails for any reason (network error, 401, 429, schema parse failure), the function returns `null` rather than throwing. The caller (server.mjs / dashboard / `olp usage` CLI) interprets `null` as "quota data unavailable" and continues gracefully.
|
||||
3. **Idempotent failure** — if the probe fails for any reason (network error, 401, 429, schema parse failure), the function returns a structured shape (`{ probe_status: 'unreachable', failure: { kind, message, backoff_until? } }` since v0.5.1; see ADR 0013 Rule 6 + ADR 0008 Amendment 2) rather than throwing. The caller (server.mjs / dashboard / `olp usage` CLI) gracefully degrades. At v0.5.0 the failure shape was the literal value `null`; v0.5.1 refined this to a structured shape so operators can distinguish auth failures from rate-limit failures from network failures from in-backoff stale-cache. The substantive idempotent-failure constraint (no throw to caller) is unchanged.
|
||||
- `healthCheck()` and other contract methods are NOT extended by this Amendment. Only `quotaStatus()` may make direct API calls. A plugin that wants live data for any other contract method must continue to use `spawn` or `readAuthArtifact`.
|
||||
- The probe MUST cache its result. Recommended TTL: 5 minutes (mirrors OCP `USAGE_CACHE_TTL`). Tighter TTLs (e.g. dashboard's 1-minute refresh) are served from the cached value if fresh; cache miss triggers a real probe.
|
||||
- The probe MUST implement exponential backoff on refresh failures: minimum 60s, maximum 3600s (mirrors OCP `OAUTH_REFRESH_MIN_BACKOFF` / `OAUTH_REFRESH_MAX_BACKOFF`). Tight loop on failure has historically burned through Anthropic's rate limit in seconds (OCP institutional lesson 2026-04).
|
||||
@@ -30,7 +30,7 @@
|
||||
- **API calls to endpoints not explicitly enumerated by the companion ADR 0013 § Rule 2.** Amendment 8 permits the *kind* of call (READ-ONLY direct API for quota probing); ADR 0013 Rule 2 enumerates *which specific endpoint* is permitted. A future reader of Amendment 8 alone should NOT infer that any READ-ONLY/idempotent endpoint is fair game — the per-endpoint containment is locked to ADR 0013. Re-opening per-endpoint scope requires an ADR 0013 amendment, not a new plugin-level interpretation of Amendment 8.
|
||||
- **Backwards compatibility:** Plugins whose `quotaStatus()` still returns `null` (mistral at v0.5.0 pending D84 audit, codex permanently per Phase 5 charter) are NOT affected. No existing behaviour changes for them.
|
||||
- **Authority cited at the implementation:** D80 commit cites this Amendment + `~/.cc-rules/memory/learnings/anthropic_plan_usage_probe_schema_2026_05_26.md` + `Claude Code v2.1.x § OAuth bearer + ratelimit-unified headers` + live-probe transcript from 2026-05-26 in the commit body. ALIGNMENT.md Rule 1 + Rule 5 (CI) both satisfied.
|
||||
- **Tests:** Suite 38 (Phase 5 D83) covers the probe: mock HTTP server returning all 13 `anthropic-ratelimit-unified-*` headers; assert parse correctness for each; assert 5min cache; assert 60s-3600s exponential backoff on simulated 429; assert stale-cache-on-failure (`null` is returned only when no cache entry exists; if a stale entry exists it's returned with a `stale: true` marker).
|
||||
- **Tests:** Suite 38 (Phase 5 D83) covers the probe: mock HTTP server returning all 13 `anthropic-ratelimit-unified-*` headers; assert parse correctness for each; assert 5min cache; assert 60s-3600s exponential backoff on simulated 429; assert stale-cache-on-failure. At v0.5.0 stale-failure returned `{ stale: true, ... }` with `null` reserved for no-cache failures; v0.5.1 refined the return contract — `null` is now reserved STRICTLY for opt-in-off, and all failure modes (auth / rate-limit / schema-drift / network / no-creds) return `{ probe_status: 'unreachable' | 'stale', failure: {...} }`. See `test-features.mjs` Suite 38 (38u/38v/38w added for the v0.5.1 hotfix regression coverage of F1 / F2 / F3 per codex review).
|
||||
- **Procedural mechanism:** Iron Rule 11 (IDR) — this Amendment, ADR 0012 (Phase 5 charter), and ADR 0013 (OAuth READ-ONLY consumption rules) land together at D79 as a single coupled commit. Reviewing them separately cannot verify consumer-producer alignment. Iron Rule 10 fresh-context reviewer per CLAUDE.md hard requirement #3.
|
||||
|
||||
### Amendment 7 — 2026-05-26: Add OPTIONAL `doctorChecks()` to the Provider contract (D67 — Phase 4 operator UX)
|
||||
|
||||
@@ -5,6 +5,67 @@
|
||||
|
||||
## Amendments
|
||||
|
||||
### Amendment 2 — 2026-05-27: v0.5.1 quota_v2 richer failure-mode shape (codex finding F3)
|
||||
|
||||
**Scope:** v0.5.1 hotfix extends `ProviderQuotaEntry` and `aggregateProviderQuota()` to surface richer failure-mode detail, addressing codex review finding F3 (operator cannot distinguish failure modes from the `unavailable` catch-all). Authority: ADR 0013 Rule 6 + codex review findings F1–F3.
|
||||
|
||||
#### 1. Extended `ProviderQuotaEntry` shape
|
||||
|
||||
```js
|
||||
{
|
||||
provider: string,
|
||||
// v0.5.1: 'unreachable' added (probe enabled, creds present, but no cache + probe failed)
|
||||
status: 'live' | 'stale' | 'unreachable' | 'unavailable',
|
||||
reason?: string, // only when status === 'unavailable' (no API or disabled)
|
||||
schema_version: string|null,
|
||||
last_fresh_at: number|null,
|
||||
utilization: { '5h': number|null, '7d': number|null } | null,
|
||||
reset: { '5h': number|null, '7d': number|null, overall: number|null, overage: number|null } | null,
|
||||
representative_claim: string|null,
|
||||
fallback_percentage: number|null,
|
||||
overage: { status: string|null, disabled_reason: string|null } | null,
|
||||
raw_available: boolean,
|
||||
// v0.5.1 (F3 — ADR 0013 Rule 6):
|
||||
failure: { kind, message, backoff_until? } | null,
|
||||
failure_kind: 'no_credentials'|'auth_failed'|'rate_limited'|'schema_drift'|'network'|'other' | null,
|
||||
// Note: 'opt_in_off' is NOT in this enum — when probe is opted out, the row's status
|
||||
// is 'unavailable' (not 'unreachable'); failure_kind stays null. Distinguishing
|
||||
// "user opted out" from "provider has no API" requires reading config separately.
|
||||
backoff_until: number | null, // epoch-ms when next probe attempt is allowed
|
||||
}
|
||||
```
|
||||
|
||||
Status semantics:
|
||||
- `'unavailable'` — probe disabled (`quota_probe_enabled: false`) OR provider has no public quota API (codex, mistral). `failure`, `failure_kind`, `backoff_until` are null.
|
||||
- `'live'` — probe succeeded within TTL. `failure` is null.
|
||||
- `'stale'` — probe failed but stale cache exists. `failure.kind` describes why the last probe failed. `last_fresh_at` is the epoch of the last successful probe. `backoff_until` tells when the next attempt is scheduled.
|
||||
- `'unreachable'` (new) — probe enabled + creds present (or missing!) but no cache available + probe failed. `failure.kind` distinguishes: `no_credentials`, `auth_failed`, `rate_limited`, `schema_drift`, `network`, `other`. `utilization` and `reset` are null (no data).
|
||||
|
||||
#### 2. `quotaStatus()` v0.5.1 return contract
|
||||
|
||||
`null` is now RESERVED for `quota_probe_enabled: false` only. All other failure paths return a structured shape:
|
||||
|
||||
```js
|
||||
null // ONLY: opt-in off
|
||||
{ probe_status: 'live', ... } // cache fresh
|
||||
{ probe_status: 'stale', ..., failure: { kind, message, backoff_until } } // cache stale + backoff
|
||||
{ probe_status: 'unreachable', source, schemaVersion, failure: { ... } } // no cache + failed
|
||||
```
|
||||
|
||||
The `stale: boolean` field is retained for backwards-compat (`stale: false` on live, `stale: true` on stale). New code should use `probe_status`.
|
||||
|
||||
#### 3. `dashboard.html` unreachable rendering
|
||||
|
||||
A new CSS class `.provider-row.unreachable` (red border + light red background) and `.unreachable-reason` text style handle the new status. `failure.message` and `failure_kind` are surfaced as a short text line under the provider badge. `failure.backoff_until` renders a "backoff active: Xs remaining" note if within window.
|
||||
|
||||
#### 4. Authority
|
||||
|
||||
- ADR 0013 Rule 6 (failure transparency mandate)
|
||||
- Codex review findings F1 (doctor bypass), F2 (200+empty-headers → schema_drift), F3 (failure-mode collapse)
|
||||
- v0.5.1 hotfix PR
|
||||
|
||||
---
|
||||
|
||||
### Amendment 1 — 2026-05-26: D81 Phase 5 quota_v2 shape + aggregateProviderQuota()
|
||||
|
||||
**Scope:** D81 (Phase 5 / ADR 0012 D81) extends the audit-query layer and dashboard-data endpoint to surface the new per-provider quota shape introduced by D80 (`lib/providers/anthropic.mjs:quotaStatus()`). This amendment documents the three new interfaces.
|
||||
|
||||
@@ -47,9 +47,10 @@ The probe MUST NOT call any other HTTP path on the provider's API. No `/v1/model
|
||||
|
||||
- Cache TTL: 5 minutes. Cache miss triggers a real probe. Cache hit returns the cached value.
|
||||
- The dashboard refreshes every 1 minute; that's served from the cache between probes. A manual refresh button MAY force-clear the cache (per maintainer request 2026-05-26); ADR 0012 D82 documents the button.
|
||||
- On refresh failure (token expired, 401/403/429, network error), the probe schedules an exponential backoff: minimum 60s, maximum 3600s. The cache entry is NOT invalidated during backoff; `quotaStatus()` returns the stale cache marked `{ stale: true, last_fresh_at: <epoch> }`. If no stale entry exists, returns `null`.
|
||||
- On refresh failure (token expired, 401/403/429, network error), the probe schedules an exponential backoff: minimum 60s, maximum 3600s. The cache entry is NOT invalidated during backoff; `quotaStatus()` returns the stale cache marked `{ stale: true, last_fresh_at: <epoch> }`. If no stale entry exists, returns an `unreachable` shape (v0.5.1+) rather than `null`.
|
||||
- Successive successful probes reset the backoff to the minimum.
|
||||
- Token refresh (`POST https://platform.claude.com/v1/oauth/token`) follows the same backoff discipline. The probe MUST NOT refresh a token more than once per backoff window. The refresh path is shared with the spawn path; both observe the same backoff.
|
||||
- **All consumers of `quotaStatus()`, including `olp doctor` checks, MUST route through `quotaStatus()` and MUST NOT call `_probeOnce()` directly.** `_probeOnce()` is an internal implementation detail. Routing doctor checks through `quotaStatus()` ensures the cache+backoff discipline is enforced for every caller — including operators running `olp doctor` in a debug loop. (Clarification added v0.5.1 to address codex finding F1: the original doctor check bypassed backoff by calling `_probeOnce` directly.)
|
||||
|
||||
### Rule 4 — Opt-in via config
|
||||
|
||||
@@ -70,7 +71,17 @@ Default: `false`. The maintainer must explicitly opt in after credentials are co
|
||||
|
||||
`olp doctor` adds a per-provider check `<provider>.quota_probe_reachable` (only runs if `quota_probe_enabled: true`). Failed check provides a `next_action.ai_executable[]` recipe to either re-authenticate or disable the probe.
|
||||
|
||||
### Rule 5 — Schema-drift mitigation protocol
|
||||
### Rule 5 — Schema-drift mitigation protocol (minimum-viable-schema gate)
|
||||
|
||||
The CC binary-distribution shift means OCP's "grep cli.js" verification is no longer applicable. OLP adopts a two-path protocol for proactive monitoring, AND enforces a minimum-viable-schema gate at parse time:
|
||||
|
||||
**Minimum-viable-schema gate (v0.5.1+).** `_probeOnce()` requires at least these 4 fields present (non-null after parse) before treating a response as successful:
|
||||
- `anthropic-ratelimit-unified-5h-utilization`
|
||||
- `anthropic-ratelimit-unified-5h-reset`
|
||||
- `anthropic-ratelimit-unified-7d-utilization`
|
||||
- `anthropic-ratelimit-unified-7d-reset`
|
||||
|
||||
If any of these 4 is absent, `_probeOnce()` classifies the probe as a schema-drift failure (`failureKind = 'schema_drift'`), schedules backoff, and returns `null`. This means a 200 OK with zero `anthropic-ratelimit-*` headers (e.g. a server-side change, a proxy stripping headers, or a mock returning `{}`) is immediately caught as drift rather than silently cached as "live" data. The other 9 fields are tolerated as absent (overage fields are conditional; top-level status fields may be absent on edge cases). The 5h/7d core 4 are load-bearing — the dashboard's progress bars depend on them. (Gate added v0.5.1 to address codex finding F2.)
|
||||
|
||||
The CC binary-distribution shift means OCP's "grep cli.js" verification is no longer applicable. OLP adopts a two-path protocol:
|
||||
|
||||
|
||||
+34
-7
@@ -449,19 +449,26 @@ export function spendTrendDaily({ days, olpHome, logEvent, _nowFn } = {}) {
|
||||
* Provider-specific: called only for 'anthropic'. Returns null if the raw
|
||||
* shape is absent or malformed.
|
||||
*
|
||||
* v0.5.1: handles probe_status field (F3 — ADR 0013 Rule 6).
|
||||
* Accepts both old shape (stale: boolean) and new shape (probe_status: string).
|
||||
*
|
||||
* @internal — used by aggregateProviderQuota()
|
||||
*/
|
||||
function _normalizeAnthropicQuota(raw) {
|
||||
if (!raw || typeof raw !== 'object') return null;
|
||||
const f = raw.fields ?? {};
|
||||
// v0.5.1: probe_status field (new) takes precedence; fall back to stale bool for compat.
|
||||
const probeStatus = raw.probe_status ?? (raw.stale === true ? 'stale' : 'live');
|
||||
return {
|
||||
schema_version: raw.schemaVersion ?? null,
|
||||
last_fresh_at: raw.stale ? (raw.last_fresh_at ?? null) : (raw.probedAt ?? null),
|
||||
utilization: {
|
||||
last_fresh_at: (probeStatus === 'stale')
|
||||
? (raw.last_fresh_at ?? null)
|
||||
: (raw.probedAt ?? null),
|
||||
utilization: probeStatus === 'unreachable' ? null : {
|
||||
'5h': f.utilization_5h ?? null,
|
||||
'7d': f.utilization_7d ?? null,
|
||||
},
|
||||
reset: {
|
||||
reset: probeStatus === 'unreachable' ? null : {
|
||||
'5h': f.reset_5h ?? null,
|
||||
'7d': f.reset_7d ?? null,
|
||||
overall: f.reset ?? null,
|
||||
@@ -469,11 +476,15 @@ function _normalizeAnthropicQuota(raw) {
|
||||
},
|
||||
representative_claim: f.representative_claim ?? null,
|
||||
fallback_percentage: f.fallback_percentage ?? null,
|
||||
overage: {
|
||||
overage: probeStatus === 'unreachable' ? null : {
|
||||
status: f.overage_status ?? null,
|
||||
disabled_reason: f.overage_disabled_reason ?? null,
|
||||
},
|
||||
raw_available: (typeof raw.raw === 'object' && raw.raw !== null),
|
||||
// v0.5.1 (F3 — ADR 0013 Rule 6): failure detail for operator diagnostics
|
||||
failure: raw.failure ?? null,
|
||||
failure_kind: raw.failure?.kind ?? null,
|
||||
backoff_until: raw.failure?.backoff_until ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -556,7 +567,8 @@ export async function aggregateProviderQuota({
|
||||
}
|
||||
|
||||
if (rawResult === null || rawResult === undefined) {
|
||||
// Plugin returned null: either disabled, no quota API, or no credentials.
|
||||
// Plugin returned null: opt-in disabled (the ONLY case per v0.5.1 contract)
|
||||
// or providers with no quota API at all (codex, mistral).
|
||||
results.push({
|
||||
provider: name,
|
||||
status: 'unavailable',
|
||||
@@ -569,14 +581,18 @@ export async function aggregateProviderQuota({
|
||||
fallback_percentage: null,
|
||||
overage: null,
|
||||
raw_available: false,
|
||||
failure: null,
|
||||
failure_kind: null,
|
||||
backoff_until: null,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
// quotaStatus() returned a non-null shape — normalize.
|
||||
// v0.5.1: handle probe_status field (live/stale/unreachable).
|
||||
// Currently only 'anthropic' returns a structured shape; other providers
|
||||
// returning structured data will work if their shape is compatible.
|
||||
const isStale = rawResult.stale === true;
|
||||
const probeStatus = rawResult.probe_status ?? (rawResult.stale === true ? 'stale' : 'live');
|
||||
const normalized = _normalizeAnthropicQuota(rawResult);
|
||||
|
||||
if (normalized === null) {
|
||||
@@ -593,13 +609,24 @@ export async function aggregateProviderQuota({
|
||||
fallback_percentage: null,
|
||||
overage: null,
|
||||
raw_available: false,
|
||||
failure: null,
|
||||
failure_kind: null,
|
||||
backoff_until: null,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
// Map probe_status to output status:
|
||||
// 'live' → 'live'
|
||||
// 'stale' → 'stale'
|
||||
// 'unreachable' → 'unreachable' (new in v0.5.1; dashboard renders with red border)
|
||||
const outputStatus = probeStatus === 'unreachable' ? 'unreachable'
|
||||
: probeStatus === 'stale' ? 'stale'
|
||||
: 'live';
|
||||
|
||||
results.push({
|
||||
provider: name,
|
||||
status: isStale ? 'stale' : 'live',
|
||||
status: outputStatus,
|
||||
...normalized,
|
||||
});
|
||||
}
|
||||
|
||||
+244
-81
@@ -182,6 +182,9 @@ export function _resetQuotaProbeStateForTest() {
|
||||
quotaProbeState.cache = null;
|
||||
quotaProbeState.backoffUntil = 0;
|
||||
quotaProbeState.backoffMs = QUOTA_BACKOFF_MIN_MS;
|
||||
// v0.5.1 (F3): reset failure-tracking fields
|
||||
quotaProbeState.lastError = null;
|
||||
quotaProbeState.failureKind = null;
|
||||
}
|
||||
|
||||
// Convenience: reset state only, leaving URL overrides in place.
|
||||
@@ -190,6 +193,9 @@ export function _resetQuotaStateOnlyForTest() {
|
||||
quotaProbeState.cache = null;
|
||||
quotaProbeState.backoffUntil = 0;
|
||||
quotaProbeState.backoffMs = QUOTA_BACKOFF_MIN_MS;
|
||||
// v0.5.1 (F3): reset failure-tracking fields
|
||||
quotaProbeState.lastError = null;
|
||||
quotaProbeState.failureKind = null;
|
||||
}
|
||||
|
||||
// Returns a direct reference to the internal quotaProbeState object.
|
||||
@@ -234,10 +240,18 @@ function _resolveSchemaVersion() {
|
||||
|
||||
// Module-level probe state — one instance per process (not per request).
|
||||
// Ported from OCP server.mjs:852 (usageCache) + 862 (oauthRefreshBackoff).
|
||||
// v0.5.1: added lastError + failureKind for F3 failure-transparency (ADR 0013 Rule 6).
|
||||
const quotaProbeState = {
|
||||
cache: null, // { fetchedAt: <ms>, data: <quotaShape> } when fresh; null when not
|
||||
backoffUntil: 0, // ms epoch: when next refresh attempt is allowed
|
||||
backoffMs: QUOTA_BACKOFF_MIN_MS, // current backoff window; doubled on failure, reset on success
|
||||
// v0.5.1 (F3 — ADR 0013 Rule 6 failure transparency):
|
||||
lastError: null, // { kind, message, statusCode?, attemptedAt } | null
|
||||
failureKind: null, // 'no_credentials'|'auth_failed'|'rate_limited'|'schema_drift'|'network'|'other' | null
|
||||
// Note: 'opt_in_off' is NOT in this enum — when the probe is opted out
|
||||
// quotaStatus() returns the literal null BEFORE any state mutation, so
|
||||
// failureKind is never produced for the disabled case. Consumers
|
||||
// (audit-query, doctor) distinguish opt-in-off by quotaStatus() === null.
|
||||
};
|
||||
|
||||
// ── Config reader: providers.<name>.<field> ───────────────────────────────
|
||||
@@ -468,21 +482,56 @@ async function _probeOnce(creds) {
|
||||
}
|
||||
|
||||
// Non-2xx AND no ratelimit headers → treat as failure
|
||||
// Classify failure kind for F3 (ADR 0013 Rule 6 failure transparency).
|
||||
if (statusCode >= 400 && Object.keys(rlHeaders).length === 0) {
|
||||
// Schedule backoff on failure
|
||||
const now = Date.now();
|
||||
const kind = (statusCode === 401 || statusCode === 403) ? 'auth_failed'
|
||||
: statusCode === 429 ? 'rate_limited'
|
||||
: 'other';
|
||||
quotaProbeState.lastError = { kind, message: `HTTP ${statusCode}`, statusCode, attemptedAt: now };
|
||||
quotaProbeState.failureKind = kind;
|
||||
_scheduleBackoff();
|
||||
return null;
|
||||
}
|
||||
|
||||
// Success (or non-2xx with headers present — headers still valid quota data)
|
||||
// Reset backoff on successful probe
|
||||
// v0.5.1 F2 (ADR 0013 Rule 5 minimum-viable-schema gate):
|
||||
// Require at least these 4 load-bearing fields (5h-utilization, 5h-reset,
|
||||
// 7d-utilization, 7d-reset). A 200 OK with zero ratelimit headers (or with
|
||||
// headers but missing these 4) is treated as schema-drift and classified as
|
||||
// failure. The other 9 fields are tolerated as missing.
|
||||
//
|
||||
// Authority: ADR 0013 Rule 5 + codex review finding F2 (v0.5.1 hotfix).
|
||||
const MIN_FIELDS = [
|
||||
'anthropic-ratelimit-unified-5h-utilization',
|
||||
'anthropic-ratelimit-unified-5h-reset',
|
||||
'anthropic-ratelimit-unified-7d-utilization',
|
||||
'anthropic-ratelimit-unified-7d-reset',
|
||||
];
|
||||
const missingMinFields = MIN_FIELDS.filter(f => rlHeaders[f] == null);
|
||||
if (missingMinFields.length > 0) {
|
||||
const now = Date.now();
|
||||
const msg = `schema_drift: missing minimum-viable fields: ${missingMinFields.join(', ')}`;
|
||||
quotaProbeState.lastError = { kind: 'schema_drift', message: msg, statusCode, attemptedAt: now };
|
||||
quotaProbeState.failureKind = 'schema_drift';
|
||||
_scheduleBackoff();
|
||||
return null;
|
||||
}
|
||||
|
||||
// Success (or non-2xx with headers present and passing min-field gate —
|
||||
// headers still valid quota data). Reset backoff on successful probe.
|
||||
quotaProbeState.backoffMs = QUOTA_BACKOFF_MIN_MS;
|
||||
quotaProbeState.backoffUntil = 0;
|
||||
quotaProbeState.lastError = null;
|
||||
quotaProbeState.failureKind = null;
|
||||
|
||||
const fields = _parseRateLimitHeaders(rlHeaders);
|
||||
return { fields, raw: rlHeaders };
|
||||
|
||||
} catch {
|
||||
} catch (err) {
|
||||
const now = Date.now();
|
||||
const msg = err?.message ?? String(err);
|
||||
quotaProbeState.lastError = { kind: 'network', message: msg, attemptedAt: now };
|
||||
quotaProbeState.failureKind = 'network';
|
||||
_scheduleBackoff();
|
||||
return null;
|
||||
}
|
||||
@@ -817,46 +866,71 @@ export function estimateCost(request) {
|
||||
// Port source: OCP server.mjs:842-1109 (usageCache, refreshOAuthToken, fetchUsageFromApi,
|
||||
// parseRateLimitHeaders, handleUsage)
|
||||
//
|
||||
// ADR 0013 Rule 4 (opt-in): returns null unless config.providers.anthropic.quota_probe_enabled
|
||||
// is explicitly true in ~/.olp/config.json.
|
||||
// ADR 0013 Rule 4 (opt-in): returns null ONLY when quota_probe_enabled is false.
|
||||
// All other failure paths return a structured failure shape instead of null.
|
||||
//
|
||||
// Return shape (when enabled + successful):
|
||||
// {
|
||||
// probedAt: <unix-ms>,
|
||||
// source: 'anthropic-ratelimit-unified-headers',
|
||||
// schemaVersion: '2026-05-26',
|
||||
// stale: false,
|
||||
// fields: { ...13 fields... },
|
||||
// raw: { ...response-headers-object... }
|
||||
// }
|
||||
// Returns null when: disabled, no credentials, probe failure with no stale cache.
|
||||
// Returns { ...shape, stale: true, last_fresh_at } when probe failed but cache exists.
|
||||
// v0.5.1 return contract (F1+F3 — ADR 0013 Rules 3+6 + codex review):
|
||||
//
|
||||
// null → ONLY when quota_probe_enabled: false (opt-in off)
|
||||
//
|
||||
// { probe_status: 'live', probedAt, source, schemaVersion,
|
||||
// stale: false, fields, raw }
|
||||
// → probe succeeded recently (within QUOTA_CACHE_TTL_MS)
|
||||
//
|
||||
// { probe_status: 'stale', ...cachedShape, stale: true, last_fresh_at,
|
||||
// failure: { kind, message, backoff_until } }
|
||||
// → probe failed but cache exists (backoff active)
|
||||
//
|
||||
// { probe_status: 'unreachable', source, schemaVersion,
|
||||
// failure: { kind, message, backoff_until? } }
|
||||
// → probe failed AND no cache (includes no_credentials case when
|
||||
// creds absent after opt-in, also in-backoff-no-cache)
|
||||
//
|
||||
// Backwards-compat note: stale:false shape gains probe_status:'live' (additive).
|
||||
// stale:true shape gains probe_status:'stale' (additive). 'unreachable' replaces
|
||||
// the prior null for failure cases (breaking only if caller checked null for
|
||||
// "disabled" — now null strictly means disabled only).
|
||||
export async function quotaStatus(_authContext) {
|
||||
// ADR 0013 Rule 4 — opt-in check
|
||||
// ADR 0013 Rule 4 — opt-in check; null means "disabled", not "failed"
|
||||
const providerCfg = _readProviderConfig('anthropic');
|
||||
if (providerCfg.quota_probe_enabled !== true) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const schemaVersion = _resolveSchemaVersion();
|
||||
const now = Date.now();
|
||||
|
||||
// Cache hit: return cached value if within TTL (ADR 0013 Rule 3)
|
||||
if (quotaProbeState.cache !== null &&
|
||||
(now - quotaProbeState.cache.fetchedAt) < QUOTA_CACHE_TTL_MS) {
|
||||
return quotaProbeState.cache.data;
|
||||
// Cache is fresh — return probe_status:'live' shape
|
||||
return { ...quotaProbeState.cache.data, probe_status: 'live' };
|
||||
}
|
||||
|
||||
// Backoff check: still in exponential backoff window — return stale cache or null
|
||||
// Backoff check: still in exponential backoff window — return stale cache or unreachable
|
||||
if (now < quotaProbeState.backoffUntil) {
|
||||
const failureInfo = {
|
||||
kind: quotaProbeState.failureKind ?? 'other',
|
||||
message: quotaProbeState.lastError?.message ?? 'probe failed (in backoff)',
|
||||
backoff_until: quotaProbeState.backoffUntil,
|
||||
};
|
||||
if (quotaProbeState.cache !== null) {
|
||||
// Return stale cache marked as stale (ADR 0013 Rule 3)
|
||||
return {
|
||||
...quotaProbeState.cache.data,
|
||||
probe_status: 'stale',
|
||||
stale: true,
|
||||
last_fresh_at: quotaProbeState.cache.fetchedAt,
|
||||
failure: failureInfo,
|
||||
};
|
||||
}
|
||||
return null; // no stale cache
|
||||
// In backoff but no cache — unreachable
|
||||
return {
|
||||
probe_status: 'unreachable',
|
||||
source: 'anthropic-ratelimit-unified-headers',
|
||||
schemaVersion,
|
||||
failure: failureInfo,
|
||||
};
|
||||
}
|
||||
|
||||
// Read auth artifact (ADR 0013 Rule 1 — same credentials as spawn path).
|
||||
@@ -864,8 +938,25 @@ export async function quotaStatus(_authContext) {
|
||||
const _authReadFn = _quotaAuthReadFnForTest ?? readAuthArtifact;
|
||||
const creds = _authReadFn();
|
||||
if (!creds?.accessToken) {
|
||||
// No credentials at all — return stale cache or null; do not hammer API with 401s
|
||||
return null;
|
||||
// No credentials — set failureKind so doctor/aggregator can distinguish this
|
||||
quotaProbeState.lastError = { kind: 'no_credentials', message: 'no OAuth credential found', attemptedAt: now };
|
||||
quotaProbeState.failureKind = 'no_credentials';
|
||||
if (quotaProbeState.cache !== null) {
|
||||
// Stale cache still available — return it with failure info
|
||||
return {
|
||||
...quotaProbeState.cache.data,
|
||||
probe_status: 'stale',
|
||||
stale: true,
|
||||
last_fresh_at: quotaProbeState.cache.fetchedAt,
|
||||
failure: { kind: 'no_credentials', message: 'no OAuth credential found' },
|
||||
};
|
||||
}
|
||||
return {
|
||||
probe_status: 'unreachable',
|
||||
source: 'anthropic-ratelimit-unified-headers',
|
||||
schemaVersion,
|
||||
failure: { kind: 'no_credentials', message: 'no OAuth credential found' },
|
||||
};
|
||||
}
|
||||
|
||||
// Perform the probe
|
||||
@@ -878,7 +969,8 @@ export async function quotaStatus(_authContext) {
|
||||
source: 'anthropic-ratelimit-unified-headers',
|
||||
// D81: schema_version sourced from models-registry.json with QUOTA_SCHEMA_VERSION
|
||||
// as fallback per ADR 0013 Rule 5 + D81 ADR 0008 Amendment requirement.
|
||||
schemaVersion: _resolveSchemaVersion(),
|
||||
schemaVersion,
|
||||
probe_status: 'live',
|
||||
stale: false,
|
||||
fields: probeResult.fields,
|
||||
raw: probeResult.raw,
|
||||
@@ -887,16 +979,29 @@ export async function quotaStatus(_authContext) {
|
||||
return shape;
|
||||
}
|
||||
|
||||
// Probe failed: _probeOnce already called _scheduleBackoff().
|
||||
// Return stale cache if present, else null.
|
||||
// Probe failed: _probeOnce already called _scheduleBackoff() and set lastError/failureKind.
|
||||
const failureInfo = {
|
||||
kind: quotaProbeState.failureKind ?? 'other',
|
||||
message: quotaProbeState.lastError?.message ?? 'probe failed',
|
||||
backoff_until: quotaProbeState.backoffUntil,
|
||||
};
|
||||
|
||||
if (quotaProbeState.cache !== null) {
|
||||
return {
|
||||
...quotaProbeState.cache.data,
|
||||
probe_status: 'stale',
|
||||
stale: true,
|
||||
last_fresh_at: quotaProbeState.cache.fetchedAt,
|
||||
failure: failureInfo,
|
||||
};
|
||||
}
|
||||
return null;
|
||||
// No cache — return unreachable shape (F3: operator can distinguish failure modes)
|
||||
return {
|
||||
probe_status: 'unreachable',
|
||||
source: 'anthropic-ratelimit-unified-headers',
|
||||
schemaVersion,
|
||||
failure: failureInfo,
|
||||
};
|
||||
}
|
||||
|
||||
// ── healthCheck ───────────────────────────────────────────────────────────
|
||||
@@ -999,73 +1104,131 @@ export function doctorChecks({ _binaryExistsFn, _authReadFn } = {}) {
|
||||
{
|
||||
id: 'anthropic.quota_probe_reachable',
|
||||
category: 'provider',
|
||||
// D80 / ADR 0013 Rule 6 — failure transparency for olp doctor.
|
||||
// Only runs if quota_probe_enabled: true in ~/.olp/config.json.
|
||||
// Status mapping:
|
||||
// ok → probe returned fresh data
|
||||
// warn → stale cache returned (probe failed but cache present)
|
||||
// fail → no cache AND probe failed; fix_commands recipe provided
|
||||
// v0.5.1 (F1 — ADR 0013 Rule 3 + codex review):
|
||||
// Routes through quotaStatus() so it respects cache + backoff.
|
||||
// Bypassing _probeOnce() directly (old code) allowed successive `olp doctor`
|
||||
// invocations to each hit upstream even when backoffUntil was in the future.
|
||||
//
|
||||
// Status mapping based on quotaStatus() probe_status:
|
||||
// null (opt-in off) → ok + advisory
|
||||
// probe_status: 'live' → ok + utilization data in message
|
||||
// probe_status: 'stale' → warn + age info + failure kind in message
|
||||
// probe_status: 'unreachable' (no_credentials) → fail + human_steps (re-login)
|
||||
// probe_status: 'unreachable' (auth_failed/rate_limited/schema_drift/network/other)
|
||||
// → fail + human_steps + fix_commands
|
||||
async run() {
|
||||
const providerCfg = _readProviderConfig('anthropic');
|
||||
if (providerCfg.quota_probe_enabled !== true) {
|
||||
return { status: 'ok', message: 'quota probe disabled (opt-in via config.providers.anthropic.quota_probe_enabled)' };
|
||||
// Inject the test auth override so tests can seed the quotaAuthReadFn.
|
||||
// In production _quotaAuthReadFnForTest is null so quotaStatus() reads
|
||||
// credentials normally. Tests can set it via _setQuotaAuthReadFnForTest.
|
||||
if (authRead !== readAuthArtifact) {
|
||||
_quotaAuthReadFnForTest = authRead;
|
||||
}
|
||||
try {
|
||||
const result = await quotaStatus(null);
|
||||
|
||||
// Attempt a fresh probe (bypassing module-level cache for doctor diagnostics)
|
||||
const auth = authRead();
|
||||
if (!auth?.accessToken) {
|
||||
// null → opt-in off
|
||||
if (result === null) {
|
||||
return { status: 'ok', message: 'quota probe disabled (opt-in via config.providers.anthropic.quota_probe_enabled)' };
|
||||
}
|
||||
|
||||
const probeStatus = result.probe_status;
|
||||
|
||||
if (probeStatus === 'live') {
|
||||
const util5h = result.fields?.utilization_5h;
|
||||
const util7d = result.fields?.utilization_7d;
|
||||
const msg = `quota probe OK — 5h utilization: ${util5h !== null && util5h !== undefined ? `${Math.round(util5h * 100)}%` : 'n/a'}, 7d utilization: ${util7d !== null && util7d !== undefined ? `${Math.round(util7d * 100)}%` : 'n/a'}`;
|
||||
return { status: 'ok', message: msg };
|
||||
}
|
||||
|
||||
if (probeStatus === 'stale') {
|
||||
const ageMin = result.last_fresh_at
|
||||
? Math.round((Date.now() - result.last_fresh_at) / 60_000)
|
||||
: null;
|
||||
const failKind = result.failure?.kind ?? 'other';
|
||||
const ageStr = ageMin !== null ? ` from ${ageMin}min ago` : '';
|
||||
return {
|
||||
status: 'warn',
|
||||
message: `quota probe failed (${failKind}); returning stale cache${ageStr}; check network or token expiry`,
|
||||
evidence: {
|
||||
fix_commands: [
|
||||
'olp doctor # re-run after verifying OAuth credentials are valid',
|
||||
],
|
||||
human_steps: [
|
||||
'run: claude (if token may have expired, re-login)',
|
||||
],
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// probe_status === 'unreachable' — distinguish failure kind
|
||||
const failKind = result.failure?.kind ?? 'other';
|
||||
const failMsg = result.failure?.message ?? 'probe failed';
|
||||
|
||||
if (failKind === 'no_credentials') {
|
||||
return {
|
||||
status: 'fail',
|
||||
message: `quota probe enabled but no OAuth credential — ${failMsg}`,
|
||||
evidence: {
|
||||
human_steps: [
|
||||
'run: claude (first interactive launch prompts browser OAuth login)',
|
||||
],
|
||||
reference: 'https://docs.anthropic.com/en/docs/claude-code/setup#authentication',
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
if (failKind === 'auth_failed') {
|
||||
return {
|
||||
status: 'fail',
|
||||
message: `quota probe auth failure (${failMsg}); OAuth token may be expired`,
|
||||
evidence: {
|
||||
human_steps: [
|
||||
'if OAuth token is expired: run: claude (re-login)',
|
||||
'to disable probe: set providers.anthropic.quota_probe_enabled = false in ~/.olp/config.json',
|
||||
],
|
||||
reference: 'https://docs.anthropic.com/en/docs/claude-code/setup#authentication',
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
if (failKind === 'schema_drift') {
|
||||
return {
|
||||
status: 'fail',
|
||||
message: `quota probe schema drift detected (${failMsg}); schema may have changed`,
|
||||
evidence: {
|
||||
human_steps: [
|
||||
'check for OLP updates: the anthropic-ratelimit-unified-* header schema may have changed',
|
||||
'file an issue if the schema has genuinely drifted (ADR 0013 Rule 5)',
|
||||
],
|
||||
fix_commands: [
|
||||
'node -e "const {_getQuotaProbeStateForTest: s}=await import(\'./lib/providers/anthropic.mjs\');console.log(JSON.stringify(s().lastError))"',
|
||||
],
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// rate_limited / network / other
|
||||
return {
|
||||
status: 'fail',
|
||||
message: 'quota probe enabled but no OAuth credential — probe cannot run',
|
||||
evidence: {
|
||||
human_steps: [
|
||||
'run: claude (first interactive launch prompts browser OAuth login)',
|
||||
],
|
||||
reference: 'https://docs.anthropic.com/en/docs/claude-code/setup#authentication',
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const result = await _probeOnce(auth);
|
||||
if (result !== null) {
|
||||
const util5h = result.fields.utilization_5h;
|
||||
const util7d = result.fields.utilization_7d;
|
||||
const msg = `quota probe OK — 5h utilization: ${util5h !== null ? `${Math.round(util5h * 100)}%` : 'n/a'}, 7d utilization: ${util7d !== null ? `${Math.round(util7d * 100)}%` : 'n/a'}`;
|
||||
return { status: 'ok', message: msg };
|
||||
}
|
||||
|
||||
// Probe failed — check if stale cache exists
|
||||
if (quotaProbeState.cache !== null) {
|
||||
const ageMin = Math.round((Date.now() - quotaProbeState.cache.fetchedAt) / 60_000);
|
||||
return {
|
||||
status: 'warn',
|
||||
message: `quota probe failed (returning stale cache from ${ageMin}min ago); check network or token expiry`,
|
||||
message: `quota probe failed and no cached data available (${failKind}: ${failMsg})`,
|
||||
evidence: {
|
||||
fix_commands: [
|
||||
'olp doctor # re-run after verifying OAuth credentials are valid',
|
||||
'node -e "const { readAuthArtifact } = await import(\'./lib/providers/anthropic.mjs\'); console.log(JSON.stringify(readAuthArtifact()?.accessToken ? \'token_present\' : \'no_token\'))"',
|
||||
],
|
||||
human_steps: [
|
||||
'run: claude (if token may have expired, re-login)',
|
||||
'verify network access: curl -s -o /dev/null -w "%{http_code}" https://api.anthropic.com/v1/messages',
|
||||
'if OAuth token is expired: run: claude (re-login)',
|
||||
'to disable probe: set providers.anthropic.quota_probe_enabled = false in ~/.olp/config.json',
|
||||
],
|
||||
},
|
||||
};
|
||||
} finally {
|
||||
// Restore auth override so this doctor run does not permanently affect
|
||||
// the module-level state beyond this call.
|
||||
if (authRead !== readAuthArtifact) {
|
||||
_quotaAuthReadFnForTest = null;
|
||||
}
|
||||
}
|
||||
|
||||
// No cache at all — hard failure
|
||||
return {
|
||||
status: 'fail',
|
||||
message: 'quota probe failed and no cached data available; check OAuth credentials and network access to api.anthropic.com',
|
||||
evidence: {
|
||||
fix_commands: [
|
||||
'node -e "const { readAuthArtifact } = await import(\'./lib/providers/anthropic.mjs\'); console.log(JSON.stringify(readAuthArtifact()?.accessToken ? \'token_present\' : \'no_token\'))"',
|
||||
],
|
||||
human_steps: [
|
||||
'verify network access: curl -s -o /dev/null -w "%{http_code}" https://api.anthropic.com/v1/messages',
|
||||
'if OAuth token is expired: run: claude (re-login)',
|
||||
'to disable probe: set providers.anthropic.quota_probe_enabled = false in ~/.olp/config.json',
|
||||
],
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "olp",
|
||||
"version": "0.5.0",
|
||||
"version": "0.5.1",
|
||||
"description": "Personal multi-provider LLM proxy. Successor to OCP. One HTTP endpoint, multiple subscriptions behind it, automatic routing + fallback + caching.",
|
||||
"type": "module",
|
||||
"main": "server.mjs",
|
||||
|
||||
+207
-21
@@ -15865,10 +15865,10 @@ describe('Suite 38 — D83 quota-probe unit tests (Phase 5, ADR 0012 D83 + ADR 0
|
||||
}
|
||||
});
|
||||
|
||||
it('38f — quotaStatus enabled + no auth → returns null without HTTP', async () => {
|
||||
it('38f — quotaStatus enabled + no auth → returns unreachable shape without HTTP (v0.5.1)', async () => {
|
||||
// v0.5.1 (F3): no_credentials is no longer null — it returns probe_status:'unreachable'
|
||||
// so the operator can distinguish "disabled" (null) from "no credentials" (unreachable).
|
||||
// Uses _setQuotaAuthReadFnForTest to inject a "no credentials" auth reader.
|
||||
// This prevents the test from being affected by real ~/.claude/.credentials.json
|
||||
// on the developer's machine (which would cause a false pass/fail).
|
||||
_saveEnv38();
|
||||
const TMP = _mkdtempSync38(_pathJoin38(_tmpdir38(), 'olp-38f-'));
|
||||
let requestCount = 0;
|
||||
@@ -15887,7 +15887,11 @@ describe('Suite 38 — D83 quota-probe unit tests (Phase 5, ADR 0012 D83 + ADR 0
|
||||
resetQuotaStateOnly38();
|
||||
|
||||
const result = await quotaStatus38();
|
||||
assert.equal(result, null, '38f: null when auth returns null (no creds)');
|
||||
// v0.5.1: no_credentials → probe_status:'unreachable' (NOT null)
|
||||
// null is now ONLY returned for opt-in disabled.
|
||||
assert.ok(result !== null, '38f: no-creds result should NOT be null in v0.5.1 (F3: null reserved for disabled)');
|
||||
assert.equal(result.probe_status, 'unreachable', '38f: probe_status should be unreachable when no creds');
|
||||
assert.equal(result.failure?.kind, 'no_credentials', '38f: failure.kind should be no_credentials');
|
||||
assert.equal(requestCount, 0, '38f: no HTTP request should be made without auth');
|
||||
} finally {
|
||||
if (mock) await mock.close();
|
||||
@@ -15916,7 +15920,8 @@ describe('Suite 38 — D83 quota-probe unit tests (Phase 5, ADR 0012 D83 + ADR 0
|
||||
|
||||
const result = await quotaStatus38();
|
||||
assert.ok(result !== null, '38g: should return non-null');
|
||||
assert.equal(result.stale, false, '38g: stale=false on fresh probe');
|
||||
assert.equal(result.probe_status, 'live', '38g: probe_status=live on fresh probe (v0.5.1)');
|
||||
assert.equal(result.stale, false, '38g: stale=false on fresh probe (backwards-compat)');
|
||||
assert.equal(result.source, 'anthropic-ratelimit-unified-headers', '38g: source field');
|
||||
assert.ok(typeof result.probedAt === 'number', '38g: probedAt should be a number');
|
||||
assert.ok(typeof result.schemaVersion === 'string', '38g: schemaVersion should be a string');
|
||||
@@ -16042,7 +16047,11 @@ describe('Suite 38 — D83 quota-probe unit tests (Phase 5, ADR 0012 D83 + ADR 0
|
||||
resetQuotaStateOnly38();
|
||||
|
||||
const result = await quotaStatus38();
|
||||
assert.equal(result, null, '38j: 401 with no refreshToken → null (idempotent-failure per ADR 0002 Amendment 8)');
|
||||
// v0.5.1 (F3): 401 with no cache → probe_status:'unreachable' + failure_kind:'auth_failed'
|
||||
// (idempotent-failure per ADR 0002 Amendment 8 still holds — no panic, no retry loop)
|
||||
assert.ok(result !== null, '38j: 401+no-refreshToken → unreachable shape (not null in v0.5.1)');
|
||||
assert.equal(result.probe_status, 'unreachable', '38j: probe_status should be unreachable');
|
||||
assert.equal(result.failure?.kind, 'auth_failed', '38j: failure.kind should be auth_failed on 401');
|
||||
assert.equal(apiCallCount, 1, '38j: only one API call made (no retry without refreshToken)');
|
||||
assert.equal(oauthCallCount, 0, '38j: no OAuth refresh call without refreshToken');
|
||||
} finally {
|
||||
@@ -16166,8 +16175,11 @@ describe('Suite 38 — D83 quota-probe unit tests (Phase 5, ADR 0012 D83 + ADR 0
|
||||
// Second call hits 429 → should return stale cache
|
||||
const r2 = await quotaStatus38();
|
||||
assert.ok(r2 !== null, '38k: 429 with stale cache → should return stale cache (not null)');
|
||||
assert.equal(r2.stale, true, '38k: stale should be true');
|
||||
assert.equal(r2.probe_status, 'stale', '38k: probe_status=stale (v0.5.1)');
|
||||
assert.equal(r2.stale, true, '38k: stale should be true (backwards-compat)');
|
||||
assert.ok(typeof r2.last_fresh_at === 'number', '38k: last_fresh_at should be present');
|
||||
assert.ok(r2.failure !== null && r2.failure !== undefined, '38k: failure info should be present (F3)');
|
||||
assert.equal(r2.failure.kind, 'rate_limited', '38k: failure.kind should be rate_limited on 429 (F3)');
|
||||
} finally {
|
||||
if (mock) await mock.close();
|
||||
resetQuotaProbe38();
|
||||
@@ -16194,7 +16206,11 @@ describe('Suite 38 — D83 quota-probe unit tests (Phase 5, ADR 0012 D83 + ADR 0
|
||||
resetQuotaStateOnly38();
|
||||
|
||||
const result = await quotaStatus38();
|
||||
assert.equal(result, null, '38l: 429 with no cache → null');
|
||||
// v0.5.1 (F3): 429 with no cache → probe_status:'unreachable' + failure_kind:'rate_limited'
|
||||
// (not null — null is now reserved for opt-in disabled only)
|
||||
assert.ok(result !== null, '38l: 429+no-cache → unreachable shape (not null in v0.5.1)');
|
||||
assert.equal(result.probe_status, 'unreachable', '38l: probe_status should be unreachable');
|
||||
assert.equal(result.failure?.kind, 'rate_limited', '38l: failure.kind should be rate_limited on 429');
|
||||
|
||||
// Verify backoff was scheduled
|
||||
const state = getQuotaProbeState38();
|
||||
@@ -16417,7 +16433,12 @@ describe('Suite 38 — D83 quota-probe unit tests (Phase 5, ADR 0012 D83 + ADR 0
|
||||
}
|
||||
});
|
||||
|
||||
it('38r — doctor: probe enabled + probe fails + cache stale → warn', async () => {
|
||||
it('38r — doctor: probe enabled + probe fails + cache stale → warn (v0.5.1)', async () => {
|
||||
// v0.5.1 (F1): doctor now routes through quotaStatus() — it respects cache+backoff.
|
||||
// To simulate "stale cache" state: seed cache via a successful probe, then manually
|
||||
// expire the cache (fetchedAt = 0) and set backoffUntil in the future (simulate a
|
||||
// failed probe having scheduled backoff). quotaStatus() will see expired cache +
|
||||
// active backoff → returns probe_status:'stale'. Doctor maps this → 'warn'.
|
||||
_saveEnv38();
|
||||
const TMP = _mkdtempSync38(_pathJoin38(_tmpdir38(), 'olp-38r-'));
|
||||
let callCount = 0;
|
||||
@@ -16428,13 +16449,8 @@ describe('Suite 38 — D83 quota-probe unit tests (Phase 5, ADR 0012 D83 + ADR 0
|
||||
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'test-token-38r';
|
||||
mock = await _startMockServer((_req, res) => {
|
||||
callCount++;
|
||||
if (callCount === 1) {
|
||||
// First: succeed to seed cache
|
||||
res.writeHead(200, _ALL_13_HEADERS);
|
||||
} else {
|
||||
// Subsequent: fail
|
||||
res.writeHead(500, {});
|
||||
}
|
||||
// Always succeed — we only call this once to seed the cache
|
||||
res.writeHead(200, _ALL_13_HEADERS);
|
||||
res.end('{}');
|
||||
});
|
||||
setQuotaUrls38(`${mock.url}/v1/messages`, `${mock.url}/v1/oauth/token`);
|
||||
@@ -16443,10 +16459,15 @@ describe('Suite 38 — D83 quota-probe unit tests (Phase 5, ADR 0012 D83 + ADR 0
|
||||
// Seed cache with a successful probe
|
||||
await quotaStatus38();
|
||||
const state = getQuotaProbeState38();
|
||||
// Set cache to expired so doctor re-probes (doctor calls _probeOnce directly)
|
||||
// Doctor check calls _probeOnce(auth) directly — it always makes a fresh probe
|
||||
// regardless of cache. After _probeOnce fails, it checks quotaProbeState.cache.
|
||||
// The state.cache is still the seeded one.
|
||||
assert.ok(state.cache !== null, '38r setup: cache should be seeded');
|
||||
|
||||
// Simulate "stale" state: expire cache + set backoff as if a probe just failed.
|
||||
// Doctor now calls quotaStatus() which will see: cache expired + backoff active
|
||||
// → returns probe_status:'stale'.
|
||||
state.cache.fetchedAt = Date.now() - 10 * 60 * 1000; // 10 min ago (beyond 5min TTL)
|
||||
state.backoffUntil = Date.now() + 60_000; // backoff active for 60s
|
||||
state.lastError = { kind: 'network', message: 'probe timeout', attemptedAt: Date.now() - 10_000 };
|
||||
state.failureKind = 'network';
|
||||
|
||||
const checks = doctorChecks38({
|
||||
_binaryExistsFn: () => true,
|
||||
@@ -16454,9 +16475,11 @@ describe('Suite 38 — D83 quota-probe unit tests (Phase 5, ADR 0012 D83 + ADR 0
|
||||
});
|
||||
const probeCheck = checks.find(c => c.id === 'anthropic.quota_probe_reachable');
|
||||
const result = await probeCheck.run();
|
||||
assert.equal(result.status, 'warn', '38r: failed probe with stale cache → warn');
|
||||
assert.equal(result.status, 'warn', '38r: stale cache → warn');
|
||||
assert.ok(result.message.includes('stale') || result.message.includes('failed'),
|
||||
'38r: message should mention stale or failure');
|
||||
// F1 regression: callCount should still be 1 — doctor did NOT make a second upstream call
|
||||
assert.equal(callCount, 1, '38r F1 regression: doctor should NOT make additional upstream HTTP calls (respects backoff)');
|
||||
} finally {
|
||||
if (mock) await mock.close();
|
||||
resetQuotaProbe38();
|
||||
@@ -16530,6 +16553,169 @@ describe('Suite 38 — D83 quota-probe unit tests (Phase 5, ADR 0012 D83 + ADR 0
|
||||
}
|
||||
});
|
||||
|
||||
// ── 38u–38w: v0.5.1 regression tests (F1/F2/F3 codex review findings) ──────
|
||||
|
||||
it('38u — F1 regression: successive doctor calls within backoff → no upstream HTTP on second call', async () => {
|
||||
// ADR 0013 Rule 3: doctor check MUST respect backoff (F1 — codex review).
|
||||
// Old code called _probeOnce() directly, bypassing backoff.
|
||||
// New code routes through quotaStatus() which enforces backoff.
|
||||
_saveEnv38();
|
||||
const TMP = _mkdtempSync38(_pathJoin38(_tmpdir38(), 'olp-38u-'));
|
||||
let upstreamCallCount = 0;
|
||||
let mock;
|
||||
try {
|
||||
_writeProbeConfig(TMP);
|
||||
process.env.OLP_HOME = TMP;
|
||||
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'test-token-38u';
|
||||
mock = await _startMockServer((_req, res) => {
|
||||
upstreamCallCount++;
|
||||
// Always return 500 → triggers backoff
|
||||
res.writeHead(500, {});
|
||||
res.end('{}');
|
||||
});
|
||||
setQuotaUrls38(`${mock.url}/v1/messages`, `${mock.url}/v1/oauth/token`);
|
||||
resetQuotaStateOnly38();
|
||||
|
||||
const checks = doctorChecks38({
|
||||
_binaryExistsFn: () => true,
|
||||
_authReadFn: () => ({ accessToken: 'test-token-38u' }),
|
||||
});
|
||||
const probeCheck = checks.find(c => c.id === 'anthropic.quota_probe_reachable');
|
||||
|
||||
// First doctor call: probe fires → 500 → backoff scheduled, counter = 1
|
||||
const r1 = await probeCheck.run();
|
||||
assert.equal(r1.status, 'fail', '38u: first doctor call should fail (500, no cache)');
|
||||
assert.equal(upstreamCallCount, 1, '38u: first doctor call should fire exactly one upstream request');
|
||||
|
||||
// Second doctor call: within backoff window → MUST NOT fire another upstream request
|
||||
const r2 = await probeCheck.run();
|
||||
assert.equal(r2.status, 'fail', '38u: second doctor call should also fail (still in backoff)');
|
||||
assert.equal(upstreamCallCount, 1,
|
||||
'38u F1 regression: second doctor call within backoff MUST NOT fire another upstream request (counter stays 1)');
|
||||
} finally {
|
||||
if (mock) await mock.close();
|
||||
resetQuotaProbe38();
|
||||
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
||||
_restoreEnv38();
|
||||
_rmSync38(TMP, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('38v — F2 regression: 200 with empty ratelimit headers → classified as failure (schema_drift)', async () => {
|
||||
// ADR 0013 Rule 5 minimum-viable-schema gate (F2 — codex review).
|
||||
// A 200 OK with zero anthropic-ratelimit-* headers was previously cached as
|
||||
// "live" data. With the minimum-field gate, it must be classified as failure.
|
||||
_saveEnv38();
|
||||
const TMP = _mkdtempSync38(_pathJoin38(_tmpdir38(), 'olp-38v-'));
|
||||
let mock;
|
||||
try {
|
||||
_writeProbeConfig(TMP);
|
||||
process.env.OLP_HOME = TMP;
|
||||
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'test-token-38v';
|
||||
mock = await _startMockServer((_req, res) => {
|
||||
// 200 OK but zero anthropic-ratelimit-* headers → schema_drift trigger
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
res.end('{}');
|
||||
});
|
||||
setQuotaUrls38(`${mock.url}/v1/messages`, `${mock.url}/v1/oauth/token`);
|
||||
resetQuotaStateOnly38();
|
||||
|
||||
const result = await quotaStatus38();
|
||||
// Must be classified as failure (unreachable with schema_drift kind)
|
||||
assert.ok(result !== null, '38v: result should not be null (v0.5.1: null only for disabled)');
|
||||
assert.equal(result.probe_status, 'unreachable', '38v F2: 200+no-ratelimit-headers → unreachable (schema_drift)');
|
||||
assert.equal(result.failure?.kind, 'schema_drift', '38v F2: failure.kind should be schema_drift');
|
||||
|
||||
// Verify backoff was scheduled (not a silent success)
|
||||
const state = getQuotaProbeState38();
|
||||
assert.ok(state.backoffUntil > Date.now(), '38v F2: backoff should be scheduled on schema_drift');
|
||||
assert.equal(state.failureKind, 'schema_drift', '38v F2: quotaProbeState.failureKind should be schema_drift');
|
||||
|
||||
// Verify cache is NOT populated (schema drift must not be cached as live data)
|
||||
assert.equal(state.cache, null, '38v F2: cache must remain null — schema_drift must NOT be cached as live data');
|
||||
} finally {
|
||||
if (mock) await mock.close();
|
||||
resetQuotaProbe38();
|
||||
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
||||
_restoreEnv38();
|
||||
_rmSync38(TMP, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('38w — F3 regression: lastError + failureKind propagate through quotaStatus() shape', async () => {
|
||||
// ADR 0013 Rule 6 failure transparency (F3 — codex review).
|
||||
// Verifies that failure details (kind, message, backoff_until) are present in
|
||||
// the quotaStatus() return shape for each distinct failure mode.
|
||||
_saveEnv38();
|
||||
const TMP = _mkdtempSync38(_pathJoin38(_tmpdir38(), 'olp-38w-'));
|
||||
let mock;
|
||||
try {
|
||||
_writeProbeConfig(TMP);
|
||||
process.env.OLP_HOME = TMP;
|
||||
process.env.CLAUDE_CODE_OAUTH_TOKEN = 'test-token-38w';
|
||||
|
||||
// Test 1: rate_limited (429)
|
||||
mock = await _startMockServer((_req, res) => {
|
||||
res.writeHead(429, {});
|
||||
res.end('{}');
|
||||
});
|
||||
setQuotaUrls38(`${mock.url}/v1/messages`, `${mock.url}/v1/oauth/token`);
|
||||
resetQuotaStateOnly38();
|
||||
|
||||
const r1 = await quotaStatus38();
|
||||
assert.equal(r1.probe_status, 'unreachable', '38w: 429 → unreachable');
|
||||
assert.ok(r1.failure !== null, '38w: failure should be present');
|
||||
assert.equal(r1.failure.kind, 'rate_limited', '38w: 429 → failure.kind = rate_limited');
|
||||
assert.ok(typeof r1.failure.message === 'string', '38w: failure.message should be a string');
|
||||
assert.ok(typeof r1.failure.backoff_until === 'number', '38w: failure.backoff_until should be a number');
|
||||
|
||||
await mock.close();
|
||||
mock = null;
|
||||
|
||||
// Test 2: auth_failed (401)
|
||||
resetQuotaStateOnly38();
|
||||
mock = await _startMockServer((_req, res) => {
|
||||
res.writeHead(401, {});
|
||||
res.end('{}');
|
||||
});
|
||||
setQuotaUrls38(`${mock.url}/v1/messages`, `${mock.url}/v1/oauth/token`);
|
||||
|
||||
const r2 = await quotaStatus38();
|
||||
assert.equal(r2.probe_status, 'unreachable', '38w: 401 → unreachable');
|
||||
assert.equal(r2.failure?.kind, 'auth_failed', '38w: 401 → failure.kind = auth_failed');
|
||||
|
||||
await mock.close();
|
||||
mock = null;
|
||||
|
||||
// Test 3: schema_drift (200 + no min fields)
|
||||
resetQuotaStateOnly38();
|
||||
mock = await _startMockServer((_req, res) => {
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
res.end('{}');
|
||||
});
|
||||
setQuotaUrls38(`${mock.url}/v1/messages`, `${mock.url}/v1/oauth/token`);
|
||||
|
||||
const r3 = await quotaStatus38();
|
||||
assert.equal(r3.probe_status, 'unreachable', '38w: schema_drift → unreachable');
|
||||
assert.equal(r3.failure?.kind, 'schema_drift', '38w: 200+no-min-fields → failure.kind = schema_drift');
|
||||
|
||||
// Test 4: no_credentials
|
||||
resetQuotaStateOnly38();
|
||||
setQuotaAuthFn38(() => null);
|
||||
|
||||
const r4 = await quotaStatus38();
|
||||
assert.equal(r4.probe_status, 'unreachable', '38w: no_credentials → unreachable');
|
||||
assert.equal(r4.failure?.kind, 'no_credentials', '38w: no creds → failure.kind = no_credentials');
|
||||
|
||||
} finally {
|
||||
if (mock) await mock.close();
|
||||
resetQuotaProbe38();
|
||||
delete process.env.CLAUDE_CODE_OAUTH_TOKEN;
|
||||
_restoreEnv38();
|
||||
_rmSync38(TMP, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
});
|
||||
|
||||
// ── Suite 39: D83 dashboard rendering smoke tests (Phase 5, ADR 0012 D83) ──
|
||||
|
||||
Reference in New Issue
Block a user