fix(server): serialize -p real-HOME token fallback + TTL-cache keychain + de-stale isolation decision (F3/F5/F6) (#150)

* fix(server): serialize -p real-HOME token fallback + TTL-cache keychain + de-stale isolation decision (F3/F5/F6)

Three audit findings in the -p spawn-token resolution + HOME-isolation layer. All are infra/
process changes to how OCP READS and GATES an OAuth token it already holds; none touch the OAuth
wire machinery.

F3 (MEDIUM) — expiry-window fallback herds concurrent -p spawns into real HOME.
When the keychain token is within 5 min of expiry, resolveSpawnToken() returns null and every
concurrent spawn simultaneously falls back to the real HOME; each spawned claude then races a
refresh_token grant against the SAME single-use refresh token — rotating it out from under the
others and the operator's real claude (the credential-fork hazard, #112/#146 class). Fix: a
promise-chain mutex (createSerialMutex) serializes ONLY the real-HOME fallback — one such spawn at
a time. When a serialized waiter is admitted (prior holder torn down → its claude has refreshed the
keychain), it re-runs resolveSpawnToken(): a now-fresh token means it proceeds ISOLATED instead of
real-HOME, so the queue drains to the fast path. Isolated spawns never touch the mutex.

F5 (LOW-MED) — per-spawn double keychain exec on the hot path.
getOAuthCredentials() sync-exec'd `security find-generic-password` up to twice (wrong label first),
worst case 5s×2, blocking the event loop and stalling in-flight SSE streams. Fix: (a) memoize the
last-good keychain label and try it first (orderLabelsLastGoodFirst); (b) a 30s TTL cache of the
read (createTtlCache). This does NOT reintroduce the #146 forever-memoized regression: the TTL
bounds only how often we re-READ the keychain; resolveSpawnToken() still applies the 5-min expiry
gate (isTokenExpiring) to the CACHED creds on EVERY use, so a token expiring within the window is
still rejected → real-HOME fallback. Call sites stay synchronous (no async conversion).

F6 (LOW) — memoized isolation decision goes stale; /health could misreport.
getSpawnHomeMode() memoized the isolated/real-home decision forever: credentials appearing after
startup never enabled isolation; deleting ~/.ocp/spawn-home at runtime ENOENT'd every isolated
spawn until restart; during an expiry stint /health reported isolated:true while spawns ran real-
HOME. Fix: re-evaluate the decision per spawn (cheap now that F5 caches the keychain read);
ensureSpawnHome() re-verifies + re-prepares the scratch dir per isolated spawn; and /health now
reports the EFFECTIVE decision (token presence AND expiry gate). The /health field set is
UNCHANGED — no field added/removed/renamed — only the values are made truthful.

Alignment:
- Class: Not a wire/endpoint change for the spawn-token layer + Class B (B.2) for /health.
- cli.js citation: DECLARED ABSENT. cli.js does NOT perform OCP's spawn-token resolution, HOME
  isolation, keychain caching, or fallback serialization — these are proxy-internal process
  concerns with no cli.js analogue, so no Class A cli.js:NNNN citation exists or is required
  (ALIGNMENT.md Rule 2: this is proxy infra, not an invented forwarded endpoint/header/body).
- The /health change is authorized by ADR 0006 (grandfathered B.2 as of v3.16.4) and is a
  behaviour-preserving contract change: same fields, truthful values.
- OCP still NEVER performs a refresh_token grant itself — that property is preserved; the fix only
  serializes/gates reads of a token refreshed by the spawned or real claude (#112).
- No new endpoint, header, or request/response field. alignment.yml blacklist unaffected.

Tests: extracted the pure primitives to lib/spawn-auth.mjs and added 11 unit tests (mutex
serialization order + idempotent release; TTL cache freshness + null-miss; expiry gate; label
ordering; and the combined invariant that the TTL cache respects the expiry gate). node --check
clean; 249 tests pass (238 + 11).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(server): drain F3 fallback queue immediately — invalidate F5 keychain cache before the serialized re-check

Follow-up to the F3/F5/F6 fix (independent-review observation). F5's 30s keychain TTL cache could
make F3's post-refresh re-check see the stale (expiring) cached creds for up to ~30s, so a waiter
admitted right after the prior real-HOME holder's claude refreshed the keychain would needlessly
fall back to real HOME again instead of proceeding ISOLATED. Serialization safety was never at risk
(still one real-HOME spawn at a time, no double-refresh); only the drain-to-fast-path optimization
lagged.

Fix: invalidateKeychainReadCache() clears the F5 TTL cache; resolveSpawnDecision() calls it under
the fallback mutex, immediately before the re-check, so the admitted waiter reads FRESH keychain
state and drains to the isolated fast path at once. The extra keychain read happens only on the rare
real-HOME fallback path and only under the mutex (serialized, one at a time).

Alignment: unchanged from the parent commit — proxy-internal keychain/HOME-isolation process logic,
no cli.js analogue (cli.js citation DECLARED ABSENT, ALIGNMENT.md Rule 2), no endpoint/header/body,
/health shape unchanged, OCP still never performs a refresh_token grant itself (#112). 249 tests pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: dtzp555 <dtzp555@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

lib/spawn-auth.mjs

@@ -0,0 +1,67 @@
+// Pure, dependency-injected primitives for the `-p` spawn-token resolution + HOME-isolation
+// layer. Extracted from server.mjs (findings F3 / F5 / F6, 2026-07-07) so the concurrency,
+// caching and expiry logic is unit-testable WITHOUT booting the server or mocking execFileSync /
+// child_process.spawn / fs. server.mjs owns all I/O (macOS keychain exec, process spawn, fs);
+// this module owns only pure decision logic.
+//
+// ALIGNMENT NOTE: none of this touches the OAuth wire machinery (no endpoint / header / body).
+// OCP still NEVER performs a refresh_token grant itself — these helpers only READ + GATE a token
+// that some other process (the operator's real claude, or a spawned claude under the real HOME)
+// refreshes. That property is load-bearing (issue #112) and preserved.
+
+// Promise-chain mutex. `acquire()` resolves to a `release()` fn; the NEXT `acquire()` does not
+// resolve until the current holder calls its `release()`. Serializes async critical sections
+// without busy-waiting. release() is idempotent.
+export function createSerialMutex() {
+  let tail = Promise.resolve();
+  return {
+    acquire() {
+      let release;
+      const gate = new Promise((r) => { release = r; });
+      const prev = tail;
+      tail = tail.then(() => gate);
+      // Hand the caller its release fn only after the previous holder has released.
+      return prev.then(() => {
+        let released = false;
+        return function releaseMutex() { if (!released) { released = true; release(); } };
+      });
+    },
+  };
+}
+
+// Short-TTL memo. `get(produce, now)` returns the cached value while `now - storedAt < ttlMs`,
+// otherwise calls `produce()` and re-stores. A miss that produces null/undefined is STILL stored
+// (so a genuinely-absent source is not re-probed on every call within the TTL window). `now` is
+// injectable for testing.
+export function createTtlCache({ ttlMs }) {
+  let value;
+  let at = -Infinity;
+  let has = false;
+  return {
+    get(produce, now = Date.now()) {
+      if (has && now - at < ttlMs) return value;
+      value = produce();
+      at = now;
+      has = true;
+      return value;
+    },
+    clear() { has = false; value = undefined; at = -Infinity; },
+  };
+}
+
+// Pure expiry gate. Returns true when `creds` carries a known expiry that is at/within `bufferMs`
+// of `now`. Creds WITHOUT `expiresAt` (e.g. long-lived env tokens) are never treated as expiring.
+// This gate is applied to the CACHED creds on EVERY use — which is precisely why a short-TTL
+// keychain cache (createTtlCache) cannot reintroduce the #146 forever-stale-token regression: the
+// cache bounds how often we re-READ the keychain, but the expiry decision is recomputed per use.
+export function isTokenExpiring(creds, now = Date.now(), bufferMs = 300000) {
+  return !!(creds && creds.expiresAt && now + bufferMs >= creds.expiresAt);
+}
+
+// Order candidate keychain labels so the last-known-good label is tried first (avoids the
+// wrong-label miss that doubles the `security` exec count on the hot path). Pure: performs no
+// read. Returns a fresh array; input is not mutated.
+export function orderLabelsLastGoodFirst(labels, lastGood) {
+  if (!lastGood || !labels.includes(lastGood)) return labels.slice();
+  return [lastGood, ...labels.filter((l) => l !== lastGood)];
+}

server.mjs

@@ -44,6 +44,7 @@ import { isLoopbackBind } from "./lib/net.mjs";
 import { runTuiTurn, reapStaleTuiSessions, resolveTuiHome } from "./lib/tui/session.mjs";
 import { detectTuiUpstreamError } from "./lib/tui/transcript.mjs";
 import { TuiSemaphore, recordTuiEntrypoint, buildTuiHealthBlock } from "./lib/tui/semaphore.mjs";
+import { createSerialMutex, createTtlCache, isTokenExpiring, orderLabelsLastGoodFirst } from "./lib/spawn-auth.mjs";
 
 const __dirname = dirname(fileURLToPath(import.meta.url));
 const _pkg = JSON.parse(readFileSync(join(__dirname, "package.json"), "utf8"));
@@ -387,50 +388,106 @@ function prepareSpawnHome(dir = SPAWN_HOME_DIR) {
   } catch { /* best effort — spawn will surface a hard error if the dir is truly unusable */ }
 }
 
-// Resolve the default-spawn HOME-isolation decision ONCE, lazily + memoized (so it runs after
+// Resolve the default-spawn HOME-isolation decision. Returns { isolated, home, reason }:
-// getOAuthCredentials is defined regardless of source order, and the token probe happens at most
-// once). Returns { isolated, home, token } where:
 //   - isolated:true  → spawn under SPAWN_HOME_DIR with cwd=SPAWN_HOME_DIR + the env token.
 //   - isolated:false → legacy real-HOME spawn, no cwd override (no token, or kill-switch on).
-// Caches only the isolation DECISION (isolated/home/reason), NOT the token — the token is
+//
-// re-resolved FRESH per spawn via resolveSpawnToken(). A memoized token goes stale when its
+// FIX F6 (2026-07-07): this decision is NO LONGER memoized permanently. The previous version
-// source rotates: the macOS keychain access token rotates (~hourly, refreshed by the operator's
+// cached it forever at first call, which meant: (a) credentials appearing after startup never
-// real claude), so a startup snapshot 401s once it expires (caused a ~31h Mac-mini 401 outage,
+// enabled isolation; (b) `rm -rf ~/.ocp/spawn-home` at runtime made every isolated spawn ENOENT
-// 2026-06-26). OCP deliberately does NOT refresh the token itself — a refresh-token grant would
+// until restart; (c) during a token-expiry stint /health reported isolated:true while spawns
-// consume the single-use refresh token and log out the operator's real claude (issue #112).
+// actually ran real-HOME. Re-evaluating per spawn is cheap because F5's 30s keychain TTL cache
-let _spawnHomeMode = null;
+// backs getOAuthCredentials(). This function is the CONFIG-level decision (isolated iff a token
+// resolves AND the kill-switch is off) and has NO fs side effects — the per-spawn EFFECTIVE
+// decision additionally applies the expiry gate (resolveSpawnDecision), and scratch-HOME dir prep
+// moved to ensureSpawnHome() at the isolated spawn site.
+//
+// The token itself is re-resolved FRESH per spawn via resolveSpawnToken(); a memoized token goes
+// stale when its source rotates (the macOS keychain access token rotates ~hourly, refreshed by the
+// operator's real claude), which 401'd every isolated spawn for ~31h on 2026-06-26 (#146). OCP
+// deliberately does NOT refresh the token itself — a refresh-token grant would consume the
+// single-use refresh token and log out the operator's real claude (issue #112).
 function getSpawnHomeMode() {
-  if (_spawnHomeMode) return _spawnHomeMode;
   if (SPAWN_REAL_HOME) {
-    _spawnHomeMode = { isolated: false, home: null, reason: "kill-switch (OCP_SPAWN_REAL_HOME=1)" };
+    return { isolated: false, home: null, reason: "kill-switch (OCP_SPAWN_REAL_HOME=1)" };
-    return _spawnHomeMode;
   }
   let hasToken = false;
   try { hasToken = !!(getOAuthCredentials()?.accessToken); } catch { hasToken = false; }
-  if (hasToken) {
+  if (hasToken) return { isolated: true, home: SPAWN_HOME_DIR, reason: "oauth token resolved" };
-    prepareSpawnHome(SPAWN_HOME_DIR);
+  return { isolated: false, home: null, reason: "no oauth token resolvable" };
-    _spawnHomeMode = { isolated: true, home: SPAWN_HOME_DIR, reason: "oauth token resolved" };
+}
-  } else {
+
-    _spawnHomeMode = { isolated: false, home: null, reason: "no oauth token resolvable" };
+// FIX F6: re-verify the scratch HOME exists before each isolated spawn and re-create it if it was
-  }
+// deleted at runtime (it used to be prepared once at startup, so a runtime deletion made every
-  return _spawnHomeMode;
+// isolated spawn fail ENOENT until restart). mkdirSync is recursive+idempotent → cheap to re-run.
+function ensureSpawnHome(dir = SPAWN_HOME_DIR) {
+  if (!existsSync(`${dir}/.claude`)) prepareSpawnHome(dir);
 }
 
 // Resolve a FRESH OAuth access token for an isolated spawn. Read-only (keychain / credentials.json
 // / env) — NEVER refreshes/rotates (see getSpawnHomeMode note). Returns null if none resolvable OR
-// if a known expiry has passed (5-min buffer): a null return makes the caller fall back to real
+// if a known expiry is within the 5-min buffer (isTokenExpiring): a null return makes the caller
-// HOME, where the spawned claude refreshes the credential natively and self-heals (the keychain
+// fall back to real HOME, where the spawned claude refreshes the credential natively and self-heals
-// token is then fresh again → next spawn is fast). The env-token path (Linux) carries no expiresAt
+// (the keychain token is then fresh again → next spawn is fast). The env-token path (Linux) carries
-// → never expiry-gated (those tokens are long-lived).
+// no expiresAt → never expiry-gated (those tokens are long-lived).
 function resolveSpawnToken() {
   try {
     const creds = getOAuthCredentials();
     if (!creds?.accessToken) return null;
-    if (creds.expiresAt && Date.now() + 300000 >= creds.expiresAt) return null;
+    if (isTokenExpiring(creds)) return null; // 5-min buffer; applied to the CACHED creds every use
     return creds.accessToken;
   } catch { return null; }
 }
 
+// FIX F3 (2026-07-07): serializes ONLY the real-HOME fallback spawns. Isolated spawns (the common
+// fast path) never touch this mutex.
+const realHomeFallbackMutex = createSerialMutex();
+
+// Resolve the EFFECTIVE per-spawn HOME/token decision. Returns
+//   { isolated, home, token, releaseFallback }
+// `releaseFallback` is non-null ONLY for a real-HOME fallback holder — the caller MUST call it on
+// spawn teardown (wired into cleanup()); it releases the serialization mutex. It is null (no-op)
+// for isolated and stable real-HOME (kill-switch / no-token) spawns.
+//
+// This is async so the real-HOME fallback can `await` the mutex; the keychain reads inside stay
+// synchronous (F5 keeps the call sites off async conversion).
+async function resolveSpawnDecision() {
+  const shm = getSpawnHomeMode();
+  if (!shm.isolated) return { isolated: false, home: null, token: null, releaseFallback: null };
+  const token = resolveSpawnToken();
+  if (token) {
+    ensureSpawnHome(shm.home);
+    return { isolated: true, home: shm.home, token, releaseFallback: null };
+  }
+  // Token is present but within the 5-min expiry window → we would fall back to real HOME, where
+  // the spawned claude refreshes the credential natively. HAZARD PREVENTED: without serialization,
+  // every concurrent -p spawn inside this window runs claude under the real HOME simultaneously,
+  // and each spawned claude races a `refresh_token` grant against the SAME single-use refresh
+  // token — rotating it out from under the others AND the operator's own real claude (the
+  // credential-fork hazard; #112 / #146 class). Serialize: admit ONE real-HOME spawn at a time.
+  // When the next waiter is admitted (the prior holder torn down → its claude has had its lifetime
+  // to refresh the keychain), re-run resolveSpawnToken(): a now-fresh token means we proceed
+  // ISOLATED and release the mutex immediately, so the queue drains to the fast path instead of
+  // piling every request into the real HOME.
+  const release = await realHomeFallbackMutex.acquire();
+  try {
+    // Drop the 30s keychain TTL cache so the re-check reads FRESH keychain state — otherwise a
+    // waiter admitted right after the prior holder's claude refreshed the token could still see the
+    // stale (expiring) cached creds and needlessly fall back to real HOME again for up to ~30s.
+    invalidateKeychainReadCache();
+    const retry = resolveSpawnToken();
+    if (retry) {
+      release();
+      ensureSpawnHome(shm.home);
+      return { isolated: true, home: shm.home, token: retry, releaseFallback: null };
+    }
+  } catch (e) {
+    release();
+    throw e;
+  }
+  return { isolated: false, home: null, token: null, releaseFallback: release };
+}
+
 // ── FIX ⑥ (concurrency): bounded wait-queue for the -p / stream-json path ──────────────
 // PROBLEM (proven): spawnClaudeProcess used `if (activeRequests >= MAX_CONCURRENT) throw` →
 // the client got an opaque 500 AND the rejection was NOT counted in stats (a 15-concurrent
@@ -931,7 +988,7 @@ function getModelTier(cliModel) {
 // budget. releaseSlot is wired into the idempotent cleanup() so the slot is freed on EVERY exit
 // path (close/error/timeout/abort). Back-compat: releaseSlot defaults to a no-op so any future
 // internal caller that does its own gating still works.
-function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlot = () => {}) {
+function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlot = () => {}, spawnDecision = null) {
   const cliModel = MODEL_MAP[model] || model;
 
   // Circuit breaker: disabled (see comment at top of breaker section)
@@ -968,28 +1025,24 @@ function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlo
     env.CLAUDE_CODE_DISABLE_AUTO_MEMORY = "1";
   }
 
-  // FIX ③ (latency): default-path spawn-home isolation. When a token is resolvable (and the
+  // FIX ③ (latency) + F3 (concurrency): apply the pre-resolved per-spawn HOME/token decision.
-  // OCP_SPAWN_REAL_HOME kill-switch is off), run claude under a credential-free minimal HOME
+  // The decision is resolved ASYNC in the caller (resolveSpawnDecision) so the real-HOME fallback
-  // with cwd = that same neutral dir, so it loads NONE of the operator's global ~/.claude
+  // serialization can await its mutex; here we only apply the result. When isolated, run claude
-  // (plugins/skills/hooks) or the ~/ocp project CLAUDE.md/skills — the measured 10–28s → 3–7s
+  // under a credential-free minimal HOME with cwd = that same neutral dir, so it loads NONE of the
-  // latency win. The env token is authoritative for `-p` (unlike interactive claude). When no
+  // operator's global ~/.claude (plugins/skills/hooks) or the ~/ocp project CLAUDE.md/skills — the
-  // token is resolvable, falls back to real HOME + inherited cwd (zero regression). See
+  // measured 10–28s → 3–7s latency win. The env token is authoritative for `-p` (unlike
-  // getSpawnHomeMode() / prepareSpawnHome() above. The DISABLE_CLAUDE_MDS / AUTO_MEMORY flags
+  // interactive claude). When no fresh token is resolvable, decision.isolated is false → real HOME
-  // are set unconditionally in isolated mode (belt-and-braces; mirrors the TUI path).
+  // + inherited cwd (zero regression), and the spawned claude resolves+refreshes credentials
-  const spawnHome = getSpawnHomeMode();
+  // natively. The DISABLE_CLAUDE_MDS / AUTO_MEMORY flags are set unconditionally in isolated mode
+  // (belt-and-braces; mirrors the TUI path).
+  const decision = spawnDecision || { isolated: false, releaseFallback: null };
   const spawnOpts = { env, stdio: ["pipe", "pipe", "pipe"] };
-  if (spawnHome.isolated) {
+  if (decision.isolated && decision.token) {
-    // Re-resolve the token FRESH per spawn (never a startup snapshot — keychain tokens rotate;
+    env.HOME = decision.home;
-    // a stale snapshot 401s). If unresolvable right now, fall through to real HOME so the spawned
+    env.CLAUDE_CODE_OAUTH_TOKEN = decision.token; // env token is authoritative for -p
-    // claude resolves + refreshes credentials natively instead of 401ing on a stale/null token.
-    const freshToken = resolveSpawnToken();
-    if (freshToken) {
-      env.HOME = spawnHome.home;
-      env.CLAUDE_CODE_OAUTH_TOKEN = freshToken; // env token is authoritative for -p
     env.CLAUDE_CODE_DISABLE_CLAUDE_MDS = "1";
     env.CLAUDE_CODE_DISABLE_AUTO_MEMORY = "1";
-      spawnOpts.cwd = spawnHome.home; // neutral cwd: no project CLAUDE.md/skills
+    spawnOpts.cwd = decision.home; // neutral cwd: no project CLAUDE.md/skills
-    }
   }
 
   const proc = spawn(CLAUDE, cliArgs, spawnOpts);
@@ -1008,6 +1061,11 @@ function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlo
     // and cleanup() is guarded by `cleaned`, so the slot is released exactly once on the first
     // exit path reached (proc 'exit' fires before 'close'; 'error' covers spawn failure).
     try { releaseSlot(); } catch { /* never let release throw out of cleanup */ }
+    // F3: release the real-HOME fallback serialization mutex (no-op for isolated/normal spawns).
+    // By now this spawn's claude has had its lifetime to refresh the keychain token, so the next
+    // queued fallback waiter re-checks resolveSpawnToken() and proceeds ISOLATED with the now-fresh
+    // token instead of piling into the real HOME. Idempotent; cleanup() is guarded by `cleaned`.
+    try { if (decision.releaseFallback) decision.releaseFallback(); } catch { /* never throw out of cleanup */ }
   }
 
   // Guarantee slot release on ANY exit path (normal close, error, timeout kill,
@@ -1083,12 +1141,16 @@ async function callClaude(model, messages, conversationId, keyName) {
   // spawn so the idempotent cleanup() frees it on every exit path. If the spawn itself throws
   // synchronously (before cleanup is wired), release here so the slot never leaks.
   const releaseSlot = await acquireClaudeSlot();
+  // F3: resolve the per-spawn HOME/token decision (may serialize on the real-HOME fallback mutex).
+  const spawnDecision = await resolveSpawnDecision();
   return new Promise((resolve, reject) => {
     let ctx;
     try {
-      ctx = spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlot);
+      ctx = spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlot, spawnDecision);
     } catch (err) {
       releaseSlot();
+      // Spawn threw before cleanup() was wired → release the fallback mutex here so it never leaks.
+      try { spawnDecision.releaseFallback?.(); } catch { /* best effort */ }
       return reject(err);
     }
 
@@ -1274,11 +1336,15 @@ async function callClaudeStreaming(model, messages, conversationId, res, authInf
     return jsonResponse(res, 500, { error: { message: sanitizeError(err.message), type: "proxy_error" } });
   }
 
+  // F3: resolve the per-spawn HOME/token decision (may serialize on the real-HOME fallback mutex).
+  const spawnDecision = await resolveSpawnDecision();
   let ctx;
   try {
-    ctx = spawnClaudeProcess(model, messages, conversationId, authInfo.keyName, releaseSlot);
+    ctx = spawnClaudeProcess(model, messages, conversationId, authInfo.keyName, releaseSlot, spawnDecision);
   } catch (err) {
     releaseSlot();
+    // Spawn threw before cleanup() was wired → release the fallback mutex here so it never leaks.
+    try { spawnDecision.releaseFallback?.(); } catch { /* best effort */ }
     return jsonResponse(res, 500, { error: { message: sanitizeError(err.message), type: "proxy_error" } });
   }
 
@@ -1550,6 +1616,51 @@ const OAUTH_REFRESH_MIN_BACKOFF = 60 * 1000;
 const OAUTH_REFRESH_MAX_BACKOFF = 3600 * 1000;
 let oauthRefreshBackoff = { nextAttemptAt: 0, currentDelay: OAUTH_REFRESH_MIN_BACKOFF };
 
+// FIX F5 (2026-07-07): the macOS keychain read (`security find-generic-password`, up to 5s × 2
+// labels when the first label misses) ran on EVERY -p spawn's hot path, blocking the event loop
+// (worst case 10s) and stalling all in-flight SSE streams. Two minimal, sync-preserving mitigations:
+//   (a) memoize the last-good keychain label and try it FIRST → one exec instead of two on the
+//       steady-state path (orderLabelsLastGoodFirst);
+//   (b) a short (30s) TTL cache of the keychain read result (createTtlCache).
+// SAFETY vs the #146 regression: #146 was a token memoized FOREVER at startup that went stale and
+// 401'd. This is a 30s TTL (not forever), AND resolveSpawnToken() re-applies the 5-min expiry gate
+// (isTokenExpiring) to the CACHED creds on EVERY use — the creds object carries `expiresAt`, so a
+// token expiring within the cache window is still rejected → real-HOME fallback. A short TTL bounds
+// how often we re-READ the keychain; it does NOT bound how often we re-DECIDE expiry. This is why a
+// short-TTL keychain cache + a per-use expiry check does not reintroduce the forever-stale bug.
+const KEYCHAIN_LABELS = ["claude-code-credentials", "Claude Code-credentials"];
+const KEYCHAIN_CACHE_TTL_MS = 30 * 1000;
+const _keychainCache = createTtlCache({ ttlMs: KEYCHAIN_CACHE_TTL_MS });
+let _lastGoodKeychainLabel = null;
+
+// Read the macOS keychain credentials, label-memoized + short-TTL cached (F5). Sync (execFileSync);
+// returns the `claudeAiOauth` creds object or null.
+function readKeychainCreds() {
+  return _keychainCache.get(() => {
+    for (const label of orderLabelsLastGoodFirst(KEYCHAIN_LABELS, _lastGoodKeychainLabel)) {
+      try {
+        const raw = execFileSync("security", [
+          "find-generic-password", "-s", label, "-w"
+        ], { encoding: "utf8", timeout: 5000 }).trim();
+        const creds = JSON.parse(raw);
+        if (creds?.claudeAiOauth?.accessToken) {
+          _lastGoodKeychainLabel = label; // remember the winner → try it first next time
+          return creds.claudeAiOauth;
+        }
+      } catch { /* try next label */ }
+    }
+    return null;
+  });
+}
+
+// F3 drain helper: drop the F5 keychain TTL cache so the NEXT getOAuthCredentials() re-reads the
+// keychain from scratch. Called under the real-HOME fallback mutex just before the re-check, so a
+// waiter admitted after the prior holder's claude refreshed the keychain sees the FRESH token
+// immediately (and proceeds ISOLATED) instead of waiting out the ≤30s TTL on the stale creds.
+function invalidateKeychainReadCache() {
+  _keychainCache.clear();
+}
+
 function getOAuthCredentials() {
   // 1. Env var fallback — highest precedence for explicit overrides.
   if (process.env.CLAUDE_CODE_OAUTH_TOKEN) {
@@ -1563,17 +1674,8 @@ function getOAuthCredentials() {
     if (creds?.claudeAiOauth?.accessToken) return creds.claudeAiOauth;
   } catch { /* fall through to macOS keychain */ }
 
-  // 3. macOS keychain (both label formats)
+  // 3. macOS keychain (both label formats) — F5: label-memoized + 30s TTL cached (see above).
-  for (const label of ["claude-code-credentials", "Claude Code-credentials"]) {
+  return readKeychainCreds();
-    try {
-      const raw = execFileSync("security", [
-        "find-generic-password", "-s", label, "-w"
-      ], { encoding: "utf8", timeout: 5000 }).trim();
-      const creds = JSON.parse(raw);
-      if (creds?.claudeAiOauth?.accessToken) return creds.claudeAiOauth;
-    } catch { /* try next */ }
-  }
-  return null;
 }
 
 async function refreshOAuthToken(refreshToken) {
@@ -2306,11 +2408,22 @@ const server = createServer(async (req, res) => {
       spawn: (() => {
         if (TUI_MODE) return { mode: "tui (default -p path unused)", isolated: false, home: null };
         const shm = getSpawnHomeMode();
+        // FIX F6: report the EFFECTIVE current decision, not just token PRESENCE. During the
+        // 5-min pre-expiry window the token exists (shm.isolated=true) but resolveSpawnToken()
+        // returns null and spawns actually run real-HOME — so `isolated` MUST also reflect the
+        // expiry gate, or /health lies. The field SET is unchanged (grandfathered B.2 contract,
+        // ADR 0006 — HARD CONSTRAINT: no field add/remove/rename); only the VALUES are made
+        // truthful. resolveSpawnToken() is read-only + backed by F5's 30s keychain cache → cheap.
+        const effIsolated = shm.isolated && resolveSpawnToken() !== null;
         return {
-          mode: shm.isolated ? "isolated-scratch-home" : "real-home",
+          mode: effIsolated ? "isolated-scratch-home" : "real-home",
-          isolated: shm.isolated,
+          isolated: effIsolated,
-          home: shm.isolated ? shm.home : null,
+          home: effIsolated ? shm.home : null,
-          reason: shm.reason,
+          reason: effIsolated
+            ? shm.reason
+            : (shm.isolated
+                ? "oauth token within 5-min expiry window → real-HOME fallback (self-heals on next refresh)"
+                : shm.reason),
         };
       })(),
       // ── FIX ⑥ -p concurrency wait-queue surface — ADDITIVE ──

test-features.mjs

@@ -5,6 +5,7 @@
  */
 import { getDb, createKey, listKeys, validateKey, recordUsage, checkQuota, updateKeyQuota, getKeyQuota, findKey, cacheHash, getCachedResponse, setCachedResponse, clearCache, getCacheStats, closeDb, hasCacheControl, singleflight, getInflightStats } from "./keys.mjs";
 import { isLoopbackBind } from "./lib/net.mjs";
+import { createSerialMutex, createTtlCache, isTokenExpiring, orderLabelsLastGoodFirst } from "./lib/spawn-auth.mjs";
 import { createHash } from "node:crypto";
 import { strict as assert } from "node:assert";
 import { unlinkSync } from "node:fs";
@@ -33,6 +34,17 @@ function test(name, fn) {
   }
 }
 
+async function testAsync(name, fn) {
+  try {
+    await fn();
+    passed++;
+    console.log(`  ✓ ${name}`);
+  } catch (e) {
+    failed++;
+    console.log(`  ✗ ${name}: ${e.message}`);
+  }
+}
+
 console.log("\n=== OCP Feature Tests (Quota + Cache) ===\n");
 
 // Initialize DB
@@ -2478,8 +2490,131 @@ test("isLoopbackBind: '100.64.0.1' → false (Tailscale IP)", () => {
   assert.equal(isLoopbackBind("100.64.0.1"), false);
 });
 
-// ── Cleanup ──
+// ── Spawn-auth primitives (F3 / F5 / F6, lib/spawn-auth.mjs) ──
-closeDb();
+// Pure, dependency-injected primitives extracted from server.mjs so the spawn-token concurrency /
+// caching / expiry logic is testable without booting the server or mocking execFileSync/spawn.
+console.log("\nSpawn-auth (F3 mutex / F5 TTL cache + label memo / F6 expiry gate):");
 
-console.log(`\n=== Results: ${passed} passed, ${failed} failed ===\n`);
+// F5: expiry gate — the load-bearing invariant that lets a short-TTL keychain cache stay safe.
-process.exit(failed > 0 ? 1 : 0);
+test("isTokenExpiring: creds within 5-min buffer → true", () => {
+  assert.equal(isTokenExpiring({ expiresAt: 1000 }, 1000 - 300000, 300000), true); // exactly at buffer edge
+  assert.equal(isTokenExpiring({ expiresAt: 1000 }, 900, 300000), true);           // past the edge
+});
+test("isTokenExpiring: creds well beyond buffer → false", () => {
+  assert.equal(isTokenExpiring({ expiresAt: 10_000_000 }, 0, 300000), false);
+});
+test("isTokenExpiring: no expiresAt (long-lived env token) → never expiring", () => {
+  assert.equal(isTokenExpiring({ accessToken: "x" }, Date.now(), 300000), false);
+  assert.equal(isTokenExpiring(null, Date.now(), 300000), false);
+});
+
+// F5: last-good label ordering — one exec instead of two on the steady-state keychain path.
+test("orderLabelsLastGoodFirst: last-good label is tried first", () => {
+  const labels = ["A", "B"];
+  assert.deepEqual(orderLabelsLastGoodFirst(labels, "B"), ["B", "A"]);
+});
+test("orderLabelsLastGoodFirst: null/unknown last-good → original order, fresh array", () => {
+  const labels = ["A", "B"];
+  assert.deepEqual(orderLabelsLastGoodFirst(labels, null), ["A", "B"]);
+  assert.deepEqual(orderLabelsLastGoodFirst(labels, "Z"), ["A", "B"]);
+  assert.notEqual(orderLabelsLastGoodFirst(labels, null), labels); // does not mutate/alias input
+});
+
+// F5: TTL cache — bounds how often we RE-READ the keychain (not how often we re-decide expiry).
+test("createTtlCache: serves cached value within TTL, re-produces after TTL", () => {
+  const cache = createTtlCache({ ttlMs: 30000 });
+  let calls = 0;
+  const produce = () => { calls++; return `v${calls}`; };
+  assert.equal(cache.get(produce, 0), "v1");
+  assert.equal(cache.get(produce, 10000), "v1"); // within TTL → cached, producer NOT called
+  assert.equal(calls, 1);
+  assert.equal(cache.get(produce, 40000), "v2"); // past TTL → re-produced
+  assert.equal(calls, 2);
+});
+test("createTtlCache: caches a null miss (absent source not re-probed within TTL)", () => {
+  const cache = createTtlCache({ ttlMs: 30000 });
+  let calls = 0;
+  const produce = () => { calls++; return null; };
+  assert.equal(cache.get(produce, 0), null);
+  assert.equal(cache.get(produce, 5000), null);
+  assert.equal(calls, 1); // the null was cached, not re-probed
+});
+
+// F5 core safety property: a short-TTL cache CANNOT reintroduce the #146 forever-stale bug because
+// the expiry gate is applied to the CACHED creds on every use. The cache keeps returning the same
+// creds object, but isTokenExpiring flips to true the moment the clock crosses the expiry buffer.
+test("TTL cache respects expiry gate: cached creds still rejected once clock passes expiry", () => {
+  const cache = createTtlCache({ ttlMs: 30000 });
+  const creds = { accessToken: "tok", expiresAt: 1_000_000 };
+  // t=980_000: cached AND not yet within the 5-min (300_000) buffer → usable.
+  const c1 = cache.get(() => creds, 980_000 - 300_000 - 1);
+  assert.equal(isTokenExpiring(c1, 980_000 - 300_000 - 1, 300000), false);
+  // t=800_000 later: SAME cached object returned (within TTL of the second read window), but now
+  // within the expiry buffer → gate rejects it → caller falls back to real HOME. No forever-stale.
+  const c2 = cache.get(() => creds, 990_000);
+  assert.equal(c2, c1, "cache returns the same creds object");
+  assert.equal(isTokenExpiring(c2, 990_000, 300000), true, "expiry gate still fires on cached creds");
+});
+
+// ── Async: F3 real-HOME fallback serialization mutex ──
+async function runAsyncTests() {
+  await testAsync("createSerialMutex: second waiter blocks until first holder releases", async () => {
+    const mutex = createSerialMutex();
+    const order = [];
+    const rel1 = await mutex.acquire();
+    order.push("h1-enter");
+    let secondEntered = false;
+    const p2 = mutex.acquire().then((rel2) => { secondEntered = true; order.push("h2-enter"); return rel2; });
+    await new Promise((r) => setTimeout(r, 15));
+    assert.equal(secondEntered, false, "second waiter must NOT enter while first holds the mutex");
+    order.push("h1-release");
+    rel1();
+    const rel2 = await p2;
+    assert.equal(secondEntered, true, "second waiter enters only after release");
+    rel2();
+    assert.deepEqual(order, ["h1-enter", "h1-release", "h2-enter"]);
+  });
+
+  await testAsync("createSerialMutex: N acquires run strictly in FIFO order, never overlapping", async () => {
+    const mutex = createSerialMutex();
+    const events = [];
+    let active = 0;
+    async function critical(id) {
+      const rel = await mutex.acquire();
+      active++;
+      assert.equal(active, 1, `only one holder at a time (id=${id})`);
+      events.push(`start${id}`);
+      await new Promise((r) => setTimeout(r, 5));
+      events.push(`end${id}`);
+      active--;
+      rel();
+    }
+    await Promise.all([critical(1), critical(2), critical(3)]);
+    assert.deepEqual(events, ["start1", "end1", "start2", "end2", "start3", "end3"]);
+  });
+
+  await testAsync("createSerialMutex: release() is idempotent (double-release does not double-admit)", async () => {
+    const mutex = createSerialMutex();
+    const rel1 = await mutex.acquire();
+    rel1();
+    rel1(); // second call must be a no-op
+    const rel2 = await mutex.acquire(); // should acquire cleanly, exactly once
+    let thirdEntered = false;
+    const p3 = mutex.acquire().then((r) => { thirdEntered = true; return r; });
+    await new Promise((r) => setTimeout(r, 15));
+    assert.equal(thirdEntered, false, "double-release must not have leaked an extra admit slot");
+    rel2();
+    (await p3)();
+  });
+}
+
+// ── Cleanup ──
+runAsyncTests().then(() => {
+  closeDb();
+  console.log(`\n=== Results: ${passed} passed, ${failed} failed ===\n`);
+  process.exit(failed > 0 ? 1 : 0);
+}).catch((e) => {
+  console.error("async test runner crashed:", e);
+  closeDb();
+  process.exit(1);
+});