fix(server): serialize -p real-HOME token fallback + TTL-cache keychain + de-stale isolation decision (F3/F5/F6) (#150)

* fix(server): serialize -p real-HOME token fallback + TTL-cache keychain + de-stale isolation decision (F3/F5/F6) Three audit findings in the -p spawn-token resolution + HOME-isolation layer. All are infra/ process changes to how OCP READS and GATES an OAuth token it already holds; none touch the OAuth wire machinery. F3 (MEDIUM) — expiry-window fallback herds concurrent -p spawns into real HOME. When the keychain token is within 5 min of expiry, resolveSpawnToken() returns null and every concurrent spawn simultaneously falls back to the real HOME; each spawned claude then races a refresh_token grant against the SAME single-use refresh token — rotating it out from under the others and the operator's real claude (the credential-fork hazard, #112/#146 class). Fix: a promise-chain mutex (createSerialMutex) serializes ONLY the real-HOME fallback — one such spawn at a time. When a serialized waiter is admitted (prior holder torn down → its claude has refreshed the keychain), it re-runs resolveSpawnToken(): a now-fresh token means it proceeds ISOLATED instead of real-HOME, so the queue drains to the fast path. Isolated spawns never touch the mutex. F5 (LOW-MED) — per-spawn double keychain exec on the hot path. getOAuthCredentials() sync-exec'd `security find-generic-password` up to twice (wrong label first), worst case 5s×2, blocking the event loop and stalling in-flight SSE streams. Fix: (a) memoize the last-good keychain label and try it first (orderLabelsLastGoodFirst); (b) a 30s TTL cache of the read (createTtlCache). This does NOT reintroduce the #146 forever-memoized regression: the TTL bounds only how often we re-READ the keychain; resolveSpawnToken() still applies the 5-min expiry gate (isTokenExpiring) to the CACHED creds on EVERY use, so a token expiring within the window is still rejected → real-HOME fallback. Call sites stay synchronous (no async conversion). F6 (LOW) — memoized isolation decision goes stale; /health could misreport. getSpawnHomeMode() memoized the isolated/real-home decision forever: credentials appearing after startup never enabled isolation; deleting ~/.ocp/spawn-home at runtime ENOENT'd every isolated spawn until restart; during an expiry stint /health reported isolated:true while spawns ran real- HOME. Fix: re-evaluate the decision per spawn (cheap now that F5 caches the keychain read); ensureSpawnHome() re-verifies + re-prepares the scratch dir per isolated spawn; and /health now reports the EFFECTIVE decision (token presence AND expiry gate). The /health field set is UNCHANGED — no field added/removed/renamed — only the values are made truthful. Alignment: - Class: Not a wire/endpoint change for the spawn-token layer + Class B (B.2) for /health. - cli.js citation: DECLARED ABSENT. cli.js does NOT perform OCP's spawn-token resolution, HOME isolation, keychain caching, or fallback serialization — these are proxy-internal process concerns with no cli.js analogue, so no Class A cli.js:NNNN citation exists or is required (ALIGNMENT.md Rule 2: this is proxy infra, not an invented forwarded endpoint/header/body). - The /health change is authorized by ADR 0006 (grandfathered B.2 as of v3.16.4) and is a behaviour-preserving contract change: same fields, truthful values. - OCP still NEVER performs a refresh_token grant itself — that property is preserved; the fix only serializes/gates reads of a token refreshed by the spawned or real claude (#112). - No new endpoint, header, or request/response field. alignment.yml blacklist unaffected. Tests: extracted the pure primitives to lib/spawn-auth.mjs and added 11 unit tests (mutex serialization order + idempotent release; TTL cache freshness + null-miss; expiry gate; label ordering; and the combined invariant that the TTL cache respects the expiry gate). node --check clean; 249 tests pass (238 + 11). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(server): drain F3 fallback queue immediately — invalidate F5 keychain cache before the serialized re-check Follow-up to the F3/F5/F6 fix (independent-review observation). F5's 30s keychain TTL cache could make F3's post-refresh re-check see the stale (expiring) cached creds for up to ~30s, so a waiter admitted right after the prior real-HOME holder's claude refreshed the keychain would needlessly fall back to real HOME again instead of proceeding ISOLATED. Serialization safety was never at risk (still one real-HOME spawn at a time, no double-refresh); only the drain-to-fast-path optimization lagged. Fix: invalidateKeychainReadCache() clears the F5 TTL cache; resolveSpawnDecision() calls it under the fallback mutex, immediately before the re-check, so the admitted waiter reads FRESH keychain state and drains to the isolated fast path at once. The extra keychain read happens only on the rare real-HOME fallback path and only under the mutex (serialized, one at a time). Alignment: unchanged from the parent commit — proxy-internal keychain/HOME-isolation process logic, no cli.js analogue (cli.js citation DECLARED ABSENT, ALIGNMENT.md Rule 2), no endpoint/header/body, /health shape unchanged, OCP still never performs a refresh_token grant itself (#112). 249 tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: dtzp555 <dtzp555@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 09:44:07 +00:00 · 2026-07-07 22:50:52 +10:00
parent 31e5a44099
commit 2538233059
3 changed files with 383 additions and 68 deletions
@@ -0,0 +1,67 @@
+// Pure, dependency-injected primitives for the `-p` spawn-token resolution + HOME-isolation
+// layer. Extracted from server.mjs (findings F3 / F5 / F6, 2026-07-07) so the concurrency,
+// caching and expiry logic is unit-testable WITHOUT booting the server or mocking execFileSync /
+// child_process.spawn / fs. server.mjs owns all I/O (macOS keychain exec, process spawn, fs);
+// this module owns only pure decision logic.
+//
+// ALIGNMENT NOTE: none of this touches the OAuth wire machinery (no endpoint / header / body).
+// OCP still NEVER performs a refresh_token grant itself — these helpers only READ + GATE a token
+// that some other process (the operator's real claude, or a spawned claude under the real HOME)
+// refreshes. That property is load-bearing (issue #112) and preserved.
+
+// Promise-chain mutex. `acquire()` resolves to a `release()` fn; the NEXT `acquire()` does not
+// resolve until the current holder calls its `release()`. Serializes async critical sections
+// without busy-waiting. release() is idempotent.
+export function createSerialMutex() {
+  let tail = Promise.resolve();
+  return {
+    acquire() {
+      let release;
+      const gate = new Promise((r) => { release = r; });
+      const prev = tail;
+      tail = tail.then(() => gate);
+      // Hand the caller its release fn only after the previous holder has released.
+      return prev.then(() => {
+        let released = false;
+        return function releaseMutex() { if (!released) { released = true; release(); } };
+      });
+    },
+  };
+}
+
+// Short-TTL memo. `get(produce, now)` returns the cached value while `now - storedAt < ttlMs`,
+// otherwise calls `produce()` and re-stores. A miss that produces null/undefined is STILL stored
+// (so a genuinely-absent source is not re-probed on every call within the TTL window). `now` is
+// injectable for testing.
+export function createTtlCache({ ttlMs }) {
+  let value;
+  let at = -Infinity;
+  let has = false;
+  return {
+    get(produce, now = Date.now()) {
+      if (has && now - at < ttlMs) return value;
+      value = produce();
+      at = now;
+      has = true;
+      return value;
+    },
+    clear() { has = false; value = undefined; at = -Infinity; },
+  };
+}
+
+// Pure expiry gate. Returns true when `creds` carries a known expiry that is at/within `bufferMs`
+// of `now`. Creds WITHOUT `expiresAt` (e.g. long-lived env tokens) are never treated as expiring.
+// This gate is applied to the CACHED creds on EVERY use — which is precisely why a short-TTL
+// keychain cache (createTtlCache) cannot reintroduce the #146 forever-stale-token regression: the
+// cache bounds how often we re-READ the keychain, but the expiry decision is recomputed per use.
+export function isTokenExpiring(creds, now = Date.now(), bufferMs = 300000) {
+  return !!(creds && creds.expiresAt && now + bufferMs >= creds.expiresAt);
+}
+
+// Order candidate keychain labels so the last-known-good label is tried first (avoids the
+// wrong-label miss that doubles the `security` exec count on the hot path). Pure: performs no
+// read. Returns a fresh array; input is not mutated.
+export function orderLabelsLastGoodFirst(labels, lastGood) {
+  if (!lastGood || !labels.includes(lastGood)) return labels.slice();
+  return [lastGood, ...labels.filter((l) => l !== lastGood)];
+}
@@ -44,6 +44,7 @@ import { isLoopbackBind } from "./lib/net.mjs";
 import { runTuiTurn, reapStaleTuiSessions, resolveTuiHome } from "./lib/tui/session.mjs";
 import { detectTuiUpstreamError } from "./lib/tui/transcript.mjs";
 import { TuiSemaphore, recordTuiEntrypoint, buildTuiHealthBlock } from "./lib/tui/semaphore.mjs";
+import { createSerialMutex, createTtlCache, isTokenExpiring, orderLabelsLastGoodFirst } from "./lib/spawn-auth.mjs";

 const __dirname = dirname(fileURLToPath(import.meta.url));
 const _pkg = JSON.parse(readFileSync(join(__dirname, "package.json"), "utf8"));
@@ -387,50 +388,106 @@ function prepareSpawnHome(dir = SPAWN_HOME_DIR) {
  } catch { /* best effort — spawn will surface a hard error if the dir is truly unusable */ }
 }

-// Resolve the default-spawn HOME-isolation decision ONCE, lazily + memoized (so it runs after
-// getOAuthCredentials is defined regardless of source order, and the token probe happens at most
-// once). Returns { isolated, home, token } where:
+// Resolve the default-spawn HOME-isolation decision. Returns { isolated, home, reason }:
 //   - isolated:true  → spawn under SPAWN_HOME_DIR with cwd=SPAWN_HOME_DIR + the env token.
 //   - isolated:false → legacy real-HOME spawn, no cwd override (no token, or kill-switch on).
-// Caches only the isolation DECISION (isolated/home/reason), NOT the token — the token is
-// re-resolved FRESH per spawn via resolveSpawnToken(). A memoized token goes stale when its
-// source rotates: the macOS keychain access token rotates (~hourly, refreshed by the operator's
-// real claude), so a startup snapshot 401s once it expires (caused a ~31h Mac-mini 401 outage,
-// 2026-06-26). OCP deliberately does NOT refresh the token itself — a refresh-token grant would
-// consume the single-use refresh token and log out the operator's real claude (issue #112).
-let _spawnHomeMode = null;
+//
+// FIX F6 (2026-07-07): this decision is NO LONGER memoized permanently. The previous version
+// cached it forever at first call, which meant: (a) credentials appearing after startup never
+// enabled isolation; (b) `rm -rf ~/.ocp/spawn-home` at runtime made every isolated spawn ENOENT
+// until restart; (c) during a token-expiry stint /health reported isolated:true while spawns
+// actually ran real-HOME. Re-evaluating per spawn is cheap because F5's 30s keychain TTL cache
+// backs getOAuthCredentials(). This function is the CONFIG-level decision (isolated iff a token
+// resolves AND the kill-switch is off) and has NO fs side effects — the per-spawn EFFECTIVE
+// decision additionally applies the expiry gate (resolveSpawnDecision), and scratch-HOME dir prep
+// moved to ensureSpawnHome() at the isolated spawn site.
+//
+// The token itself is re-resolved FRESH per spawn via resolveSpawnToken(); a memoized token goes
+// stale when its source rotates (the macOS keychain access token rotates ~hourly, refreshed by the
+// operator's real claude), which 401'd every isolated spawn for ~31h on 2026-06-26 (#146). OCP
+// deliberately does NOT refresh the token itself — a refresh-token grant would consume the
+// single-use refresh token and log out the operator's real claude (issue #112).
 function getSpawnHomeMode() {
-  if (_spawnHomeMode) return _spawnHomeMode;
  if (SPAWN_REAL_HOME) {
-    _spawnHomeMode = { isolated: false, home: null, reason: "kill-switch (OCP_SPAWN_REAL_HOME=1)" };
-    return _spawnHomeMode;
+    return { isolated: false, home: null, reason: "kill-switch (OCP_SPAWN_REAL_HOME=1)" };
  }
  let hasToken = false;
  try { hasToken = !!(getOAuthCredentials()?.accessToken); } catch { hasToken = false; }
-  if (hasToken) {
-    prepareSpawnHome(SPAWN_HOME_DIR);
-    _spawnHomeMode = { isolated: true, home: SPAWN_HOME_DIR, reason: "oauth token resolved" };
-  } else {
-    _spawnHomeMode = { isolated: false, home: null, reason: "no oauth token resolvable" };
-  }
-  return _spawnHomeMode;
+  if (hasToken) return { isolated: true, home: SPAWN_HOME_DIR, reason: "oauth token resolved" };
+  return { isolated: false, home: null, reason: "no oauth token resolvable" };
+}
+
+// FIX F6: re-verify the scratch HOME exists before each isolated spawn and re-create it if it was
+// deleted at runtime (it used to be prepared once at startup, so a runtime deletion made every
+// isolated spawn fail ENOENT until restart). mkdirSync is recursive+idempotent → cheap to re-run.
+function ensureSpawnHome(dir = SPAWN_HOME_DIR) {
+  if (!existsSync(`${dir}/.claude`)) prepareSpawnHome(dir);
 }

 // Resolve a FRESH OAuth access token for an isolated spawn. Read-only (keychain / credentials.json
 // / env) — NEVER refreshes/rotates (see getSpawnHomeMode note). Returns null if none resolvable OR
-// if a known expiry has passed (5-min buffer): a null return makes the caller fall back to real
-// HOME, where the spawned claude refreshes the credential natively and self-heals (the keychain
-// token is then fresh again → next spawn is fast). The env-token path (Linux) carries no expiresAt
-// → never expiry-gated (those tokens are long-lived).
+// if a known expiry is within the 5-min buffer (isTokenExpiring): a null return makes the caller
+// fall back to real HOME, where the spawned claude refreshes the credential natively and self-heals
+// (the keychain token is then fresh again → next spawn is fast). The env-token path (Linux) carries
+// no expiresAt → never expiry-gated (those tokens are long-lived).
 function resolveSpawnToken() {
  try {
    const creds = getOAuthCredentials();
    if (!creds?.accessToken) return null;
-    if (creds.expiresAt && Date.now() + 300000 >= creds.expiresAt) return null;
+    if (isTokenExpiring(creds)) return null; // 5-min buffer; applied to the CACHED creds every use
    return creds.accessToken;
  } catch { return null; }
 }

+// FIX F3 (2026-07-07): serializes ONLY the real-HOME fallback spawns. Isolated spawns (the common
+// fast path) never touch this mutex.
+const realHomeFallbackMutex = createSerialMutex();
+
+// Resolve the EFFECTIVE per-spawn HOME/token decision. Returns
+//   { isolated, home, token, releaseFallback }
+// `releaseFallback` is non-null ONLY for a real-HOME fallback holder — the caller MUST call it on
+// spawn teardown (wired into cleanup()); it releases the serialization mutex. It is null (no-op)
+// for isolated and stable real-HOME (kill-switch / no-token) spawns.
+//
+// This is async so the real-HOME fallback can `await` the mutex; the keychain reads inside stay
+// synchronous (F5 keeps the call sites off async conversion).
+async function resolveSpawnDecision() {
+  const shm = getSpawnHomeMode();
+  if (!shm.isolated) return { isolated: false, home: null, token: null, releaseFallback: null };
+  const token = resolveSpawnToken();
+  if (token) {
+    ensureSpawnHome(shm.home);
+    return { isolated: true, home: shm.home, token, releaseFallback: null };
+  }
+  // Token is present but within the 5-min expiry window → we would fall back to real HOME, where
+  // the spawned claude refreshes the credential natively. HAZARD PREVENTED: without serialization,
+  // every concurrent -p spawn inside this window runs claude under the real HOME simultaneously,
+  // and each spawned claude races a `refresh_token` grant against the SAME single-use refresh
+  // token — rotating it out from under the others AND the operator's own real claude (the
+  // credential-fork hazard; #112 / #146 class). Serialize: admit ONE real-HOME spawn at a time.
+  // When the next waiter is admitted (the prior holder torn down → its claude has had its lifetime
+  // to refresh the keychain), re-run resolveSpawnToken(): a now-fresh token means we proceed
+  // ISOLATED and release the mutex immediately, so the queue drains to the fast path instead of
+  // piling every request into the real HOME.
+  const release = await realHomeFallbackMutex.acquire();
+  try {
+    // Drop the 30s keychain TTL cache so the re-check reads FRESH keychain state — otherwise a
+    // waiter admitted right after the prior holder's claude refreshed the token could still see the
+    // stale (expiring) cached creds and needlessly fall back to real HOME again for up to ~30s.
+    invalidateKeychainReadCache();
+    const retry = resolveSpawnToken();
+    if (retry) {
+      release();
+      ensureSpawnHome(shm.home);
+      return { isolated: true, home: shm.home, token: retry, releaseFallback: null };
+    }
+  } catch (e) {
+    release();
+    throw e;
+  }
+  return { isolated: false, home: null, token: null, releaseFallback: release };
+}
+
 // ── FIX ⑥ (concurrency): bounded wait-queue for the -p / stream-json path ──────────────
 // PROBLEM (proven): spawnClaudeProcess used `if (activeRequests >= MAX_CONCURRENT) throw` →
 // the client got an opaque 500 AND the rejection was NOT counted in stats (a 15-concurrent
@@ -931,7 +988,7 @@ function getModelTier(cliModel) {
 // budget. releaseSlot is wired into the idempotent cleanup() so the slot is freed on EVERY exit
 // path (close/error/timeout/abort). Back-compat: releaseSlot defaults to a no-op so any future
 // internal caller that does its own gating still works.
-function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlot = () => {}) {
+function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlot = () => {}, spawnDecision = null) {
  const cliModel = MODEL_MAP[model] || model;

  // Circuit breaker: disabled (see comment at top of breaker section)
@@ -968,28 +1025,24 @@ function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlo
    env.CLAUDE_CODE_DISABLE_AUTO_MEMORY = "1";
  }

-  // FIX ③ (latency): default-path spawn-home isolation. When a token is resolvable (and the
-  // OCP_SPAWN_REAL_HOME kill-switch is off), run claude under a credential-free minimal HOME
-  // with cwd = that same neutral dir, so it loads NONE of the operator's global ~/.claude
-  // (plugins/skills/hooks) or the ~/ocp project CLAUDE.md/skills — the measured 10–28s → 3–7s
-  // latency win. The env token is authoritative for `-p` (unlike interactive claude). When no
-  // token is resolvable, falls back to real HOME + inherited cwd (zero regression). See
-  // getSpawnHomeMode() / prepareSpawnHome() above. The DISABLE_CLAUDE_MDS / AUTO_MEMORY flags
-  // are set unconditionally in isolated mode (belt-and-braces; mirrors the TUI path).
-  const spawnHome = getSpawnHomeMode();
+  // FIX ③ (latency) + F3 (concurrency): apply the pre-resolved per-spawn HOME/token decision.
+  // The decision is resolved ASYNC in the caller (resolveSpawnDecision) so the real-HOME fallback
+  // serialization can await its mutex; here we only apply the result. When isolated, run claude
+  // under a credential-free minimal HOME with cwd = that same neutral dir, so it loads NONE of the
+  // operator's global ~/.claude (plugins/skills/hooks) or the ~/ocp project CLAUDE.md/skills — the
+  // measured 10–28s → 3–7s latency win. The env token is authoritative for `-p` (unlike
+  // interactive claude). When no fresh token is resolvable, decision.isolated is false → real HOME
+  // + inherited cwd (zero regression), and the spawned claude resolves+refreshes credentials
+  // natively. The DISABLE_CLAUDE_MDS / AUTO_MEMORY flags are set unconditionally in isolated mode
+  // (belt-and-braces; mirrors the TUI path).
+  const decision = spawnDecision || { isolated: false, releaseFallback: null };
  const spawnOpts = { env, stdio: ["pipe", "pipe", "pipe"] };
-  if (spawnHome.isolated) {
-    // Re-resolve the token FRESH per spawn (never a startup snapshot — keychain tokens rotate;
-    // a stale snapshot 401s). If unresolvable right now, fall through to real HOME so the spawned
-    // claude resolves + refreshes credentials natively instead of 401ing on a stale/null token.
-    const freshToken = resolveSpawnToken();
-    if (freshToken) {
-      env.HOME = spawnHome.home;
-      env.CLAUDE_CODE_OAUTH_TOKEN = freshToken; // env token is authoritative for -p
+  if (decision.isolated && decision.token) {
+    env.HOME = decision.home;
+    env.CLAUDE_CODE_OAUTH_TOKEN = decision.token; // env token is authoritative for -p
    env.CLAUDE_CODE_DISABLE_CLAUDE_MDS = "1";
    env.CLAUDE_CODE_DISABLE_AUTO_MEMORY = "1";
-      spawnOpts.cwd = spawnHome.home; // neutral cwd: no project CLAUDE.md/skills
-    }
+    spawnOpts.cwd = decision.home; // neutral cwd: no project CLAUDE.md/skills
  }

  const proc = spawn(CLAUDE, cliArgs, spawnOpts);
@@ -1008,6 +1061,11 @@ function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlo
    // and cleanup() is guarded by `cleaned`, so the slot is released exactly once on the first
    // exit path reached (proc 'exit' fires before 'close'; 'error' covers spawn failure).
    try { releaseSlot(); } catch { /* never let release throw out of cleanup */ }
+    // F3: release the real-HOME fallback serialization mutex (no-op for isolated/normal spawns).
+    // By now this spawn's claude has had its lifetime to refresh the keychain token, so the next
+    // queued fallback waiter re-checks resolveSpawnToken() and proceeds ISOLATED with the now-fresh
+    // token instead of piling into the real HOME. Idempotent; cleanup() is guarded by `cleaned`.
+    try { if (decision.releaseFallback) decision.releaseFallback(); } catch { /* never throw out of cleanup */ }
  }

  // Guarantee slot release on ANY exit path (normal close, error, timeout kill,
@@ -1083,12 +1141,16 @@ async function callClaude(model, messages, conversationId, keyName) {
  // spawn so the idempotent cleanup() frees it on every exit path. If the spawn itself throws
  // synchronously (before cleanup is wired), release here so the slot never leaks.
  const releaseSlot = await acquireClaudeSlot();
+  // F3: resolve the per-spawn HOME/token decision (may serialize on the real-HOME fallback mutex).
+  const spawnDecision = await resolveSpawnDecision();
  return new Promise((resolve, reject) => {
    let ctx;
    try {
-      ctx = spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlot);
+      ctx = spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlot, spawnDecision);
    } catch (err) {
      releaseSlot();
+      // Spawn threw before cleanup() was wired → release the fallback mutex here so it never leaks.
+      try { spawnDecision.releaseFallback?.(); } catch { /* best effort */ }
      return reject(err);
    }

@@ -1274,11 +1336,15 @@ async function callClaudeStreaming(model, messages, conversationId, res, authInf
    return jsonResponse(res, 500, { error: { message: sanitizeError(err.message), type: "proxy_error" } });
  }

+  // F3: resolve the per-spawn HOME/token decision (may serialize on the real-HOME fallback mutex).
+  const spawnDecision = await resolveSpawnDecision();
  let ctx;
  try {
-    ctx = spawnClaudeProcess(model, messages, conversationId, authInfo.keyName, releaseSlot);
+    ctx = spawnClaudeProcess(model, messages, conversationId, authInfo.keyName, releaseSlot, spawnDecision);
  } catch (err) {
    releaseSlot();
+    // Spawn threw before cleanup() was wired → release the fallback mutex here so it never leaks.
+    try { spawnDecision.releaseFallback?.(); } catch { /* best effort */ }
    return jsonResponse(res, 500, { error: { message: sanitizeError(err.message), type: "proxy_error" } });
  }

@@ -1550,6 +1616,51 @@ const OAUTH_REFRESH_MIN_BACKOFF = 60 * 1000;
 const OAUTH_REFRESH_MAX_BACKOFF = 3600 * 1000;
 let oauthRefreshBackoff = { nextAttemptAt: 0, currentDelay: OAUTH_REFRESH_MIN_BACKOFF };

+// FIX F5 (2026-07-07): the macOS keychain read (`security find-generic-password`, up to 5s × 2
+// labels when the first label misses) ran on EVERY -p spawn's hot path, blocking the event loop
+// (worst case 10s) and stalling all in-flight SSE streams. Two minimal, sync-preserving mitigations:
+//   (a) memoize the last-good keychain label and try it FIRST → one exec instead of two on the
+//       steady-state path (orderLabelsLastGoodFirst);
+//   (b) a short (30s) TTL cache of the keychain read result (createTtlCache).
+// SAFETY vs the #146 regression: #146 was a token memoized FOREVER at startup that went stale and
+// 401'd. This is a 30s TTL (not forever), AND resolveSpawnToken() re-applies the 5-min expiry gate
+// (isTokenExpiring) to the CACHED creds on EVERY use — the creds object carries `expiresAt`, so a
+// token expiring within the cache window is still rejected → real-HOME fallback. A short TTL bounds
+// how often we re-READ the keychain; it does NOT bound how often we re-DECIDE expiry. This is why a
+// short-TTL keychain cache + a per-use expiry check does not reintroduce the forever-stale bug.
+const KEYCHAIN_LABELS = ["claude-code-credentials", "Claude Code-credentials"];
+const KEYCHAIN_CACHE_TTL_MS = 30 * 1000;
+const _keychainCache = createTtlCache({ ttlMs: KEYCHAIN_CACHE_TTL_MS });
+let _lastGoodKeychainLabel = null;
+
+// Read the macOS keychain credentials, label-memoized + short-TTL cached (F5). Sync (execFileSync);
+// returns the `claudeAiOauth` creds object or null.
+function readKeychainCreds() {
+  return _keychainCache.get(() => {
+    for (const label of orderLabelsLastGoodFirst(KEYCHAIN_LABELS, _lastGoodKeychainLabel)) {
+      try {
+        const raw = execFileSync("security", [
+          "find-generic-password", "-s", label, "-w"
+        ], { encoding: "utf8", timeout: 5000 }).trim();
+        const creds = JSON.parse(raw);
+        if (creds?.claudeAiOauth?.accessToken) {
+          _lastGoodKeychainLabel = label; // remember the winner → try it first next time
+          return creds.claudeAiOauth;
+        }
+      } catch { /* try next label */ }
+    }
+    return null;
+  });
+}
+
+// F3 drain helper: drop the F5 keychain TTL cache so the NEXT getOAuthCredentials() re-reads the
+// keychain from scratch. Called under the real-HOME fallback mutex just before the re-check, so a
+// waiter admitted after the prior holder's claude refreshed the keychain sees the FRESH token
+// immediately (and proceeds ISOLATED) instead of waiting out the ≤30s TTL on the stale creds.
+function invalidateKeychainReadCache() {
+  _keychainCache.clear();
+}
+
 function getOAuthCredentials() {
  // 1. Env var fallback — highest precedence for explicit overrides.
  if (process.env.CLAUDE_CODE_OAUTH_TOKEN) {
@@ -1563,17 +1674,8 @@ function getOAuthCredentials() {
    if (creds?.claudeAiOauth?.accessToken) return creds.claudeAiOauth;
  } catch { /* fall through to macOS keychain */ }

-  // 3. macOS keychain (both label formats)
-  for (const label of ["claude-code-credentials", "Claude Code-credentials"]) {
-    try {
-      const raw = execFileSync("security", [
-        "find-generic-password", "-s", label, "-w"
-      ], { encoding: "utf8", timeout: 5000 }).trim();
-      const creds = JSON.parse(raw);
-      if (creds?.claudeAiOauth?.accessToken) return creds.claudeAiOauth;
-    } catch { /* try next */ }
-  }
-  return null;
+  // 3. macOS keychain (both label formats) — F5: label-memoized + 30s TTL cached (see above).
+  return readKeychainCreds();
 }

 async function refreshOAuthToken(refreshToken) {
@@ -2306,11 +2408,22 @@ const server = createServer(async (req, res) => {
      spawn: (() => {
        if (TUI_MODE) return { mode: "tui (default -p path unused)", isolated: false, home: null };
        const shm = getSpawnHomeMode();
+        // FIX F6: report the EFFECTIVE current decision, not just token PRESENCE. During the
+        // 5-min pre-expiry window the token exists (shm.isolated=true) but resolveSpawnToken()
+        // returns null and spawns actually run real-HOME — so `isolated` MUST also reflect the
+        // expiry gate, or /health lies. The field SET is unchanged (grandfathered B.2 contract,
+        // ADR 0006 — HARD CONSTRAINT: no field add/remove/rename); only the VALUES are made
+        // truthful. resolveSpawnToken() is read-only + backed by F5's 30s keychain cache → cheap.
+        const effIsolated = shm.isolated && resolveSpawnToken() !== null;
        return {
-          mode: shm.isolated ? "isolated-scratch-home" : "real-home",
-          isolated: shm.isolated,
-          home: shm.isolated ? shm.home : null,
-          reason: shm.reason,
+          mode: effIsolated ? "isolated-scratch-home" : "real-home",
+          isolated: effIsolated,
+          home: effIsolated ? shm.home : null,
+          reason: effIsolated
+            ? shm.reason
+            : (shm.isolated
+                ? "oauth token within 5-min expiry window → real-HOME fallback (self-heals on next refresh)"
+                : shm.reason),
        };
      })(),
      // ── FIX ⑥ -p concurrency wait-queue surface — ADDITIVE ──
@@ -5,6 +5,7 @@
 */
 import { getDb, createKey, listKeys, validateKey, recordUsage, checkQuota, updateKeyQuota, getKeyQuota, findKey, cacheHash, getCachedResponse, setCachedResponse, clearCache, getCacheStats, closeDb, hasCacheControl, singleflight, getInflightStats } from "./keys.mjs";
 import { isLoopbackBind } from "./lib/net.mjs";
+import { createSerialMutex, createTtlCache, isTokenExpiring, orderLabelsLastGoodFirst } from "./lib/spawn-auth.mjs";
 import { createHash } from "node:crypto";
 import { strict as assert } from "node:assert";
 import { unlinkSync } from "node:fs";
@@ -33,6 +34,17 @@ function test(name, fn) {
  }
 }

+async function testAsync(name, fn) {
+  try {
+    await fn();
+    passed++;
+    console.log(`  ✓ ${name}`);
+  } catch (e) {
+    failed++;
+    console.log(`  ✗ ${name}: ${e.message}`);
+  }
+}
+
 console.log("\n=== OCP Feature Tests (Quota + Cache) ===\n");

 // Initialize DB
@@ -2478,8 +2490,131 @@ test("isLoopbackBind: '100.64.0.1' → false (Tailscale IP)", () => {
  assert.equal(isLoopbackBind("100.64.0.1"), false);
 });

-// ── Cleanup ──
-closeDb();
+// ── Spawn-auth primitives (F3 / F5 / F6, lib/spawn-auth.mjs) ──
+// Pure, dependency-injected primitives extracted from server.mjs so the spawn-token concurrency /
+// caching / expiry logic is testable without booting the server or mocking execFileSync/spawn.
+console.log("\nSpawn-auth (F3 mutex / F5 TTL cache + label memo / F6 expiry gate):");

-console.log(`\n=== Results: ${passed} passed, ${failed} failed ===\n`);
-process.exit(failed > 0 ? 1 : 0);
+// F5: expiry gate — the load-bearing invariant that lets a short-TTL keychain cache stay safe.
+test("isTokenExpiring: creds within 5-min buffer → true", () => {
+  assert.equal(isTokenExpiring({ expiresAt: 1000 }, 1000 - 300000, 300000), true); // exactly at buffer edge
+  assert.equal(isTokenExpiring({ expiresAt: 1000 }, 900, 300000), true);           // past the edge
+});
+test("isTokenExpiring: creds well beyond buffer → false", () => {
+  assert.equal(isTokenExpiring({ expiresAt: 10_000_000 }, 0, 300000), false);
+});
+test("isTokenExpiring: no expiresAt (long-lived env token) → never expiring", () => {
+  assert.equal(isTokenExpiring({ accessToken: "x" }, Date.now(), 300000), false);
+  assert.equal(isTokenExpiring(null, Date.now(), 300000), false);
+});
+
+// F5: last-good label ordering — one exec instead of two on the steady-state keychain path.
+test("orderLabelsLastGoodFirst: last-good label is tried first", () => {
+  const labels = ["A", "B"];
+  assert.deepEqual(orderLabelsLastGoodFirst(labels, "B"), ["B", "A"]);
+});
+test("orderLabelsLastGoodFirst: null/unknown last-good → original order, fresh array", () => {
+  const labels = ["A", "B"];
+  assert.deepEqual(orderLabelsLastGoodFirst(labels, null), ["A", "B"]);
+  assert.deepEqual(orderLabelsLastGoodFirst(labels, "Z"), ["A", "B"]);
+  assert.notEqual(orderLabelsLastGoodFirst(labels, null), labels); // does not mutate/alias input
+});
+
+// F5: TTL cache — bounds how often we RE-READ the keychain (not how often we re-decide expiry).
+test("createTtlCache: serves cached value within TTL, re-produces after TTL", () => {
+  const cache = createTtlCache({ ttlMs: 30000 });
+  let calls = 0;
+  const produce = () => { calls++; return `v${calls}`; };
+  assert.equal(cache.get(produce, 0), "v1");
+  assert.equal(cache.get(produce, 10000), "v1"); // within TTL → cached, producer NOT called
+  assert.equal(calls, 1);
+  assert.equal(cache.get(produce, 40000), "v2"); // past TTL → re-produced
+  assert.equal(calls, 2);
+});
+test("createTtlCache: caches a null miss (absent source not re-probed within TTL)", () => {
+  const cache = createTtlCache({ ttlMs: 30000 });
+  let calls = 0;
+  const produce = () => { calls++; return null; };
+  assert.equal(cache.get(produce, 0), null);
+  assert.equal(cache.get(produce, 5000), null);
+  assert.equal(calls, 1); // the null was cached, not re-probed
+});
+
+// F5 core safety property: a short-TTL cache CANNOT reintroduce the #146 forever-stale bug because
+// the expiry gate is applied to the CACHED creds on every use. The cache keeps returning the same
+// creds object, but isTokenExpiring flips to true the moment the clock crosses the expiry buffer.
+test("TTL cache respects expiry gate: cached creds still rejected once clock passes expiry", () => {
+  const cache = createTtlCache({ ttlMs: 30000 });
+  const creds = { accessToken: "tok", expiresAt: 1_000_000 };
+  // t=980_000: cached AND not yet within the 5-min (300_000) buffer → usable.
+  const c1 = cache.get(() => creds, 980_000 - 300_000 - 1);
+  assert.equal(isTokenExpiring(c1, 980_000 - 300_000 - 1, 300000), false);
+  // t=800_000 later: SAME cached object returned (within TTL of the second read window), but now
+  // within the expiry buffer → gate rejects it → caller falls back to real HOME. No forever-stale.
+  const c2 = cache.get(() => creds, 990_000);
+  assert.equal(c2, c1, "cache returns the same creds object");
+  assert.equal(isTokenExpiring(c2, 990_000, 300000), true, "expiry gate still fires on cached creds");
+});
+
+// ── Async: F3 real-HOME fallback serialization mutex ──
+async function runAsyncTests() {
+  await testAsync("createSerialMutex: second waiter blocks until first holder releases", async () => {
+    const mutex = createSerialMutex();
+    const order = [];
+    const rel1 = await mutex.acquire();
+    order.push("h1-enter");
+    let secondEntered = false;
+    const p2 = mutex.acquire().then((rel2) => { secondEntered = true; order.push("h2-enter"); return rel2; });
+    await new Promise((r) => setTimeout(r, 15));
+    assert.equal(secondEntered, false, "second waiter must NOT enter while first holds the mutex");
+    order.push("h1-release");
+    rel1();
+    const rel2 = await p2;
+    assert.equal(secondEntered, true, "second waiter enters only after release");
+    rel2();
+    assert.deepEqual(order, ["h1-enter", "h1-release", "h2-enter"]);
+  });
+
+  await testAsync("createSerialMutex: N acquires run strictly in FIFO order, never overlapping", async () => {
+    const mutex = createSerialMutex();
+    const events = [];
+    let active = 0;
+    async function critical(id) {
+      const rel = await mutex.acquire();
+      active++;
+      assert.equal(active, 1, `only one holder at a time (id=${id})`);
+      events.push(`start${id}`);
+      await new Promise((r) => setTimeout(r, 5));
+      events.push(`end${id}`);
+      active--;
+      rel();
+    }
+    await Promise.all([critical(1), critical(2), critical(3)]);
+    assert.deepEqual(events, ["start1", "end1", "start2", "end2", "start3", "end3"]);
+  });
+
+  await testAsync("createSerialMutex: release() is idempotent (double-release does not double-admit)", async () => {
+    const mutex = createSerialMutex();
+    const rel1 = await mutex.acquire();
+    rel1();
+    rel1(); // second call must be a no-op
+    const rel2 = await mutex.acquire(); // should acquire cleanly, exactly once
+    let thirdEntered = false;
+    const p3 = mutex.acquire().then((r) => { thirdEntered = true; return r; });
+    await new Promise((r) => setTimeout(r, 15));
+    assert.equal(thirdEntered, false, "double-release must not have leaked an extra admit slot");
+    rel2();
+    (await p3)();
+  });
+}
+
+// ── Cleanup ──
+runAsyncTests().then(() => {
+  closeDb();
+  console.log(`\n=== Results: ${passed} passed, ${failed} failed ===\n`);
+  process.exit(failed > 0 ? 1 : 0);
+}).catch((e) => {
+  console.error("async test runner crashed:", e);
+  closeDb();
+  process.exit(1);
+});