fix(server): pair the active-request counter with the process lifecycle (supersedes #180) (#193)

* fix(server): pair the active-request counter with the process lifecycle (#180)

`stats.activeRequests` was incremented ~40 lines before the child process was
spawned, but its only decrement site — cleanup() — is wired to that process's
exit/close/error events. Any synchronous throw in the window between them
(buildCliArgs, env assembly, applying the spawn decision, or spawn() itself)
therefore leaked +1 permanently: the increment had happened, and cleanup() was
not yet attached to anything that could undo it. /health and /stats then
over-report in-flight work forever, and the drift is monotonic.

Fix: move the increment to immediately after `activeProcesses.add(proc)`, so
increment and decrement are structurally paired to exactly one process
lifecycle. No try/catch and no reconciliation pass is needed — there is no
longer a window in which the counter is incremented but unowned.

`stats.activeRequests` is observability-only, not a gate: the old
`if (activeRequests >= MAX_CONCURRENT) throw` admission check was removed in
favour of the releaseSlot semaphore (see the comment at server.mjs:1327), and
the only remaining readers are /health, /stats and the settings snapshot. So
deferring the increment past the spawn changes no admission decision.

Reproduced and verified on the real server (no test double). A deliberate
throw injected between the increment and the spawn, then three requests:

  before the fix : 3 requests -> http 500 -> stats.activeRequests == 3  (leaked)
  after the fix  : 3 requests -> http 500 -> stats.activeRequests == 0
  happy path     : real completion "OK", activeRequests 0, total 1, errors 0

The fault injection was removed before committing (no residue: `grep -rn
OCP_FAULT_INJECT` is clean) — it is documented in the PR so any reviewer can
re-run the same three commands.

cli.js citation: NOT APPLICABLE — this touches no cli.js-derived wire
behavior. Scope is justified under ALIGNMENT.md Rule 2 as OCP-owned internal
bookkeeping: the counter is OCP's own observability state, cli.js has no
equivalent operation, and the spawn arguments and protocol are untouched.

Supersedes the approach in #180, which additionally reconciled the counter to
`activeProcesses.size`. That reconciliation is racy: a concurrent request that
has incremented but not yet registered its process is invisible in
`activeProcesses`, so reconciling stomps it back down and under-reports live
work. Pairing the increment to the spawn removes the drift at its source, so
no reconciliation is required. Original report and diagnosis by @konceptnet.

Tests: 449 passed, 0 failed.

Co-Authored-By: Hermes Agent <hermes-agent@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017gbqUZ8HfBZpjjbzQ85oH8

* fix(review): correct three wrong claims in this PR, and add the regression test

The diff was right; three things I wrote about it were not. An independent
reviewer falsified each one, and leaving a correct fix on an incorrect
rationale in git history is exactly what ALIGNMENT.md governance exists to
prevent. Corrections:

1. "No regression test is possible here" — WRONG, and the test is now here.
   Two compounding errors. `test-features.mjs:955` already has `ltBoot`, a
   real-server integration fixture (spawns server.mjs, fake claude binary,
   HTTP) that I had used myself in #191; I reasoned from "cannot import
   server.mjs" straight to "cannot test" without grepping for it. And the
   synchronous throw is reachable from ENV ALONE, needing no production hook:
   buildCliArgs does `args.push("--allowedTools", ...ALLOWED_TOOLS)`, and a
   spread of enough elements throws RangeError synchronously, inside the leak
   window. Measured, zero production changes:

     origin/main : 3 requests -> 500 x3 -> stats.activeRequests == 3
     this branch : 3 requests -> 500 x3 -> stats.activeRequests == 0

   The committed test DISCOVERS the element count rather than hard-coding it
   (the threshold is stack-size dependent, so a fixed number would silently
   stop triggering on another machine and pass vacuously), and asserts the
   requests actually 500 before trusting the counter. Mutation-proven: moving
   the increment back before the spawn fails it with "got 3".

2. My reason for rejecting #180's approach was WRONG. I claimed reconciling to
   activeProcesses.size is racy because a request that incremented but has not
   yet registered is invisible in the Set. There is no `await` anywhere in that
   window — it is fully synchronous, so on Node's single thread that state is
   unobservable and the race cannot occur. The real reason reconciliation is
   unsafe: the decrement is wired to 'exit' while activeProcesses.delete is
   wired to 'close', and exit precedes close (measured exit @2ms, close
   @2017ms with a grandchild holding stdout), so activeProcesses.size
   OVER-reports during that window — the opposite direction from what I wrote.
   Separately, a throw after activeProcesses.add leaves permanent Set residue
   that reconciliation would inherit. Corrected publicly on #180; the
   contributor was owed an accurate rejection.

3. The alignment citation named the wrong authority. ALIGNMENT.md:17 scopes
   Rules 1-5 to Class A only, and Rule 2 is a prohibition, not an
   authorization — citing it as justification is a category error. Every reader
   of stats.activeRequests is a grandfathered Class B.2 endpoint (/health,
   /status, /usage). Correct citation, per ALIGNMENT.md:142 and the in-repo
   precedent at server.mjs:3271-3276 (the F6 spawn.isolated fix):

     Authorized by ADR 0006 (grandfathered as of v3.16.4). The field SET is
     unchanged — no field added, removed or renamed — only the VALUE is made
     truthful. Behaviour-preserving; no new ADR required.

   The "cli.js: no hit, declared" part stands: cli.js has no equivalent
   operation, and spawn argv and protocol are byte-identical.

Also corrected the new comment at the increment: only 'exit' is wired inside
spawnClaudeProcess; 'close'/'error' are wired by each caller, and that is
REQUIRED because a failed spawn never emits 'exit'.

Note for a follow-up issue, not folded in (Iron Rule 11): stats.errors stays 0
across all of these 500s — synchronous throws from spawnClaudeProcess are never
counted. And CLAUDE.md hard-requirement #1 literally says "justify scope under
Rule 2", which is what I followed; CLAUDE.md predates ADR 0006's Class B
section and needs its own governance PR.

Tests: 450 passed, 0 failed.

Co-Authored-By: Hermes Agent <hermes-agent@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017gbqUZ8HfBZpjjbzQ85oH8

* test: skip the leak probe where the OS rejects the trigger, instead of failing CI

The regression test I added broke CI with `spawn E2BIG`, and the reason matters
more than the fix: the env-only trigger is NOT portable, and the review that
proposed it was measured only on darwin (the reviewer said so).

Delivering the trigger needs a single env var of ~2n bytes where n is the
spread-throw threshold (~130k elements here). Linux caps ONE env string at
MAX_ARG_STRLEN = 32 * PAGE_SIZE = 131072 bytes, so execve rejects it outright.
Encoding tricks do not rescue it: empty items cost 1 byte each, but 120k empty
items do not reach the throw threshold, and the threshold itself moves with
stack size — the margin does not exist. buildCliArgs has exactly one spread
(verified), so there is no alternative env-only trigger.

So the test now runs where the OS allows the trigger (macOS/BSD) and SKIPS
LOUDLY where it does not, printing the platform and byte count. It never
passes vacuously: it already asserts the requests actually return 500 before
trusting the counter, and E2BIG is detected both as a synchronous throw and as
an async 'error' event.

Consequence, stated plainly rather than buried: CI runs on Linux, so CI does
NOT exercise this test. The fix is protected on macOS dev machines only. That
is weaker than I would like and is why #197 (portable in-process fault
injection) matters — it is the difference between this class of bug being
guarded and merely having been checked once by hand.

Verified after the change:
  macOS, fixed        -> test RUNS and passes, 450/0
  macOS, bug restored -> test goes RED ("got 3"), 449/1
  Linux CI            -> skips with a printed reason

Co-Authored-By: Hermes Agent <hermes-agent@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017gbqUZ8HfBZpjjbzQ85oH8

* test: make the leak probe run on Linux CI — my "impossible" conclusion was wrong

f84d4c6 concluded the trigger could not be delivered on Linux and made the test
skip there. The reviewer refuted that and built the working version; this is it.
Three things I got wrong, all the same shape — treating a darwin measurement as
universal:

1. "empty items don't reach the throw threshold" — the REASON was wrong. The
   spread's cost is per ELEMENT, not per byte: empty and "x" have an identical
   threshold. What actually kills the empty-item encoding is `.filter(Boolean)`
   at server.mjs:355, which strips empty entries, so ALLOWED_TOOLS ends up empty
   and the spread branch is never entered. The new control mutation demonstrates
   exactly this: tools filtered to empty -> requests return 200, not 500.

2. "the threshold moves with stack size — the margin does not exist" — exactly
   backwards. The stack dependence is the LEVER, and ltBoot already spawns the
   server, so the test owns its argv. Under --stack-size=200 the threshold drops
   ~5x: measured 24069 elements, so 1.5x margin is 36104 elements = 72207 bytes,
   45% UNDER Linux's MAX_ARG_STRLEN of 131072.

3. "buildCliArgs has only one spread, so there is no alternative trigger" — the
   premise is true (verified) but irrelevant. The problem was never finding a
   second spread; it was making the one spread throw with fewer elements.

Test-side only; server.mjs is untouched by this commit.
  - ltBoot(env, dir, nodeArgs = []) — one line, so a caller can set V8 flags
  - the threshold is discovered in a CHILD under the same --stack-size (the old
    version measured it in the parent, whose stack is not the one that matters)
  - hard assert entryBytes <= MAX_ARG_STRLEN, so a future regression fails loudly
    instead of silently degrading to a skip
  - dynamic free port instead of a fixed 39370 (a flake was observed on the fixed
    port across back-to-back runs)
  - the 500s must carry "call stack size exceeded", so an unrelated fault that
    also left the counter at 0 cannot be mistaken for a pass

The Linux claim here is still ARITHMETIC (72207 < 131072), not a measurement —
neither I nor the reviewer has a Linux box. CI is the check: the log must show
this test RUNNING, not skipping.

Co-Authored-By: Hermes Agent <hermes-agent@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017gbqUZ8HfBZpjjbzQ85oH8

---------

Co-authored-by: dtzp555 <dtzp555@gmail.com>
Co-authored-by: Hermes Agent <hermes-agent@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

server.mjs

@@ -1371,7 +1371,6 @@ function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlo
     promptChars = stdinPayload.length;
   }
 
-  stats.activeRequests++;
   stats.totalRequests++;
   stats.oneOffRequests++;
   if (conversationId) {
@@ -1414,6 +1413,17 @@ function spawnClaudeProcess(model, messages, conversationId, keyName, releaseSlo
 
   const proc = spawn(CLAUDE, cliArgs, spawnOpts);
   activeProcesses.add(proc);
+  // Counter drift (#180, reported by @konceptnet): increment ONLY after the spawn has
+  // succeeded and the process is registered. Incrementing before the spawn (as this did) leaked
+  // +1 permanently on any synchronous throw in between — buildCliArgs, env assembly, the spawn
+  // decision, or spawn() itself — because nothing was yet attached that could undo it.
+  //
+  // cleanup() is the SOLE decrement site, but note how it is reached: only 'exit' is wired HERE
+  // (below); 'close' and 'error' are wired by each CALLER (callClaude / callClaudeStreaming).
+  // That caller wiring is REQUIRED, not belt-and-braces — a FAILED spawn emits 'error' and
+  // 'close' but never 'exit', so without it a spawn failure would never decrement. A future
+  // third caller of spawnClaudeProcess must wire them too.
+  stats.activeRequests++;
 
   const t0 = Date.now();
   let gotFirstByte = false;