memory-continuity/docs/superpowers/plans/2026-03-31-v3.1-short-term-memory.md

# v3.1 Short-Term Memory Layer Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add session logging, smart tail protection during compaction, CJK token awareness, and noise filtering to the Memory Continuity plugin.

**Architecture:** Four independent features added to the existing lifecycle hook system. Session logs write to `memory/sessions/YYYY-MM-DD.md` (one file per day, append-only). Tail protection injects recent critical messages into `before_compaction`. CJK token estimation uses a helper function for line-count decisions. Ignore patterns filter out cron/subagent noise at `agent_end`.

**Tech Stack:** Node.js (ES modules), pure filesystem, zero dependencies

---

## File Structure

| File | Action | Responsibility |
|------|--------|---------------|
| `index.js` | Modify | Add session logging in `agent_end`, enhance `before_compaction` with tail protection, add CJK helper, add ignore-pattern filtering |
| `openclaw.plugin.json` | Modify | Add new config keys: `sessionLogging`, `tailProtectCount`, `ignorePatterns` |
| `mc-plugin/index.js` | Modify | Add `/mc sessions` subcommand, update help text |
| `mc-plugin/openclaw.plugin.json` | No change | — |

---

### Task 1: CJK Token Estimation Helper

**Files:**
- Modify: `index.js` (after line 15, the PLACEHOLDER_VALUES block)

- [ ] **Step 1: Add `estimateTokens` helper function to `index.js`**

Add this after the `PLACEHOLDER_VALUES` constant (line 15):

```javascript
/**
 * Estimate token count with CJK awareness.
 * CJK characters ≈ 1.5 tokens each; Latin words ≈ 1 token per ~4 chars.
 */
function estimateTokens(text) {
  if (!text) return 0;
  // Count CJK characters (CJK Unified Ideographs + common CJK ranges)
  const cjkCount = (text.match(/[\u4e00-\u9fff\u3400-\u4dbf\uf900-\ufaff\u3000-\u303f\uff00-\uffef]/g) || []).length;
  // Remove CJK chars, count remaining as ~1 token per 4 chars
  const nonCjk = text.replace(/[\u4e00-\u9fff\u3400-\u4dbf\uf900-\ufaff\u3000-\u303f\uff00-\uffef]/g, "");
  const latinTokens = Math.ceil(nonCjk.length / 4);
  return Math.ceil(cjkCount * 1.5) + latinTokens;
}
```

- [ ] **Step 2: Update `truncate` in `extractStateFromMessages` to use token-aware truncation**

In `extractStateFromMessages` (around line 175), replace the existing `truncate` function:

```javascript
  // Token-aware truncation
  const truncate = (s, maxTokens = 200) => {
    if (estimateTokens(s) <= maxTokens) return s;
    // Binary search for the right cut point
    let lo = 0, hi = s.length;
    while (lo < hi) {
      const mid = (lo + hi + 1) >> 1;
      if (estimateTokens(s.slice(0, mid)) <= maxTokens) lo = mid;
      else hi = mid - 1;
    }
    return s.slice(0, lo) + "...";
  };
```

- [ ] **Step 3: Verify no syntax errors**

Run:
```bash
cd /Users/taodeng/.openclaw/projects/memory-continuity && node -c index.js
```
Expected: no output (syntax OK)

- [ ] **Step 4: Commit**

```bash
git add index.js
git commit -m "feat: add CJK token-aware estimation helper (v3.1)"
```

---

### Task 2: Ignore Patterns — Filter Cron/Subagent Noise

**Files:**
- Modify: `index.js` (agent_end hook, around line 295)
- Modify: `openclaw.plugin.json` (add `ignorePatterns` config)

- [ ] **Step 1: Add `ignorePatterns` to `openclaw.plugin.json` config schema**

Add after the `maxArchiveCount` property block (after line 29):

```json
      "ignorePatterns": {
        "type": "array",
        "default": [],
        "description": "Regex patterns to ignore sessions (e.g. cron jobs, subagent noise). Matched against first user message."
      }
```

And add corresponding uiHints after the `maxArchiveCount` hint (after line 47):

```json
    "ignorePatterns": {
      "label": "Ignore patterns",
      "help": "Skip state extraction for sessions matching these patterns (regex)"
    }
```

- [ ] **Step 2: Add ignore-pattern check in `agent_end` hook in `index.js`**

In the `agent_end` hook, after the `realUserMsgs` filtering block (after line 321), add:

```javascript
      // Check ignore patterns — skip sessions matching cron/subagent noise
      const ignorePatterns = (config.ignorePatterns || [])
        .map(p => { try { return new RegExp(p, "i"); } catch { return null; } })
        .filter(Boolean);

      if (ignorePatterns.length > 0 && realUserMsgs.length > 0) {
        const firstMsg = realUserMsgs[0];
        if (ignorePatterns.some(re => re.test(firstMsg))) {
          log.info?.("[memory-continuity] Session matches ignorePattern, skipping");
          return;
        }
      }
```

- [ ] **Step 3: Verify no syntax errors**

Run:
```bash
cd /Users/taodeng/.openclaw/projects/memory-continuity && node -c index.js
```
Expected: no output (syntax OK)

- [ ] **Step 4: Commit**

```bash
git add index.js openclaw.plugin.json
git commit -m "feat: add ignorePatterns to filter cron/subagent noise (v3.1)"
```

---

### Task 3: Session Logging

**Files:**
- Modify: `index.js` (add `writeSessionLog` helper + call it in `agent_end`)

- [ ] **Step 1: Add `writeSessionLog` helper function to `index.js`**

Add after the `cleanupMemoryDir` function (after line 140):

```javascript
/**
 * Append a session summary to the daily session log.
 * File: memory/sessions/YYYY-MM-DD.md (one per day, append-only)
 */
function writeSessionLog(workspaceDir, messages, config = {}) {
  if (config.sessionLogging === false) return;
  if (!messages || messages.length === 0) return;

  const now = new Date();
  const pad = (n) => String(n).padStart(2, "0");
  const dateStr = `${now.getFullYear()}-${pad(now.getMonth() + 1)}-${pad(now.getDate())}`;
  const timeStr = `${pad(now.getHours())}:${pad(now.getMinutes())}`;

  const sessionsDir = path.join(workspaceDir, "memory", "sessions");
  fs.mkdirSync(sessionsDir, { recursive: true });

  const logFile = path.join(sessionsDir, `${dateStr}.md`);

  // Extract first meaningful user message as topic
  let topic = "(no topic)";
  for (const msg of messages) {
    if (msg?.role !== "user") continue;
    const text = typeof msg.content === "string"
      ? msg.content
      : Array.isArray(msg.content)
        ? msg.content.filter(b => b?.type === "text").map(b => b.text).join("\n")
        : "";
    const cleaned = text
      .replace(/^Conversation info \(untrusted metadata\):[\s\S]*?\n\n/m, "")
      .replace(/^Sender \(untrusted metadata\):[\s\S]*?\n\n/m, "")
      .trim();
    if (cleaned.length > 10) {
      topic = cleaned.split("\n")[0].slice(0, 120);
      break;
    }
  }

  // Count messages by role
  const userCount = messages.filter(m => m?.role === "user").length;
  const assistantCount = messages.filter(m => m?.role === "assistant").length;
  const totalTokens = estimateTokens(
    messages.map(m => typeof m?.content === "string" ? m.content : "").join("")
  );

  // Build log entry
  const entry = [
    `### ${timeStr}`,
    `- **Topic:** ${topic}`,
    `- **Messages:** ${userCount} user / ${assistantCount} assistant`,
    `- **Est. tokens:** ~${totalTokens}`,
    "",
  ].join("\n");

  // Append to daily log (create with header if new)
  if (!fs.existsSync(logFile)) {
    const header = `# Session Log — ${dateStr}\n\n`;
    fs.writeFileSync(logFile, header + entry, "utf8");
  } else {
    fs.appendFileSync(logFile, entry, "utf8");
  }
}
```

- [ ] **Step 2: Add `sessionLogging` config to `openclaw.plugin.json`**

Add after the `ignorePatterns` property in the config schema:

```json
      "sessionLogging": {
        "type": "boolean",
        "default": true,
        "description": "Write session summaries to memory/sessions/ daily logs"
      }
```

And add corresponding uiHints:

```json
    "sessionLogging": {
      "label": "Session logging",
      "help": "Append session summaries to daily markdown logs"
    }
```

- [ ] **Step 3: Call `writeSessionLog` from `agent_end` hook**

In the `agent_end` hook, right before the `const existing = readFile(statePath);` line (around line 324), add:

```javascript
      // Write session log entry
      writeSessionLog(ws, messages, config);
```

- [ ] **Step 4: Verify no syntax errors**

Run:
```bash
cd /Users/taodeng/.openclaw/projects/memory-continuity && node -c index.js
```
Expected: no output (syntax OK)

- [ ] **Step 5: Commit**

```bash
git add index.js openclaw.plugin.json
git commit -m "feat: add session logging to memory/sessions/ daily files (v3.1)"
```

---

### Task 4: Smart Tail Protection in `before_compaction`

**Files:**
- Modify: `index.js` (enhance `before_compaction` hook, around line 254)
- Modify: `openclaw.plugin.json` (add `tailProtectCount` config)

- [ ] **Step 1: Add `tailProtectCount` config to `openclaw.plugin.json`**

Add after the `sessionLogging` property in the config schema:

```json
      "tailProtectCount": {
        "type": "number",
        "default": 3,
        "description": "Number of recent critical message pairs to protect during compaction"
      }
```

And add corresponding uiHints:

```json
    "tailProtectCount": {
      "label": "Tail protect count",
      "help": "Keep N recent user/assistant pairs visible after compaction"
    }
```

- [ ] **Step 2: Add `extractTailMessages` helper to `index.js`**

Add after the `writeSessionLog` function:

```javascript
/**
 * Extract the last N meaningful user/assistant exchange pairs from messages.
 * Returns a formatted string for injection into compaction context.
 */
function extractTailMessages(messages, count = 3) {
  if (!messages || messages.length === 0) return null;

  // Walk backwards, collect up to `count` user+assistant pairs
  const pairs = [];
  let i = messages.length - 1;

  while (i >= 0 && pairs.length < count) {
    // Find assistant message
    while (i >= 0 && messages[i]?.role !== "assistant") i--;
    if (i < 0) break;
    const assistantMsg = messages[i];
    i--;

    // Find preceding user message
    while (i >= 0 && messages[i]?.role !== "user") i--;
    if (i < 0) break;
    const userMsg = messages[i];
    i--;

    const getText = (msg) => {
      if (typeof msg?.content === "string") return msg.content;
      if (Array.isArray(msg?.content)) {
        return msg.content.filter(b => b?.type === "text").map(b => b.text).join("\n");
      }
      return "";
    };

    const userText = getText(userMsg).trim();
    const assistantText = getText(assistantMsg).trim();

    // Skip trivial exchanges
    if (userText.length < 10 && assistantText.length < 20) continue;

    // Token-aware truncation per message
    const maxPerMsg = 150;
    const truncMsg = (s) => {
      if (estimateTokens(s) <= maxPerMsg) return s;
      let lo = 0, hi = s.length;
      while (lo < hi) {
        const mid = (lo + hi + 1) >> 1;
        if (estimateTokens(s.slice(0, mid)) <= maxPerMsg) lo = mid;
        else hi = mid - 1;
      }
      return s.slice(0, lo) + "...";
    };

    pairs.unshift({ user: truncMsg(userText), assistant: truncMsg(assistantText) });
  }

  if (pairs.length === 0) return null;

  const lines = ["=== RECENT EXCHANGE (protected) ==="];
  for (const p of pairs) {
    lines.push(`User: ${p.user}`);
    lines.push(`Assistant: ${p.assistant}`);
    lines.push("---");
  }
  lines.push("=== END RECENT EXCHANGE ===");
  return lines.join("\n");
}
```

- [ ] **Step 3: Enhance `before_compaction` hook to inject tail messages**

Replace the existing `before_compaction` hook body (lines 254-270) with:

```javascript
    api.on("before_compaction", async (_event, _ctx) => {
      const ws = _ctx?.workspaceDir;
      const config = getConfig();
      const statePath = resolveStatePath(ws);
      if (!statePath) return;

      const md = readFile(statePath);
      if (!md) return;

      const snapshot = buildSnapshot(md);
      if (!snapshot) return;

      log.info?.("[memory-continuity] Injecting state before compaction");

      // Smart tail protection: also inject recent critical messages
      const tailCount = config.tailProtectCount ?? 3;
      const messages = _event?.messages;
      const tail = tailCount > 0 ? extractTailMessages(messages, tailCount) : null;

      const parts = [snapshot];
      if (tail) {
        parts.push(tail);
        log.info?.("[memory-continuity] Tail protection: injected recent exchanges");
      }

      return {
        prependSystemContext: parts.join("\n\n"),
      };
    }, { priority: 10 });
```

- [ ] **Step 4: Verify no syntax errors**

Run:
```bash
cd /Users/taodeng/.openclaw/projects/memory-continuity && node -c index.js
```
Expected: no output (syntax OK)

- [ ] **Step 5: Commit**

```bash
git add index.js openclaw.plugin.json
git commit -m "feat: smart tail protection during compaction (v3.1)"
```

---

### Task 5: `/mc sessions` Command

**Files:**
- Modify: `mc-plugin/index.js` (add `cmdSessions` function + wire into switch)

- [ ] **Step 1: Add `cmdSessions` handler to `mc-plugin/index.js`**

Add after the `cmdExport` function (before `cmdHelp`, around line 437):

```javascript
function cmdSessions(args) {
  const parts = (args || "").trim().split(/\s+/);
  const agent = parts.find(p => !p.startsWith("-") && !p.startsWith("2")) || "main";
  const dateArg = parts.find(p => /^\d{4}-\d{2}-\d{2}$/.test(p));

  const memDir = resolveMemDir(agent);
  if (!memDir) return `Agent "${agent}" not found.`;

  const sessionsDir = path.join(memDir, "sessions");
  let files;
  try { files = fs.readdirSync(sessionsDir).filter(f => f.endsWith(".md")).sort().reverse(); }
  catch { return `No session logs for "${agent}".`; }

  if (!files.length) return `No session logs for "${agent}".`;

  // If date specified, show that day's log
  if (dateArg) {
    const target = `${dateArg}.md`;
    const content = readFile(path.join(sessionsDir, target));
    if (!content) return `No session log for ${dateArg}.`;
    // Truncate to last 50 lines to stay compact
    const lines = content.split("\n");
    const shown = lines.length > 50 ? lines.slice(-50) : lines;
    let out = shown.join("\n");
    if (lines.length > 50) out = `... (${lines.length - 50} earlier lines omitted)\n\n` + out;
    return out;
  }

  // List recent session logs
  let out = `Session Logs: ${agent} (${files.length} days)\n`;
  out += "─────────────────────────────\n";

  for (const f of files.slice(0, 14)) {
    const date = f.replace(".md", "");
    const content = readFile(path.join(sessionsDir, f));
    // Count session entries (### HH:MM headers)
    const sessionCount = content ? (content.match(/^### \d{2}:\d{2}/gm) || []).length : 0;
    out += `${date}   ${String(sessionCount).padStart(3)} session(s)\n`;
  }

  if (files.length > 14) out += `\n  ... and ${files.length - 14} more days`;
  out += `\n\nUse /mc sessions <YYYY-MM-DD> to view a specific day.`;
  return out.trimEnd();
}
```

- [ ] **Step 2: Wire `sessions` into the command switch**

In the switch statement (around line 471), add a new case before `help`:

```javascript
          case "sessions": text = cmdSessions(subargs || null); break;
```

- [ ] **Step 3: Update `cmdHelp` to include `sessions`**

Add this line in the help text after the `/mc state --all` line:

```
/mc sessions [date]     Session logs (daily activity)
```

- [ ] **Step 4: Verify no syntax errors**

Run:
```bash
cd /Users/taodeng/.openclaw/projects/memory-continuity && node -c mc-plugin/index.js
```
Expected: no output (syntax OK)

- [ ] **Step 5: Commit**

```bash
git add mc-plugin/index.js
git commit -m "feat: add /mc sessions command for daily session logs (v3.1)"
```

---

### Task 6: Version Bump and Final Verification

**Files:**
- Modify: `openclaw.plugin.json` (version 3.0.0 → 3.1.0)
- Modify: `mc-plugin/openclaw.plugin.json` (version if present)
- Modify: `package.json` (version if present)

- [ ] **Step 1: Bump version in `openclaw.plugin.json` to 3.1.0**

Change `"version": "3.0.0"` to `"version": "3.1.0"`.

- [ ] **Step 2: Bump version in `package.json` to 3.1.0**

Change `"version"` value to `"3.1.0"`.

- [ ] **Step 3: Verify both plugins parse correctly**

Run:
```bash
cd /Users/taodeng/.openclaw/projects/memory-continuity && node -c index.js && node -c mc-plugin/index.js && node -e "JSON.parse(require('fs').readFileSync('openclaw.plugin.json','utf8'))" && echo "All OK"
```
Expected: `All OK`

- [ ] **Step 4: Commit and tag**

```bash
git add openclaw.plugin.json package.json
git commit -m "release: v3.1.0 — short-term memory layer"
```

- [ ] **Step 5: Push to GitHub**

```bash
git push
```

---

## Summary of New Config Keys

| Key | Type | Default | Purpose |
|-----|------|---------|---------|
| `ignorePatterns` | `string[]` | `[]` | Regex patterns to skip sessions (cron/subagent) |
| `sessionLogging` | `boolean` | `true` | Enable daily session log files |
| `tailProtectCount` | `number` | `3` | Recent exchange pairs to protect during compaction |

## Summary of New/Changed Hooks

| Hook | Change |
|------|--------|
| `agent_end` | +session logging, +ignore pattern filtering, +CJK-aware truncation |
| `before_compaction` | +tail message protection |

## New Command

| Command | Purpose |
|---------|---------|
| `/mc sessions [date]` | List daily session logs or view specific day |