# her-os Gap Analysis — Vision vs. Reality

**Generated:** 2026-03-04 (Session 183)
**Last updated:** 2026-03-04 (Sprint 2 review complete)

## The Scoreboard

| | Rajesh Scenes | Annie Scenes | Total |
|--|--|--|--|
| **In narrative** | 23 | 28 | 51 |
| **Fully built** | 5 | 3 | 8 |
| **Partially built** | 6 | 5 | 11 |
| **Not started** | 12 | 20 | 32 |

---

## Phase 1: Session-Based Voice (DEPLOYED — 100%)

| Scene | Vision | Reality | Gap |
|---|---|---|---|
| **R-01 / A-01: First Words / Pipeline** | WebRTC voice, STT→LLM→TTS | **100% built.** Whisper GPU (62x RT), Claude/Ollama, Kokoro TTS (~30ms), Pipecat pipeline | None |
| **R-02 / A-02: Web Question / Search Within** | Ask Annie to search web | **100% built.** SearXNG, Llama 3.1 8B tool calling, fetch_webpage, SSRF protection, readability parsing | None |
| **R-03: The Comparison** | Multi-turn product research | **100% built.** Same tools as R-02, multi-turn context via Pipecat aggregator | None |
| **R-04: The Reflection** | Evening chat, personality | **100% built.** System prompt personality, Claude API | None |
| **R-05 / A-03: Honest Limit / Forgetting** | Admits what it doesn't know, session cleanup | **100% built.** Clean teardown, no hallucination guardrails (Claude native) | None |

---

## Phase 2: Memory + Proactive (IN PROGRESS)

### Backend Infrastructure — What's Built

| Capability | Status | Location |
|---|---|---|
| JSONL transcript ingest | ✅ Working | `context-engine/ingest.py` |
| Entity extraction (9 types) | ✅ Working | `context-engine/extract.py` — Claude-powered NER |
| Hybrid search (BM25 + Vector) | ✅ Working | `context-engine/retrieve.py` — 70/30 fusion |
| Temporal decay logic | ✅ Scheduled | `context-engine/nightly.py` — asyncio loop in lifespan (Sprint 1) |
| Memory tiers (L0/L1/L2) | ⚠️ Framework only | Classification logic present, promotion not running |
| Knowledge graph (Neo4j/Graphiti) | ✅ Working | `context-engine/graphiti_client.py` |
| Vector embeddings (Qdrant) | ✅ Working | Qwen3-Embedding-8B (14.1 GB VRAM) |
| Daily reflection generation | ✅ Working | `context-engine/daily.py` |
| Speaker diarization | ✅ Working | `audio-pipeline` — pyannote 3.1 |
| Emotion recognition | ✅ Working | `ser-pipeline` — emotion2vec+ large + wav2vec2 |
| Observability (27 creatures) | ✅ Working | SSE events, ring buffer, dashboard |
| Dashboard (full) | ✅ Working | 27 creatures, connections, time machine, entity cards |
| Pre-session memory loading | ✅ Working | `annie-voice/context_loader.py` |

### Phase 2 Scenes — Gap Detail

| Scene | What's Missing | Difficulty | Unlocked By |
|---|---|---|---|
| **R-06 / A-05: Morning Briefing** | ⏳ Sprint 2 (plan reviewed, ready to implement): scheduler.py + 5:30 AM push via Telegram. daily.py + telegram-bot foundation exist. | Medium | Sprint 2 Step 1 |
| **R-07: Coffee Memory Lookup** | Omi wearable integration. Entity search works via API but no voice-triggered "what was that?" flow. | Medium | Omi webhook + intent classifier |
| **R-08 / A-08: Silent Capture** | Omi webhook gateway. Lane Queue designed but not coded. Audio pipeline can process audio but nothing sends Omi audio. | High | Omi integration |
| **R-09 / A-10: Phone Call Capture** | Same Omi gap. Diarization works. Entity enrichment designed but not coded. | High | Omi integration |
| **R-10 / A-11: "What did Arun say?"** | ✅ Sprint 1 complete: `search_memory` tool wired into Annie (memory_tools.py → bot.py → text_llm.py). 33 tests. | **Done** | Sprint 1 Task 1a |
| **R-11 / A-12: The Nudge** | Not built. Nudge composer, humanizer (25 rules), activity gap detector — all designed, none coded. | High | New module |
| **R-12 / A-17: Family Hour** | Child speech filter not built. Selective extraction not built. | Medium | Filter on diarization |
| **R-13 / A-18: Mom Call** | Health tracker skill, emotional arc tracker, promise fulfillment — not built. | High | Multiple new modules |
| **R-14 / A-22: Emotional Arc** | Emotion recognition works (SER deployed). But no visualization — no SVG, no temporal curve. | Medium | Frontend + API endpoint |
| **R-15 / A-25: Dinner Info** | Entity search works but no just-in-time notification. No clipboard bridge. No presence detection. | High | Mobile integration |
| **R-16 / A-24: Evening Question** | ⏳ Sprint 2 (plan reviewed): emotional peak DB query + 9 PM reflective question push via Telegram. | Medium | Sprint 2 Steps 2a+2b |
| **A-04: Nightly Garden** | ✅ Sprint 1 complete: nightly decay scheduled (nightly.py, asyncio loop in lifespan). No backup pipeline. No Moltbook observer. | Partial | Sprint 1 Task 1b |
| **A-06: Promise Ledger** | Not built. Promise extraction, urgency scoring, heartbeat scheduler. | High | New module |
| **A-09: Context Wall** | Not built. Channel classifier, trust levels, prompt injection defense. | High | New module |
| **A-20: Approaching Wall** | Not built. Context window monitor, pre-emptive flush, conversation compressor. | Medium | LLM prompt engineering |
| **A-23: Living Files** | Entity files structured in PostgreSQL, but not human-readable markdown on disk. No git versioning. | Medium | Export pipeline |

---

## Phase 3: Agent Autonomy (NOT STARTED)

| Scene | What It Needs | Status |
|---|---|---|
| **R-17: Gift Idea** | Shopping assistant, image extraction, product ranking | Not started |
| **R-18 / A-15: The Booking** | Browser automation (Playwright), calendar API, double-confirmation | Not started |
| **R-19 / A-16: The Research** | Sub-agent spawning, multi-source search, comparison tables | Not started |
| **R-20 / A-19: Save Pile** | Clipboard bridge, auto-tagging, knowledge inbox | Not started |
| **R-21 / A-21: Email Glance** | Multi-account OAuth2, email triage, drafting | Not started |
| **R-22 / A-24: Getting Ready** | ⏳ Sprint 2 (plan reviewed): `/v1/daily/audio` TTS endpoint in annie-voice (Kokoro GPU). | Sprint 2 Step 3 |
| **R-23 / A-28: The Mirror** | Multi-day pattern synthesis, meditation engine | Not started |
| **A-07: Soul Check** | IDENTITY.md, divergence tracker, self-modification | Not started |
| **A-13: Skill Creation** | SKILL.md hot-reload, autonomy directive | Not started |
| **A-14: Voice Command** | 5-tier approval system, Telegram confirmation cards | Not started |
| **A-26: Nightly Maintenance** | Self-improvement review, credential rotation, recovery testing | Not started |

---

## What We Built BEYOND the Narrative

| Built | Not in Original Vision | Value |
|---|---|---|
| **27-creature neural reef dashboard** | Narrative mentions dashboard briefly, never details the visualization | Major differentiator — no personal AI has this level of introspection |
| **Review/Time Machine mode** | Not in narrative | Replay any moment in system history with full visual fidelity |
| **Text chat alongside voice** | Narrative is voice-only | Flexibility for quiet environments |
| **Visual tools** (render_table, render_chart) | Not described | Annie can show data, not just speak it |
| **Code execution** (execute_python) | Not described | Annie can run Python, do math, data analysis |
| **Entity context cards** | Not described | Click any entity → rich HUD with stats, mentions, chat |
| **Portal context cards** | Not described | Click any external service → connection stats |
| **Multi-LLM routing** (7 models) | Narrative implies Claude-only | Qwen3, Mistral, Sarvam, Claude, Gemma3, etc. |
| **MCP Knowledge Graph Server** | Not described | Read-only MCP adapter: external tools (Claude Desktop, Cursor) query entities, memory, daily summaries |

---

## Critical Path — What to Build Next

### Tier 1: Quick Wins (< 1 week each, high impact)

1. **Wire `search_memory` tool into Annie voice**
   - retrieve.py works, Annie just can't call it
   - Add tool that calls `/v1/context`
   - **Unlocks: R-10, R-07**

2. **Schedule nightly decay**
   - `apply_nightly_decay()` exists, needs APScheduler/cron
   - **Unlocks: A-04 (partially)**

3. **Morning briefing push**
   - daily.py generates reflections, add 5:30 AM scheduler + Telegram
   - **Unlocks: R-06, A-05**

### Tier 2: Medium Effort (1-2 weeks each)

4. **Omi webhook gateway**
   - Audio pipeline ready, need webhook for Omi POST payload
   - **Unlocks: R-08, R-09, R-12, R-13, A-08, A-10** (6 scenes!)

5. **Emotional arc visualization**
   - SER works, need API endpoint + SVG frontend
   - **Unlocks: R-14**

6. **Promise tracker**
   - Entity extraction catches promises, build ledger + urgency scorer
   - **Unlocks: A-06, A-12**

### Tier 3: Big Lifts (2+ weeks each)

7. **Nudge engine** — promise tracker + activity gap detector + humanizer + delivery
8. **Email agent** — multi-account OAuth2, triage, drafting
9. **Browser automation** — Playwright + screenshot verification + OTP flow

---

## Summary Numbers

| Metric | Value |
|---|---|
| Total vision scenes | 51 (23 Rajesh + 28 Annie) |
| Fully working | 8 (16%) |
| Backend ready, UX missing | 11 (22%) |
| Not started | 32 (63%) |
| Tests passing | ~4,400+ |
| Beyond-narrative features | 9 capabilities not in original vision |
| Most impactful next step | MCP write tools (Phase 2) or email agent |