# her-os — Feature Guide (Sprints 1–8)

Everything you can use today, organized by how you interact with it.

---

## Telegram Bot (@her_os_bot)

### Commands You Can Type

| Command | What It Does |
|---------|-------------|
| `/start` | Welcome message with available commands |
| `/briefing` | Fetch yesterday's daily reflection on demand |
| `/search <query>` | Search past conversations (e.g., `/search Arun Bangalore project`) |
| `/promises` | List active promises with urgency scores and deadlines |
| `/help` | Show all available commands |
| *(any text)* | Treated as a free-text memory search |

### Automatic Daily Messages

| Time (IST) | What You Receive |
|-------------|-----------------|
| **5:30 AM** | **Morning Briefing** — Yesterday's reflection + active promises + Wonder photo + Comic photo |
| **9:00 PM** | **Evening Question** — Personalized reflective prompt based on today's emotional peak |

### Morning Briefing Contents (5:30 AM)

1. **Daily Reflection** — Text summary of yesterday's conversations, speakers, key topics
2. **Daily Wonder** — A curiosity-sparking fact with SVG visual (rotates through 12 domains: astronomy, biology, philosophy, mathematics, physics, art, technology, geography, psychology, history, language, synthesis)
3. **Daily Comic** — An XKCD-style 3-panel comic strip drawn from yesterday's conversations (only uses positive moments, sensitivity-gated)

### Nudge Cards (10:00 AM daily)

Proactive promise reminders with inline buttons:
- **✅ Done** — Mark promise as fulfilled
- **⏱️ Snooze 24h** — Defer the reminder
- **🙅 Not helpful** — Log feedback to improve future nudges

Nudges are emotionally gated — won't send if recent emotional valence is negative.

### Promise Heartbeat (Every 6 hours)

If you have promises approaching their deadline, you'll get a gentle reminder with the promise name and urgency level (low/medium/high/critical).

### Fulfillment Confirmation

When Annie detects you may have fulfilled a promise during voice conversation, you get a confirmation card:
- **"Yes, fulfilled"** → Marks promise done
- **"No, not yet"** → Keeps it active

---

## Annie Voice Agent

### What Annie Can Do During Conversations

| Tool | How to Trigger | Example |
|------|---------------|---------|
| **Memory Search** | Ask about past conversations | "What did Arun say about the Bangalore project?" |
| **Web Search** | Ask about current events/facts | "What's the weather in Bangalore?" |
| **Webpage Fetch** | Ask to summarize a URL | "Can you summarize this article?" |
| **Render Table** | Ask for comparisons | "Compare these three laptops" → table appears on screen |
| **Render Chart** | Ask for visualizations | "Show my mood this week" → chart appears on screen |
| **Render SVG** | Ask for diagrams | "Draw a flowchart of the process" → SVG on screen |

Annie also:
- Knows your conversation history (pulls context before each session)
- Speaks with personality (companion persona, not assistant)
- Detects when you've fulfilled promises and triggers confirmation

---

## Context Engine API

Base URL: `http://<titan-ip>:8100`

### Memory & Search
| Endpoint | Method | What |
|----------|--------|------|
| `/v1/context?query=...` | GET | Hybrid search across all conversations |
| `/v1/entities?entity_type=...` | GET | List extracted entities (person, promise, event, topic) |
| `/v1/daily?date=...` | GET | Daily reflection for a date |

### Emotional Analysis
| Endpoint | Method | What |
|----------|--------|------|
| `/v1/emotions/arc?date=...` | GET | Emotion arc data (valence, arousal, peaks) |
| `/v1/emotions/arc/svg?date=...` | GET | SVG visualization of emotion timeline |
| `/v1/emotional-peak?date=...` | GET | Highest-arousal moment of the day |

### Promise Tracking
| Endpoint | Method | What |
|----------|--------|------|
| `/v1/promises?status=active` | GET | All promises with urgency scores |
| `/v1/promises/due?min_urgency=0.4` | GET | Promises eligible for notification |
| `/v1/promises/{id}` | PATCH | Update status (fulfilled/broken) |
| `/v1/promises/{id}/notify` | PATCH | Acknowledge notification (server-side) |

### Nudge Engine
| Endpoint | Method | What |
|----------|--------|------|
| `/v1/nudge/evaluate` | POST | Trigger nudge evaluation cycle |
| `/v1/nudge/feedback` | POST | Log user reaction to nudge |

### Daily Wonder & Comic
| Endpoint | Method | What |
|----------|--------|------|
| `/v1/wonder?date=...` | GET | Cached wonder fact (JSON) |
| `/v1/wonder/svg?date=...` | GET | Wonder visual (SVG image) |
| `/v1/comic?date=...` | GET | Cached comic script (JSON) |
| `/v1/comic/svg?date=...` | GET | Comic strip (SVG image) |

### Memory Management
| Endpoint | Method | What |
|----------|--------|------|
| `/v1/decay` | POST | Manually trigger nightly decay cycle (entity tier demotion) |
| `/v1/entities/{entity_id}` | DELETE | Permanently delete an entity and all mentions |
| `/v1/entities?entity_type=...` | DELETE | Bulk delete all entities of a type |
| `/v1/entities/clustered` | GET | List entities with crowded types auto-clustered |
| `/v1/emotions/colors` | GET | Resolve emotion colors via embedding similarity |

### Observability
| Endpoint | Method | What |
|----------|--------|------|
| `/v1/events/stream` | GET | SSE real-time creature activity stream |
| `/v1/events` | GET | Historical events (time machine) |
| `/v1/events/emit` | POST | Emit observability events |
| `/health` | GET | Health check (no auth) |

---

## Background Processes

### Nightly Window (3:00 AM IST)

Three tasks run in parallel:
1. **Entity Decay** — Fades old entities, promotes/demotes memory tiers (L0→L1→L2)
2. **Wonder Generation** — Picks domain, calls LLM, generates SVG, caches in DB
3. **Comic Generation** — Extracts moments, sensitivity-gates, calls LLM, generates SVG strip

### Automatic Ingestion

- **File Watcher** — Monitors JSONL files from audio pipeline, auto-ingests new segments
- **Entity Extraction** — Claude NER extracts persons, promises, events, topics, emotions
- **Promise Lifecycle** — Auto-expires overdue promises (deadline + 2-day grace period)

---

## Dashboard

URL: `http://192.168.68.52:5174/?token=bUFUGVI9n_vxO-mnWBF4lamWg9PjmE3Qs4I8NgmCDj0`

Features:
- **Aquarium** — Real-time creature activity (31 creatures across LISTENING/THINKING/ACTING zones)
- **Memory Zone** — Entity memory visualization with L0/L1/L2 tier bands, colored by type (person, topic, place, event, promise)
- **Decay Button** — Toolbar button to manually trigger the nightly decay cycle and watch entities shift tiers in real-time
- **Emotion-Specific Colors** — Each emotion entity gets a unique color matching human intuition: joy = golden sunshine, love = warm pink, sadness = deep indigo, anger = red, fear = dark purple, calm = sky blue. Unknown emotions (e.g., "cute", "awe") are resolved via embedding similarity to the nearest known emotion color.
- **Entity Deletion** — Click any entity dot → context card → "Forget" button to permanently delete unwanted memories. Bulk delete by type via API.
- **Adaptive Dot Sizing** — Crowded type columns (>8 entities) automatically shrink dots to fit. Sparse columns keep full-size dots. Minimum size floor prevents invisible dots.
- **Hierarchical Clustering** — When a type column overflows, overflow entities are grouped by semantic similarity (embedding-based) into "+N" cluster dots with member tooltips.
- **Draggable Context Cards** — Grab the context card header to reposition it anywhere on screen.
- **Entity Graph** — Visual knowledge graph of extracted entities
- **Emotion Timeline** — Interactive emotion arc display
- **Session Navigator** — Browse conversation sessions
- **Thought Bubbles** — Floating entity insights
- **Trace Waterfall** — Deep-dive into processing traces
- **Chat Interface** — Text chat with Annie

---

## Architecture Summary

```
Omi Wearable → Flutter App → Audio Pipeline → JSONL files
                                                    ↓
                                            Context Engine (PostgreSQL)
                                           /        |        \
                                     Annie Voice  Dashboard  Telegram Bot
                                     (Pipecat)    (Vite)     (python-telegram-bot)
```

### Service Ports
| Service | Port |
|---------|------|
| PostgreSQL | 5432 |
| Context Engine | 8100 |
| Audio Pipeline | 9100 |
| Ollama | 11434 |
| llama-server | 8003 |
| SearXNG | 8888 |
| Annie Voice | 7860 |
| Dashboard | 5174 |

---

## What's Coming Next

- **Sprint 9+**: Check `docs/GAP-ANALYSIS.md` for critical path
- Richer pre-session briefing (session summaries, key entities, emotional state)
- MCP Knowledge Graph Memory Server integration
- Entity consolidation (cross-session dedup)
- Email agent integration
- Voice call infrastructure