# Next Session: WhatsApp Agent → Annie Tool Routing

**Priority:** HIGH — WhatsApp is Rajesh's primary messaging channel but has zero tool access
**Parallel session:** Independent of other sessions
**Estimated effort:** 3-4 hours (research existing pipelines + design + implement + test)

## What

The WhatsApp agent currently uses its own Gemma 4 responder (`responder.py`) — plain text generation with NO tool calling. When Rajesh asks "How do I get to Nandi Hills?" on WhatsApp, Annie hallucinates directions instead of calling the `google_maps` tool.

**Goal:** Route WhatsApp queries through Annie's `/v1/chat` SSE endpoint (same as Telegram) so that maps, weather, AQI, web search, memory search, and all other tools work on WhatsApp.

## Why (the Nandi Hills incident)

2026-04-08: Rajesh asked "How do I get to Nandi Hills?" on WhatsApp.
- Annie responded: "You can take the highway towards Devanahalli! Do you want me to send you the Google Maps location?"
- Rajesh said: "Yeah, send me Google Maps location"
- Annie responded: "Sure, sending it right now!" — but sent nothing. No tool calling = no actual directions, no map image.

On Telegram, the same query calls `google_maps` tool → real directions with distance/time/traffic + a static map image.

## Current Architecture

### Telegram (WORKS — has tools)
```
User message → bot.py → chat_with_annie() → /v1/chat SSE → text_llm.py → tool dispatch
                                                              ↓
                                                    Tool results + map_image events
                                                              ↓
                                            ← SSE stream → bot.py → reply_text + reply_photo
```

### WhatsApp (BROKEN — no tools)
```
User message → agent.py → responder.py → Gemma 4 (plain text, no tools)
                                              ↓
                                    "Sure, sending it right now!" (hallucination)
                                              ↓
                              ← wa_sender.py → send_message (text only)
```

## Proposed Architecture

```
User message → agent.py → _check_trigger() → should_use_tools()?
                                                  ↓ YES
                                          chat_with_annie()  ← NEW (same as Telegram)
                                                  ↓
                                          /v1/chat SSE → tool dispatch
                                                  ↓
                                    text + map_image filenames
                                                  ↓
                              wa_sender.send_message(text) + wa_sender.send_image(map)
                                                  
                                                  ↓ NO (simple chat)
                                          responder.py → Gemma 4 (existing, keep for casual)
```

## Key Design Questions

1. **When to route to Annie vs keep local Gemma 4?**
   - Option A: ALWAYS route to Annie (simplest — Telegram does this)
   - Option B: Classify intent first — tool-needing queries → Annie, casual chat → local Gemma 4
   - Option C: Try Annie first, fall back to local on 503 (voice session active)
   - Note: Annie's `/v1/chat` returns 503 when a voice session is active. Telegram handles this by falling back to `search_memory()`. WhatsApp needs the same fallback.

2. **How to send map images on WhatsApp?**
   - `wa_sender.send_image(page, image_path, chat_name, page_lock)` already exists
   - Need to: download from `/v1/images/{filename}` → save to temp file → call `send_image()`
   - The `image_search.py` module already has `download_image()` for this pattern

3. **Context continuity — session management?**
   - Telegram stores `session_id` per chat in `_chat_sessions` dict
   - WhatsApp agent has its own `context_mgr.py` with compaction
   - Need to decide: use Annie's session management (like Telegram) or keep WhatsApp's own context?
   - Probably: use Annie's sessions for tool queries, keep WhatsApp's context for casual chat

4. **Channel parameter?**
   - Annie's `/v1/chat` accepts `channel` parameter (text, telegram, dashboard)
   - Need to add `whatsapp` to `_VALID_CHANNELS` in `server.py` (currently not in the set!)
   - Tool specs already have `channels=("text", "telegram", "phone", "whatsapp")` — tools are ready

5. **WhatsApp agent runs on Panda, Annie runs on Titan — network path?**
   - Panda can reach Titan at `192.168.68.52:7860` (same network)
   - Auth token: same `CONTEXT_ENGINE_TOKEN` (needs to be in WhatsApp agent's `.env`)
   - Latency: ~1ms (same LAN)

## Files to Research

| File | What to check |
|------|--------------|
| `services/whatsapp-agent/agent.py` | Where to insert Annie routing (after trigger classification) |
| `services/whatsapp-agent/responder.py` | Current Gemma 4 call — what to bypass |
| `services/whatsapp-agent/config.py` | Add `ANNIE_VOICE_URL`, `CONTEXT_ENGINE_TOKEN` |
| `services/whatsapp-agent/context_mgr.py` | Current context management — how to integrate |
| `services/telegram-bot/context_client.py` | `chat_with_annie()` — reusable or needs WhatsApp copy? |
| `services/telegram-bot/bot.py:460` | How Telegram handles the 3-tuple return + map images |
| `services/annie-voice/server.py:528` | `_VALID_CHANNELS` — add "whatsapp" |
| `services/whatsapp-agent/wa_sender.py` | `send_image()` for map delivery |
| `services/whatsapp-agent/image_search.py` | `download_image()` pattern for downloading from URL |

## Verification

```bash
# 1. WhatsApp: "How do I get to Nandi Hills?"
# Expected: Real directions with distance/time + static map image

# 2. WhatsApp: "What's the weather?"
# Expected: Temperature/humidity from Weather API + map pin

# 3. WhatsApp: "Is it safe to run outside?"
# Expected: AQI + weather from tools (not hallucinated)

# 4. WhatsApp: casual chat "Good morning Annie"
# Expected: Still responds naturally (not routed to tool dispatch)

# 5. Voice session active → WhatsApp query
# Expected: Falls back gracefully (local Gemma 4 or search_memory)
```

## Complexity Estimate

- **Simplest (Option A — always Annie):** ~2 hours. Add `chat_with_annie()` to WhatsApp agent, wire it in `agent.py`, add image download+send. Risk: loses WhatsApp's own context management.
- **Medium (Option B — classify first):** ~3 hours. Intent classification already exists in `trigger.py`. Route tool-needing queries to Annie, keep casual chat local.
- **Full (Option C — hybrid with fallback):** ~4 hours. Annie first, local Gemma 4 fallback on 503, context bridging. Most robust but most complex.