VLM-Primary
Hybrid Navigation

The deepest truth: at 1 m/s with 58 Hz VLM, the world moves only 1.7 cm between frames. Temporal consistency is free — consecutive answers should agree because the scene barely changed. Any disagreement is either hallucination or a genuine scene transition, detectable through variance tracking.

The most important convention-disguised-as-physics: VLM must output text. The 150M-param ViT encoder produces a 280-token feature vector in 14ms — text decoding adds 4ms. For place recognition, the embedding IS the output.

The abstraction leak between 10,000 ft and ground level is WiFi. The clean 4-tier hierarchy assumes instant inter-tier communication, but tiers run on different hardware (Titan, Panda, Pi) connected by household WiFi. Another leak: the 30,000 ft pitch says "named goals" but ground-level outputs "LEFT MEDIUM" — qualitative directions, not coordinates. Semantic mapping (Phase 2c) bridges this gap.

The most dangerous dependency: llama-server can't expose intermediate embeddings. Phase 2d is blocked. The workaround — deploying separate SigLIP 2 — costs 800MB VRAM on already-constrained Panda. This upstream limitation cascades into hardware budget decisions. The downstream potential is massive: if semantic maps work, spatial memory feeds voice agent, context engine, and home automation.

WiFi has a cliff edge. Below 20ms, fine. At 50ms, degraded. Above 100ms, blind for multiple frames. No graceful degradation — a phase transition. The surprise: VLM query rate barely matters above 15 Hz. The multi-query pipeline's value isn't speed — it's using surplus frames for different questions.

Every 1-2 years, the "learned vs classical" boundary shifts. Annie sits at the pragmatic fusion point — off-the-shelf VLMs for perception, classical SLAM and A* for planning. The 2027 question: will 1B VLAs become trainable on small datasets? If so, the 4-tier hierarchy collapses into one model. But OK-Robot's lesson persists — clean integration of replaceable components may remain more practical for low-volume robotics.

The killer second-order effect: spatial memory meets conversational memory. "Mom mentioned needing her glasses" (Context Engine) + "glasses on bedroom nightstand" (semantic map) + "Mom sounded tired" (SER) = proactive care without being asked. The concerning effect: a camera robot mapping rooms creates comprehensive surveillance, even unintentionally. Needs explicit consent architecture.

Annie occupies the sweet spot: zero training with medium-high spatial understanding. The only system that's better without training is VLMaps — Annie's Phase 2c target. The empty quadrant (top-left: high understanding + zero training) suggests an opportunity: a plug-and-play semantic SLAM using foundation model embeddings.

The analogy predicts something new: submarines use sound signatures to classify contacts through sonar alone. Annie could similarly use lidar scan patterns — chair legs have distinctive shapes in lidar returns — for obstacle classification without VLM. "Lidar fingerprinting" isn't in the research but the analogy suggests it.

The tradeoff is clear: VLM+SLAM trades robustness for semantics. Pure SLAM is more robust (no WiFi, no hallucinations, no GPU) but understands nothing. The hidden tradeoff: deployment complexity. Setting up VLM pipeline is far harder than SLAM-only, but this doesn't show on the radar.

Most likely failure: WiFi degrades under real conditions (monsoon, family streaming). The glass door scenario is the unknown unknown — both VLM and lidar agree on the wrong answer. No temporal smoothing fixes systematic errors.

The CTO's challenge is hardest: why 2B params for 2 tokens? The value isn't in those tokens — it's in the 150M-param vision encoder's scene understanding. Text output is lossy compression. Phase 2d (embeddings) makes this explicit.

Most seductive mistake: asking VLM for metric estimation. "How far is that chair?" feels natural but monocular depth is fundamentally ambiguous. The research wisely keeps VLM outputs qualitative and uses lidar for all geometry.

Two constraints are artificially imposed: single camera and 2D-only. Both relaxable with $15-50 hardware. The constraint most likely relaxed by technology: VLM model size. In 2 years, 1B params will match today's 2B capability, freeing VRAM.

Most productive inversion: "lidar proposes, VLM confirms." Let SLAM compute A* path, then ask VLM "is this clear?" once per second. This IS Phase 2c, reframed. The radical inversion: no explicit map. PRISM-TopoMap uses embeddings as spatial memory. Eliminates SLAM drift but can't do metric path planning.

Most revealing: if accuracy dropped by half, you could run on Pi 5 CPU directly. No GPU, no WiFi, no Panda. The entire system complexity exists to push accuracy from "barely useful" to "reliable." That last 40% costs 10x the hardware. Constraint to be relaxed soonest: VRAM per model, as vision encoders get more efficient.

Killer combination: semantic map + voice agent. "Last time I was in the bedroom, I saw your glasses on the nightstand." This bridges Context Engine conversation memory with navigation spatial memory. Neither alone is this powerful. Untried: VLM + audio SLAM — room acoustics differ between rooms and nobody uses this for indoor robots.

The multi-query VLM pipeline transfers to any edge robot with a camera and VLM. Security patrols, greenhouses, retail all benefit. Startup opportunity: package this as open-source ROS2 middleware before the space gets crowded.

Three binary questions: camera, semantics needed, edge VLM speed. All yes = use VLM+SLAM hybrid. This is wrong for pure obstacle avoidance (lidar cheaper) and wrong for cloud robots (latency kills fusion).

Phase transition at ~10 robots: "dedicated GPU per robot" breaks. Need shared inference service. At 1000, need fleet learning (Tesla's approach). Annie's architecture is explicitly artisanal, not industrial.

The 42-second journey reveals: the value isn't speed (walking is faster). It's not having to get up. The architecture serves a lifestyle use case, not efficiency. Debugging spans 4 tiers on 3 machines — the hidden cost of distributed hybrid systems.

Gap between Rajesh (sees architecture) and Mom (sees behavior). The 4-tier hierarchy is invisible to users. Phase 2a has zero user-visible benefit unless paired with a capability like "Annie, what room are you in?" Visitor's perspective is most underrepresented — needs a privacy mode.

The plateau is SLAM integration. Sessions 86-92 were spent on ROS2, Zenoh, MessageFilter bugs. VLM side (Levels 1-3) is straightforward prompting. SLAM requires deep expertise. If packaged for others, SLAM deployment must be radically simplified.

Biggest barrier isn't technical — it's the "good enough" incumbent. Lidar-only works for basic avoidance. The VLM hybrid's activation energy is high until you need "go to the kitchen." Catalytic event: a pre-built Docker Compose that works in one command.

Most significant gap: night/low-light. Indian homes have intermittent hallway lighting. Solutions: IR illumination, lidar-only fallback, ambient light sensor adjusting VLM trust. The voice + nav interaction gap is critical for her-os — "what do you see?" mid-navigation requires VLM to switch from nav to descriptive queries.

Cultural blind spot: Annie navigates an Indian home with room types the VLM won't recognize — "pooja room" is unlikely in Gemma 4's training data. The CV researcher's point is valid: a $50 depth camera eliminates the glass-door problem. VLM value is specifically semantic understanding, not geometry.

Most exciting: conversation maps fusing spatial + emotional context. "Mom mentioned glasses" (Context Engine) + "glasses on nightstand" (semantic map) + "Mom sounded tired" (SER) = proactive care. Most practical: what's the minimum VLM? If 400M works on Pi 5 CPU, the entire Panda GPU dependency disappears.