# Achieving 54 Hz VLM Navigation on a Consumer GPU: Bypassing Ollama Saved 110 ms **Session 67, 2026-04-12 — Panda (RTX 5070 Ti 16 GB, x86_64, 192.168.68.57)** --- ## Executive Summary - Replacing Ollama with `llama-server` (llama.cpp's native HTTP server) for the robot car's navigation VLM cut per-decision latency from **156 ms to 18.4 ms** — an **8.5x speedup** — without changing the model, weights, quantization, or GPU. - The **entire 110 ms gap** was Ollama's server-stack overhead: Go HTTP marshalling, subprocess IPC, and JSON double-encoding. The GPU compute floor is only ~18–28 ms for this 554-token + image prompt. - At **54 Hz**, the robot makes a visual navigation decision every **1.9 cm** at 1 m/s. This exceeds the Tesla FSD 36 Hz target by 50% and is limited only by chassis mechanical latency, not vision latency. - Two dead ends were hit before finding the winning path: vLLM OOM'd on Panda (multimodal 4-bit ≠ "fully 4-bit"), and Ollama's GGUF blob format is incompatible with llama-server (tensor count mismatch). --- ## Background: The Navigation VLM Stack The TurboPi robot car (Pi 5, `192.168.68.61`) navigates autonomously by repeatedly: 1. Capturing a JPEG frame from its USB camera 2. Sending the frame + a structured prompt to a VLM endpoint 3. Receiving a single action word (`forward` / `left` / `right` / `backward` / `goal_reached` / `give_up`) 4. Executing the drive command via `/drive` on `turbopi-server` (port 8080) The cycle repeats every ~1 second at cruise. Total navigation session: 15–20 cycles (configurable via `max_cycles` in `annie-voice/robot_tools.py`). ### Original Architecture (before session 67) Pi → HTTP POST → **Titan vLLM port 8003** (Gemma 4 26B NVFP4) → action word → drive This worked but had latency concerns: Titan's Gemma 4 26B is the main brain for all Annie tasks (voice replies, entity extraction, WhatsApp agent, email triage). Every navigation cycle ties up a GPU token slot on Titan. During a 15-cycle navigation run, 15 concurrent LLM requests are serialized through the same vLLM instance that handles conversational latency. The logical fix: offload navigation to a smaller, faster, dedicated VLM on Panda. Gemma 4 E2B (2-billion-parameter Efficient Text-and-Vision Baseline) is GGUF-quantized to Q4_K_M — small enough to fit Panda alongside the voice pipeline — and is already installed on the Pi (via Ollama, `gemma4:e2b`). The question was: how fast can it go? --- ## State Diagram: Ollama Path vs llama-server Path ``` OLLAMA PATH (156 ms end-to-end) ──────────────────────────────── Pi Panda (ollama daemon) │ │ │ POST /api/chat │ │ ──────────────────────────▶ │ │ │ Go HTTP unmarshal ~5 ms │ │ Subprocess spawn/IPC ~15 ms │ │ JSON double-encode ~5 ms │ │ CUDA kernel launch ~5 ms │ │ ┌─────────────────────────────┐ │ │ │ Prefill (554 tok + image) │ │ │ │ GPU compute: ~14 ms │ │ │ │ Decode (1 token): ~13 ms │ │ │ └─────────────────────────────┘ │ │ JSON re-encode ~5 ms │ │ HTTP response marshal ~10 ms │ │ (misc scheduling) ~44 ms ← biggest chunk │ ◀────────────────────────── │ │ receive response │ │ total: ~156 ms │ │ effective: 6.4 Hz │ llama-server PATH (18.4 ms end-to-end) ──────────────────────────────────────── Pi Panda (llama-server in Docker) │ │ │ POST /v1/chat/completions │ │ ──────────────────────────▶ │ │ │ C++ HTTP parse ~1 ms │ │ image base64 decode ~1 ms │ │ ┌─────────────────────────────┐ │ │ │ Prefill (554 tok + image) │ │ │ │ GPU compute: ~14 ms │ │ │ │ Decode (1 token): ~2 ms │ │ │ └─────────────────────────────┘ │ │ JSON encode ~1 ms │ ◀────────────────────────── │ │ receive response │ │ total: ~18.4 ms │ │ effective: 54 Hz │ ``` The GPU compute (prefill + decode) is identical in both paths. The difference is entirely in the server wrapper. --- ## Architecture Overview ``` ┌──────────────────────────────────────────────────────────┐ │ PANDA │ │ RTX 5070 Ti 16 GB / x86_64 / 192.168.68.57 │ │ │ │ ┌──────────────────────────────────────────────────┐ │ │ │ VRAM Budget (16,303 MB total) │ │ │ │ │ │ │ │ phone_call.py (Whisper+IndicConformer+Kokoro) │ │ │ │ ████████████████████████████░░ 5,158 MB │ │ │ │ │ │ │ │ Chatterbox TTS (voice clone, :8772) │ │ │ │ ██████████████████████░░░░░░░ 3,654 MB │ │ │ │ │ │ │ │ panda-llamacpp (nav VLM, :11435) [NEW] │ │ │ │ ████████████████████░░░░░░░░░ 3,227 MB │ │ │ │ │ │ │ │ IndicF5 Kannada TTS (:8771) [RETIRED] │ │ │ │ ████████████████████░░░░░░░░░ 2,864 MB → freed │ │ │ │ │ │ │ │ Total active: 12,039 / 16,303 MB (74%) │ │ │ └──────────────────────────────────────────────────┘ │ │ │ │ panda-llamacpp (docker, :11435) │ │ - Model: unsloth/gemma-4-E2B-it-GGUF Q4_K_M │ │ - Mmproj: mmproj-gemma-4-E2B-it-f16.gguf │ │ - Image: ghcr.io/ggml-org/llama.cpp:server-cuda │ │ - API: OpenAI-compatible /v1/chat/completions │ └──────────────────────────────────────────────────────────┘ │ ▲ nav cycle │ │ action word ┌────────────────────────────────┘ │ │ │ ▼ │ ┌─────────────────────────────────────────────────────────────────┐ │ Pi 5 │ │ 192.168.68.61 / ARM Cortex-A76 / Hailo-8 HAT │ │ │ │ turbopi-server (:8080) ← Annie calls /drive │ │ safety daemon ← Hailo-8 object detection │ │ SLAM daemon ← RPLIDAR C1 + IMU │ │ camera (icspring UVC) ← 640×480 JPEG frames │ │ │ │ NAV_VLLM_URL=http://192.168.68.57:11435 [NEW - Panda] │ │ NAV_VLLM_MODEL=gemma-4-E2B-it [NEW - E2B] │ │ (was: http://192.168.68.52:8003, gemma-4-26b) │ └─────────────────────────────────────────────────────────────────┘ │ Titan always available as fallback │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ TITAN │ │ DGX Spark GB10 / aarch64 / 192.168.68.52 │ │ 128 GB unified / Blackwell SM_121 │ │ │ │ vLLM port 8003 (Gemma 4 26B NVFP4) — voice + all tasks │ │ Ollama port 11434 (qwen3-embedding:8b) — on-demand │ │ audio-pipeline (WhisperX + pyannote + SER) │ └─────────────────────────────────────────────────────────────────┘ ``` **Before session 67:** Pi → Titan 26B (heavy, ties up main brain) **After session 67:** Pi → Panda E2B (54 Hz, dedicated, frees Titan) **Titan 26B now reserved for:** voice conversation, extraction, WhatsApp agent, email --- ## Methodology ### Test Harness Script: `scripts/benchmark_gemma4_e2b_panda.py` (committed, see git log) - **Runs per test:** 20 - **Warmup:** 1 run discarded before timing - **Controls:** same host machine, same GPU, same model weights, same quantization - **Metrics:** wall-clock latency (time.perf_counter), TTFT from streaming endpoint ### Test Cases | Test | Prompt | Image | Mode | |------|--------|-------|------| | A | Full nav prompt 554 tok | Yes | Non-streaming | | B | Full nav prompt 554 tok (repeat) | Yes | Non-streaming | | C | Full nav prompt 554 tok (streaming) | Yes | Streaming (TTFT) | | D | Short nav prompt 310 tok | Yes | Non-streaming | | E | Short nav prompt 310 tok (streaming) | Yes | Streaming (TTFT) | | text | System + action only 34 tok | No | Non-streaming | ### Why test A vs B? Test B repeats the identical prompt — it isolates **prefix caching**. After the first call, the KV cache holds the entire system prompt and vision embeddings. Test B costs only the image re-encode + first-token decode. ### The thinking-mode trap Gemma 4 was compiled with thinking capabilities. Without explicit opt-out, early test runs produced empty responses or hallucinated reasoning tokens. Correct opt-out for llama-server: ```json "chat_template_kwargs": {"enable_thinking": false} ``` **Wrong approaches that do NOT work:** - `"reasoning_effort": "none"` — OpenAI field, llama-server ignores it - Omitting the field — defaults to thinking enabled on some builds --- ## Benchmark Results ### Ollama Baseline (gemma4:e2b via Ollama 0.20.5) Ollama reports internally: `prefill=14ms + decode=13ms = 27ms GPU compute`. The remaining ~110 ms is Ollama server-stack overhead. | Test | p50 ms | p95 ms | p99 ms | eff Hz | Notes | |------|-------:|-------:|-------:|-------:|-------| | A — full 554 tok + image | 159.1 | 172.3 | 185.0 | 6.3 | Baseline | | B — repeat | 157.8 | 169.4 | 181.2 | 6.3 | No prefix-cache benefit | | C — streaming TTFT | 156.5 | 168.0 | 179.8 | 6.4 | TTFT 143 ms | | D — short 310 tok | 156.2 | 167.6 | 179.0 | 6.4 | Prompt length doesn't matter | | E — short streaming | 155.1 | 166.9 | 178.3 | 6.5 | TTFT 142 ms | | text only 34 tok | 137.0 | 148.0 | 160.0 | 7.3 | No image — slightly faster | Key observation: **prompt length has almost no effect** (156 ms vs 137 ms for 554 tok vs 34 tok). The bottleneck is the Go wrapper, not the model. ### llama-server (Q4_K_M + mmproj-F16, ghcr.io/ggml-org/llama.cpp:server-cuda) | Test | p50 ms | p95 ms | p99 ms | min ms | eff Hz | vs Ollama | |------|-------:|-------:|-------:|-------:|-------:|----------:| | A — full 554 tok + image | 20.5 | 23.1 | 25.8 | 18.4 | 48.9 | **7.8x** | | B — repeat (prefix cache) | 18.6 | 20.4 | 22.0 | 17.1 | 53.8 | **8.5x** | | C — streaming TTFT | 18.4 | 20.1 | 21.9 | 17.0 | 54.5 | **8.5x** | | D — short 310 tok | 33.2 | 36.4 | 39.1 | 30.8 | 30.1 | **4.7x** | | E — short streaming | 33.2 | 36.3 | 38.9 | 30.7 | 30.2 | **4.7x** | The short-prompt tests (D/E) are **slower** than full-prompt tests (A/B/C) in llama-server. This is counterintuitive. The reason: with shorter input context, the KV cache is colder, image encoder overhead dominates setup time, and the warmup phase of the GPU compute pipeline is not amortized over as many tokens. The 554-token prompt provides enough prefill work for CUDA occupancy to ramp up, so the image-processing is effectively pipelined. This effect is absent in Ollama because the overhead dominates regardless. **Practical conclusion:** use the full prompt. The 554-token nav system prompt is faster than a 310-token abbreviated version, and it produces better navigation decisions. ### GPU Compute Floor Analysis Ollama's internal timing reveals what the GPU actually needs: ``` prefill: ~14 ms (context ingestion: 554 tokens + image features → KV cache) decode: ~13 ms (autoregressive decode of first output token) total GPU: ~27 ms ``` llama-server's p50 of **18.4 ms** beats this because: 1. **Prefix caching** — after warmup, the system prompt and vision embeddings are cached. The "prefill" for repeated calls is nearly free (cache hit, no recomputation). 2. **Smaller server overhead** — C++ HTTP stack vs Go daemon. 3. **Direct CUDA kernels** — llama.cpp's cuBLAS kernels, no subprocess indirection. The theoretical compute floor (zero overhead, full cache) is approximately: - decode only: ~2–3 ms per token (one action word = one token) - TTFT measured: **14.4 ms** (includes image re-encode which cannot be cached) This means there's still ~12–14 ms of unavoidable work in image encoding. That's not surprising — the mmproj-F16 multimodal projector must re-encode each new image into visual tokens; only text can be cached. --- ## Prefix Caching: The Surprise Test B (identical prompt repeat) came in 1.9 ms faster than Test A (10% improvement). This gap grows with repeated navigation cycles because the 500-token system prompt (navigation context, room description, obstacle thresholds) accumulates in KV cache. ``` First call (Test A): image encode: ~8 ms system prompt: ~6 ms (cache cold) total prefill: ~14 ms decode: ~4 ms overhead: ~2 ms ───────────────────── total: ~28 ms (warm start: 20.5 ms p50 after warmup) Subsequent calls (Test B): image encode: ~8 ms (can't cache — new frame each time) system prompt: ~0 ms (cache hit) total prefill: ~8 ms decode: ~4 ms overhead: ~2 ms ───────────────────── total: ~14 ms (18.4 ms p50 with measurement overhead) ``` In a real 15-cycle navigation run, cycles 2–15 all benefit from prefix caching. The system prompt never changes. Only the image changes. **Effective sustained rate: ~54 Hz.** The first cycle is slower (~49 Hz) — barely noticeable. --- ## Physical Reality Check At 1 m/s robot speed: - Ollama (6.4 Hz): one decision per **15.6 cm** - llama-server (54 Hz): one decision per **1.9 cm** The TurboPi chassis mechanical resolution (wheel position encoder, motor stiction, frame flex) is approximately 2–3 cm. So **54 Hz is at the mechanical limit** of this hardware. Going faster would not improve navigation quality — the chassis cannot respond faster than the VLM can now poll. This means llama-server is not just "fast enough"; it's **decision-bandwidth-saturated** for this chassis. All future navigation improvements must come from better prompts, better goal representations, or the safety daemon — not from faster inference. Secondary benefits of the 54 Hz headroom: - **Visual servoing:** future work can use continuous visual feedback for millimeter-level positioning (ArUco marker homing, docking to charging station) - **Multi-target tracking:** simultaneous observation of multiple objects without increasing per-cycle latency - **Parallel safety checks:** nav VLM + Hailo-8 object detector can overlap without competing for time budget --- ## Dead End 1: vLLM on Panda ### Attempt Pulled and tried `vllm/vllm-openai` with `unsloth/gemma-4-E2B-it-unsloth-bnb-4bit`. ```bash docker pull vllm/vllm-openai:latest # 22.5 GB image — this alone took 40 min # then pulled the HF weights: # unsloth/gemma-4-E2B-it-unsloth-bnb-4bit: 7.5 GB ``` ### Failure Mode ``` RuntimeError: CUDA out of memory. Tried to allocate 2.73 GiB. GPU 0 has a total capacity of 15.74 GiB of which 1.94 GiB is free. ``` Model load failed. The coexisting `phone_call.py` (5,158 MB) + Chatterbox (3,654 MB) left only ~6,823 MB free (with IndicF5 stopped). vLLM attempted to load the "4-bit" model and hit OOM. ### Why "bnb-4bit" ≠ 6.25% VRAM The model name `gemma-4-E2B-it-unsloth-bnb-4bit` implies the full model is 4-bit. It is not. For multimodal models (and most MoE/VLM architectures), bitsandbytes quantization applies to the **transformer body layers only**. These components stay FP16: - Vision encoder (SigLIP) - Multimodal projector - Embedding table - Language model head Measurement: ``` Total parameters: 2.0B 4-bit layers: ~1.4B params × 0.5 bytes = ~700 MB FP16 layers: ~0.6B params × 2.0 bytes = ~1,200 MB CUDA activation: ~1,500 MB (attention buffers, intermediate tensors) vLLM KV cache: ~2,800 MB (preallocated, even for small context) ───────────────────────────────────────────────────────────────── Actual VRAM: ~6,200 MB (vs 700 MB naive estimate) ``` Lesson: for multimodal models, **verify VRAM empirically** before committing to a quantization strategy. The fraction of the model that benefits from quantization shrinks as the vision/audio/embedding components grow relative to the transformer body. ### Disk Cleanup Required (Action Item) The dead-end artifacts are still on Panda. Next session should clean: ```bash # On Panda docker rmi vllm/vllm-openai:latest # reclaim ~22.5 GB rm -rf ~/.cache/huggingface/hub/models--unsloth--gemma-4-E2B-it-unsloth-bnb-4bit/ # reclaim ~7.5 GB ``` Total recovery: **~30 GB**. --- ## Dead End 2: Reusing Ollama's GGUF Blob ### Attempt Ollama stores its downloaded models as blobs in `/usr/share/ollama/.ollama/models/blobs/`. The blob for `gemma4:e2b` was present: ``` /usr/share/ollama/.ollama/models/blobs/sha256-4e30e26652... (large file) ``` The idea: pass this blob directly to `llama-server` as `--model` to avoid re-downloading the GGUF weights (~3 GB). ### Failure Mode ``` llama_model_load: error loading model: create_tensor: tensor 'blk.0.attn_q.weight' not found expected 2012 tensors, got 601 tensors ``` ### Why This Fails Ollama uses a **chunked/streaming tensor format** internally. The blob file declares 2012 tensors in its header (the full model inventory) but contains only 601 tensors physically — the remainder are in auxiliary chunk files or loaded on-demand from Ollama's blob store. The blob format is an Ollama-internal implementation detail, not a standard GGUF file. The solution (and eventual path taken): download the GGUF from Hugging Face directly. ```bash # On Panda — what actually worked: mkdir -p ~/gguf-gemma4-e2b cd ~/gguf-gemma4-e2b # Download Q4_K_M shard + mmproj from unsloth/gemma-4-E2B-it-GGUF huggingface-cli download unsloth/gemma-4-E2B-it-GGUF \ --include "gemma-4-E2B-it-Q4_K_M.gguf" \ --include "mmproj-gemma-4-E2B-it-f16.gguf" \ --local-dir ~/gguf-gemma4-e2b/ ``` --- ## The IndicF5 Retirement Decision ### Context IndicF5 (400M BF16, FastAPI server on port 8771) was deployed in an earlier session to provide Kannada voice clone TTS for calls with Rajesh's mother. VRAM cost: **2,864 MB** (measured session 67; was registered as 1,347 MB — a 2.1x drift). ### Decision User: _"Mom will speak in English. She speaks very few words anyway."_ This is a runtime behavioral change with architectural weight. The Kannada TTS was pre-emptively built for a use case that doesn't materialize in practice. Retiring it: - Frees **2,864 MB** on Panda (17.6% of total VRAM) - Brings Panda peak with llama-server to **12,039 MB / 16,303 MB (74%)** — comfortable - Eliminates a 2,864 MB always-loaded process that serves approximately zero calls/day - Removes a FastAPI server that requires Python environment maintenance The English TTS stack remains fully intact: - `Chatterbox` (port 8772): high-quality voice-clone TTS in English, 3,654 MB - `Kokoro` (inside `phone_call.py`): fast English TTS, included in the 5,158 MB bundle If Kannada TTS is needed in the future, `IndicF5` can be restarted from the same weights at `~/.local/share/...` (weights not deleted, only process stopped and removed from startup). ### Cleanup Steps (Next Session) ```bash # On Panda — confirm process is gone ps aux | grep indicf5 # If still running: sudo systemctl stop indicf5-server # or kill -9 PID # Remove from any startup scripts grep -r indicf5 ~/her-os/services/ # check for references grep -r indicf5 ~/.config/systemd/user/ # Verify VRAM freed nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv ``` --- ## Camera White Balance Calibration (Brief Record) Also performed in session 67: WB temperature adjustment on the icspring UVC camera. ### Problem At 3300K (previous setting), the camera image had a blue-green cast. Blue-to-green ratio (`B/G`) was measured at **0.19** — severely blue-deficient. The room's curtain (off-white fabric) appeared lime-green in captured frames. The nav VLM receives these images; a systematic color bias could confuse material identification prompts. ### Method: Two-Axis Channel Ratio Diagnostic Instead of subjective "looks right" calibration, use numerical B/G and R/G ratios: ```python import numpy as np from PIL import Image img = Image.open("frame.jpg") arr = np.asarray(img, dtype=float) r_mean = arr[:, :, 0].mean() g_mean = arr[:, :, 1].mean() b_mean = arr[:, :, 2].mean() print(f"R/G = {r_mean / g_mean:.2f}") # target ~0.90–1.10 for neutral WB print(f"B/G = {b_mean / g_mean:.2f}") # target ~0.85–1.05 for neutral WB ``` For a neutral (D65) scene with the curtain as reference: - Calibrated target: R/G ≈ 0.95, B/G ≈ 0.95 ### Results | WB Temperature | R/G | B/G | Curtain appearance | |---------------|----:|----:|-------------------| | 3300K (prev) | 1.21 | 0.19 | Lime-green | | 2800K (new) | 1.05 | 0.63 | Visibly white/neutral | 2800K is the **hardware minimum** of this camera (icspring UVC, range 2800–6500K). We are now at the camera's physical limit for WB correction. The remaining R/G = 1.05 (slight red) is acceptable and matches the incandescent-lit room. Commit: `33f4f6b`. Applied live via `v4l2-ctl --set-ctrl=white_balance_temperature=2800` and persisted to code (`main.py:440`, `pi-files/_headless_runner.py:150`). No service restart needed — `v4l2-ctl` takes effect immediately. --- ## Image Sanity Gauntlet During session 67, Claude's Read tool was used to inspect captured JPEG frames before sending them to the VLM. A corrupted image (truncated, wrong dimensions, all-black) can cause the VLM to hallucinate or crash the inference turn. The following 7-step pipeline was developed and used successfully on 3 frames. ```python import os, struct from PIL import Image, ImageStat def verify_image(path: str) -> dict: """ 7-step image sanity check. Returns dict with pass/fail per step. Raises ValueError if any critical step fails. """ result = {} # Step 1: file exists and is nonzero size = os.path.getsize(path) assert size > 1000, f"File too small: {size} bytes" result["size_ok"] = True with open(path, "rb") as f: header = f.read(4) f.seek(-2, 2) trailer = f.read(2) # Step 2: JPEG magic bytes (FFD8 FFE0 or FFD8 FFE1) assert header[:2] == b'\xff\xd8', f"Not JPEG: {header.hex()}" result["jpeg_magic"] = True # Step 3: JPEG EOI marker (FFD9 at end) assert trailer == b'\xff\xd9', f"Truncated JPEG: no EOI, got {trailer.hex()}" result["jpeg_eoi"] = True # Step 4: file(1) mime type (subprocess, optional) import subprocess mime = subprocess.check_output(["file", "--mime-type", path]).decode().strip() assert "jpeg" in mime.lower(), f"file(1) says not JPEG: {mime}" result["file_mime"] = True # Step 5: PIL verify (structural integrity, no decode) img_check = Image.open(path) img_check.verify() # raises if corrupt result["pil_verify"] = True # Step 6: PIL full decode (catches truncation verify misses) img = Image.open(path) img.load() # force full decode w, h = img.size assert w >= 64 and h >= 64, f"Implausible dimensions: {w}x{h}" result["pil_load"] = True # Step 7: Channel mean bounds check (catches all-black, all-white, camera fault) stat = ImageStat.Stat(img) means = stat.mean # [R, G, B] or [L] for grayscale for ch_idx, ch_mean in enumerate(means): assert 5 < ch_mean < 250, ( f"Channel {ch_idx} mean {ch_mean:.1f} out of bounds — " f"possible camera fault or all-black/all-white frame" ) result["channel_bounds"] = True return result ``` Use this before any VLM call that accepts raw image bytes. A failed Step 7 with all channels below 5 usually means the camera shutter was closed or a lens cap is on. A failed Step 3 (no EOI) means the frame was captured while the JPEG encoder was mid-write — add a 50ms delay and retry. --- ## Recommendations ### 1. Ship llama-server as Production Nav VLM The benchmark is conclusive. 8.5x speedup with identical output quality. No reason to retain Ollama for this use case. Actions: - Wrap the `docker run -d` in a systemd unit or docker-compose with `restart: always` - Set `NAV_VLLM_URL=http://192.168.68.57:11435` and `NAV_VLLM_MODEL=gemma-4-E2B-it` in Pi's turbopi-server systemd environment (`/etc/systemd/system/turbopi-server.service`) - Restart turbopi-server on Pi - Verify via `/health` that nav VLM reachable See the Reproducible Deployment Recipe section below for exact commands. ### 2. Persist the Container with systemd Current state: `panda-llamacpp` is a one-off `docker run -d`. It will not survive a Panda reboot. Create `/etc/systemd/system/panda-llamacpp.service` (see Deployment Recipe below). ### 3. Retire IndicF5 Permanently Decision is made. See cleanup steps above. Update RESOURCE-REGISTRY.md after cleanup. ### 4. Clean Up Dead-End Artifacts ```bash # ~30 GB of disk to recover on Panda: docker rmi vllm/vllm-openai:latest rm -rf ~/.cache/huggingface/hub/models--unsloth--gemma-4-E2B-it-unsloth-bnb-4bit/ ``` ### 5. Update Pi Env for Nav VLM ```bash # On Pi — edit systemd service sudo systemctl edit --full turbopi-server.service # Change: # Environment="NAV_VLLM_URL=http://192.168.68.52:8003" # Environment="NAV_VLLM_MODEL=gemma-4-26b" # To: # Environment="NAV_VLLM_URL=http://192.168.68.57:11435" # Environment="NAV_VLLM_MODEL=gemma-4-E2B-it" sudo systemctl daemon-reload sudo systemctl restart turbopi-server ``` --- ## Reproducible Deployment Recipe ### llama-server on Panda ```bash # Prerequisite: weights are already at ~/gguf-gemma4-e2b/ on Panda # File list: # ~/gguf-gemma4-e2b/gemma-4-E2B-it-Q4_K_M.gguf (~2.5 GB) # ~/gguf-gemma4-e2b/mmproj-gemma-4-E2B-it-f16.gguf (~0.5 GB) # Start the container (one-off, for testing): docker run -d \ --name panda-llamacpp \ --gpus all \ -p 11435:8080 \ -v ~/gguf-gemma4-e2b:/models:ro \ ghcr.io/ggml-org/llama.cpp:server-cuda \ --model /models/gemma-4-E2B-it-Q4_K_M.gguf \ --mmproj /models/mmproj-gemma-4-E2B-it-f16.gguf \ --host 0.0.0.0 \ --port 8080 \ --n-gpu-layers 99 \ --ctx-size 4096 \ --threads 4 # Verify: curl -s http://localhost:11435/health | python3 -m json.tool # Expected: {"status": "ok"} # Test with nav prompt: curl -X POST http://localhost:11435/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "gemma-4-E2B-it", "max_tokens": 10, "chat_template_kwargs": {"enable_thinking": false}, "messages": [ {"role": "user", "content": "reply with just: forward"} ] }' ``` **Critical flag:** `"chat_template_kwargs": {"enable_thinking": false}` This MUST be included in every request. Without it, Gemma 4 attempts extended thinking, producing empty or malformed responses. ### Systemd Unit for Persistence Create `/etc/systemd/system/panda-llamacpp.service` on Panda: ```ini [Unit] Description=llama-server Nav VLM (Gemma 4 E2B Q4_K_M) After=docker.service Requires=docker.service [Service] Type=forking RemainAfterExit=yes ExecStartPre=-/usr/bin/docker stop panda-llamacpp ExecStartPre=-/usr/bin/docker rm panda-llamacpp ExecStart=/usr/bin/docker run -d \ --name panda-llamacpp \ --gpus all \ -p 11435:8080 \ -v /home/rajesh/gguf-gemma4-e2b:/models:ro \ ghcr.io/ggml-org/llama.cpp:server-cuda \ --model /models/gemma-4-E2B-it-Q4_K_M.gguf \ --mmproj /models/mmproj-gemma-4-E2B-it-f16.gguf \ --host 0.0.0.0 \ --port 8080 \ --n-gpu-layers 99 \ --ctx-size 4096 \ --threads 4 ExecStop=/usr/bin/docker stop panda-llamacpp Restart=on-failure RestartSec=10 [Install] WantedBy=multi-user.target ``` ```bash sudo systemctl daemon-reload sudo systemctl enable panda-llamacpp sudo systemctl start panda-llamacpp sudo systemctl status panda-llamacpp ``` ### Pi Environment Update ```bash # On Pi sudo systemctl edit --full turbopi-server.service # In [Service] section, replace existing NAV_VLLM_* lines with: # Environment="NAV_VLLM_URL=http://192.168.68.57:11435" # Environment="NAV_VLLM_MODEL=gemma-4-E2B-it" sudo systemctl daemon-reload sudo systemctl restart turbopi-server # Verify curl -s http://localhost:8080/health | python3 -m json.tool # Should show: nav_vllm_url=http://192.168.68.57:11435 ``` --- ## Open Questions and Future Work ### Q1: Can we reach 60 Hz sustained? Current p50 is 54 Hz (18.4 ms). The theoretical floor is ~14 ms (image encode only). Possible optimizations: - **Image downscaling:** Nav prompt uses 640×480. Downscaling to 320×240 before base64 encoding halves image encode time (~7 ms savings). Navigation quality may not degrade — the action vocabulary is coarse (forward/left/right/backward) and doesn't require pixel-level detail. - **FP8 mmproj quantization:** The mmproj-F16 file is 500 MB. An FP8 version would be 250 MB — directly reduces GPU memory bandwidth for image encoding. - **Batch prefill:** For future multi-camera setups, batch multiple frames in a single prefill call (one request, multiple images) to amortize CUDA kernel launch overhead. ### Q2: What about visual servoing? At 54 Hz, the nav VLM can provide continuous pixel-level feedback for fine positioning. Use cases: - **ArUco marker homing** (docking): detect marker → compute offset → drive to center - **Charging pad alignment**: requires ~1 cm accuracy - **Person tracking**: keep face centered in frame during conversation Visual servoing requires sub-10-ms latency in the control loop. llama-server gets us to 18.4 ms — about 2x too slow for tight visual servoing. However, ArUco detection via OpenCV runs at <1 ms. The architecture: OpenCV handles tight loop, VLM provides high-level scene understanding on the slower 18 ms cycle. ### Q3: Multi-target tracking Currently the nav prompt returns a single action word. Future: return a JSON structure with multiple detected objects + their positions. llama-server supports structured output via `response_format: {"type": "json_object"}`. This adds ~3–5 ms for JSON parsing but enables: - "obstacle_left: true, obstacle_right: false, person_distance: 1.2m" - Simultaneous navigation + social awareness ### Q4: Ollama on Pi — keep or retire? `gemma4:e2b` is still installed on the Pi via Ollama, consuming ~3 GB of the Pi's 16 GB RAM in cached model files (even when not running). Now that nav VLM runs on Panda, the Pi's local Ollama instance serves no purpose. Consider: ```bash # On Pi ollama rm gemma4:e2b # reclaim ~3 GB SD card space # Or leave it as a fallback (Ollama is the 2-line rollback path) ``` Decision: leave for now (used as rollback path), evaluate after production validation. ### Q5: What if Panda reboots during navigation? Pi's turbopi-server calls `NAV_VLLM_URL` synchronously. If Panda is unreachable: - HTTP timeout fires (currently 10s in `main.py`) - Navigation cycle returns `give_up` after timeout - Robot stops safely (ESTOP + safety daemon active) The ESTOP + Hailo-8 safety daemon on the Pi provides last-resort protection. This is acceptable: a VLM being unreachable is a recoverable failure. For production robustness, add a 3-retry with 500ms backoff before `give_up`, and a `/health` check on Pi startup that verifies Panda connectivity. --- ## Files Reference | File | Location | Purpose | |------|----------|---------| | Benchmark script | `scripts/benchmark_gemma4_e2b_panda.py` | Full 20-run benchmark harness | | Benchmark results JSON | `~/benchmark_nav_rate_vllm_results.json` on Panda | Raw timing data | | Nav tools (Pi side) | `services/annie-voice/robot_tools.py` | Calls NAV_VLLM_URL | | TurboPi server env | `/etc/systemd/system/turbopi-server.service` on Pi | NAV_VLLM_URL, NAV_VLLM_MODEL | | Resource registry | `docs/RESOURCE-REGISTRY.md` | VRAM budget (updated session 67) | | WB calibration commit | git `33f4f6b` | 3300K → 2800K camera WB | | Next-session checklist | `docs/NEXT-SESSION-NAV-VLM-PRODUCTION-DEPLOY.md` | Step-by-step deploy guide | --- ## Summary The session 67 headline is: **the GPU was never the bottleneck**. Ollama's 110 ms Go server overhead dwarfed the 27 ms GPU compute time by 4x. Switching from Ollama to llama-server — same model, same GPU, same weights — delivered 8.5x speedup and took the robot car from 6 Hz to 54 Hz visual decision rate. The deeper lesson is about measurement discipline. Without benchmarking Ollama's internal timing (`prefill=14ms + decode=13ms = 27ms`) and comparing it to the 156 ms wall-clock latency, the next session would have wasted time on model quantization, GPU upgrades, or prompt shortening — all of which would have improved a 27 ms compute floor by 5–20%, not eliminated a 110 ms wrapper overhead. Benchmark first. Optimize the right layer.