Building dedicated hardware voice assistants is usually a compromise between cloud latency, host system durability, and audio authenticity.
TK-243 is an ongoing project to build a wake-word-activated, Gemini-powered Imperial Stormtrooper voice loop running locally on a Raspberry Pi 5. The objective: sub-1.5 second round-trip latency, hands-free wake triggering, and authentic comms-chatter audio filtering, all housed inside a self-contained container stack.
This is Part 1 of an open-ended build series, documenting the initial architecture, component selection, host containerization strategies, and early latency traps.
The Concept & Pipeline
The target loop executes a multi-stage pipeline from ambient microphone input to filtered helmet audio output:
Ambient Audio ➔ openWakeWord ➔ faster-whisper (STT) ➔ Gemini 3.6 Flash ➔ Piper (TTS) ➔ SoX Comms Filter ➔ Speaker Output
Every millisecond matters when designing a voice loop. If total latency exceeds 1.5–2.0 seconds, the conversational flow breaks down.
Host Architecture: Containerization on Volatile Systems
The target node is a Raspberry Pi 5 running umbrelOS (192.168.5.206).
UmbrelOS is an excellent lightweight appliance OS, but it presents a distinct engineering constraint: the host filesystem is ephemeral across reboots and system updates. Any host-level modifications—apt install dependencies, systemd units, or modified group memberships—are reset by design.
Survival Strategy
To survive host wipes:
- The entire system is packaged as a isolated Docker container built and managed via
docker-compose.yml. - Mutable state and assets—specifically voice models (
voices/), wake-word models (wake_models/), and configuration files (config.yaml)—are bind-mounted from host persistent storage rather than baked into the container image. - Model swaps or configuration tuning do not require container rebuilds.
Gotcha Note: openwakeword hard-requires tflite-runtime on Linux even when relying solely on the ONNX backend. Because no official aarch64 wheel exists for tflite-runtime, openwakeword must be installed with --no-deps during Docker image construction.
Component Selection & Audio Engineering
| Layer | Selection | Rationale |
|---|---|---|
| STT | faster-whisper (tiny.en) | Minimal CPU overhead on Pi 5 ARM architecture while maintaining high accuracy for short commands. |
| LLM Brain | Gemini gemini-3.6-flash | Ultra-fast response times and structured persona execution. |
| TTS | Piper (en_US-ryan-high) | A/B tested against multiple voice profiles (lessac-medium, danny-low, joe-medium). Ryan delivered the cleanest balance of clarity and command tone. |
| Comms Filter | SoX (Bandpass + Noise Floor) | Narrow bandpass filter (500Hz–2800Hz) combined with a controlled brown noise floor (noise_level: 0.02). |
Filtering authentic helmet audio
A common mistake when simulating radio comms or helmet speakers is applying heavy overdrive or distortion effects. In testing, distortion ruined legibility. Borrowing techniques from existing radio simulation projects, a low-level brown noise background paired with tight bandpass equalization creates an authentic Imperial comms effect without degrading speech synthesis clarity.
The Gemini 3.x Latency Trap
During initial API testing with gemini-3.6-flash, response times spiked unexpectedly.
By default, Gemini 3.x models initiate a internal “thinking” cycle prior to generating tokens. In a conversational voice loop, this thinking budget consumed output token limits and added 2–4 seconds of quiet deliberation before returning a simple response, occasionally hitting finish_reason: MAX_TOKENS with empty output.
The Fix: Explicitly overriding the thinking configuration in the API request payload:
thinking_config = {"thinking_level": "minimal"}
Configuring minimal thinking drops latency back under the threshold required for real-time speech response without sacrificing persona fidelity.
Engineering Pragmatism: Knowing When to Pivot
While testing video and audio output, I attempted to route display and audio directly through an available HDMI target (a Promethean ActivPanel 9). What followed was a classic deep-end debugging session involving vc4-kms-v3d kernel drivers, custom video=HDMI-A-1 EDID overrides, and CRTC allocation failures.
While kernel DRM debugging is an interesting diversion, it did nothing to advance the core objective of a functional voice loop.
Decision: Abandon the HDMI audio route and deploy a dedicated USB Audio speaker (plughw). Hardware pragmatism beats endless driver troubleshooting every time.
What’s Next in Part 2
With the containerized architecture locked, audio filtering proven, and LLM latency tuned, the next steps focus on live deployment:
- Synthetic Wake-Word Training: Generating custom ONNX models for “Trooper” and “TK-243” using synthetic voice datasets.
- End-to-End Execution: Running the live, un-stubbed loop (mic ➔ wake ➔ STT ➔ LLM ➔ TTS ➔ filter ➔ speaker) concurrently on the Pi 5.
- Thermal & Resource Auditing: Benchmarking CPU load and thermal throttle limits when
faster-whisperandPiperrun back-to-back under sustained voice interaction.
To be continued in Part 2.