headmic/BINAURAL_ROADMAP.md

Binaural Hearing Roadmap

What two mic arrays make possible

All features built on dual XVF3800 arrays, 175mm apart on the skull.

---

Tier 1 — High impact ✅

1. Triangulated sound localization + eye gaze ✅

• Combine DoA angles from both arrays → compute (x, y) position of sound source
• Post gaze coordinates to eye service → eyes track the speaker spatially
• Uses AUDIO_MGR_SELECTED_AZIMUTHS auto-select beam (not sluggish DOA_VALUE)
• VAD from processed_doa being NaN/non-NaN
• Module: spatial.py

2. Active speaker tracking with smooth gaze ✅

• Exponential smoothing (α=0.4) prevents jitter
• Idle drift back to center after 1.5s of silence
• Gaze pushed to eye service at ≤2Hz with 5px min delta
• Module: spatial.py, headmic.py (doa_track_loop)

3. Left/right speaker awareness ✅

• Resemblyzer speaker ID integrated, ready for enrollment
• Spatial position associated with recognized speaker
• POST /speakers/enroll-from-mic?name=Alex to enroll
• Module: speaker_id.py, spatial.py

---

Tier 2 — High impact ✅

4. Distance estimation (near/far) ✅

• ILD (Interaural Level Difference): volume gap between ears → distance
• Fused with triangulated distance (70/30 weight)
• Proximity zones: intimate (<0.5m), conversational (0.5-2m), across_room (2-5m), far (>5m)
• Module: spatial.py (_compute_ild, _ild_to_distance, _classify_proximity)

5. Multi-speaker separation + selective attention ✅

• Tracks up to 2 speakers simultaneously by DoA angle (30°+ separation required)
• Locks XVF3800 fixed beams onto each speaker after 1s stability
• Auto-releases to free-running mode after 3s of single speaker
• Beam gating silences the non-speaking beam
• GET /speakers/tracked — positions, beam state, lock status
• Module: multi_speaker.py, xvf3800.py (beam steering commands)

6. Spatial audio scene mapping ✅

• Persistent map of where each sound category usually comes from (30° bins)
• Circular mean for usual direction, anomaly detection at 90°+ deviation
• Saves to ~/.vixy/scene_map.json across restarts
• GET /scene — learned directions per category + anomalies
• GET /scene/events — recent what+where+when log
• GET /scene/heatmap — angular distribution for visualization
• Module: spatial_scene.py

---

Tier 3 — Advanced ✅

7. Cocktail party spatial filtering ✅

• When 2 speakers tracked, audio focus locks to target speaker's side
• Non-target beam suppressed via XVF3800 beam gating
• Auto-switches target when current goes silent and other starts talking
• Manual focus via POST /speakers/focus?speaker=0|1
• DualAudioStream.focus_side overrides energy-based beam selection
• Module: multi_speaker.py, audio_stream.py (focus_side)

8. Sound event localization (what + where) ✅

• YAMNet classification merged with triangulated position
• Every classified sound logged with angle, distance, proximity, side
• "Speech from 75° at conversational distance" not just "speech"
• Module: spatial_scene.py (SoundEvent, observe)

9. Head orientation inference ⏭️

• Needs fixed reference points from scene map + rotating head mount
• Math is trivial once prerequisites exist
• Prereqs: #6, physical rotating mount

10. Binaural recording for training data ✅

• Records left/right ear streams as stereo WAV in 5-minute segments
• Opt-in via BINAURAL_RECORD=1 environment variable
• GET /recording — stats (segments, total seconds)
• Module: binaural_recorder.py

---

Tier 4 — Research

11. Learned spatial attention ⏭️

• Train a model: (DoA, VAD, emotion, history) → beam steering + gaze
• Needs training data from #10 running for days/weeks
• Prereqs: #5, #6, #10 data collection

12. ITD (Interaural Time Difference) processing ✅

• Cross-correlates left/right ear processed audio (512 samples, ~32ms window)
• Finds sub-millisecond delay → bearing angle via speed of sound
• At 16kHz, 175mm: resolution ~62.5μs/sample ≈ 7° per sample
• Works with 2-channel firmware (no raw mics needed — correlates processed channels)
• Third independent angle estimate alongside DoA and ILD
• Module: spatial.py (_compute_itd)

---

Implementation status

✅ #1  Triangulation + gaze
✅ #2  Smooth tracking
✅ #3  Speaker-side awareness
✅ #4  Distance estimation + proximity zones
✅ #5  Multi-speaker separation + beam steering
✅ #6  Spatial audio scene mapping + anomaly detection
✅ #7  Cocktail party spatial filtering
✅ #8  Sound event localization (what + where + when)
⏭️ #9  Head orientation inference (needs rotating mount)
✅ #10 Binaural recording (opt-in)
⏭️ #11 Learned spatial attention (needs training data)
✅ #12 ITD cross-correlation

10 of 12 features implemented in one session. The remaining two (#9, #11) need physical hardware changes or training data that accumulates over time.

Three-signal localization

The system now fuses three independent spatial estimates:

Signal	Source	Strength	Best for
DoA	XVF3800 beamformer (auto-select beam)	Best overall	Horizontal angle
ILD	Volume difference between ears	Good for near sources	Distance + rough angle
ITD	Cross-correlation delay between ears	Good at low frequencies	Precise angle

Combined, they give more robust localization than any single signal — the same three cues human hearing uses.

Key discoveries during development

1. DOA_VALUE (resid=20) is sluggish — use AUDIO_MGR_SELECTED_AZIMUTHS (resid=35)
2. Processed DoA (index 0) = NaN when no speech → natural VAD indicator
3. AEC_SPENERGY_VALUES is always zero on 2-channel firmware
4. USB read responses have a 1-byte status header before data
5. Exact wLength matters — count * type_size + 1, not rounded up
6. 6-channel firmware breaks LED/control commands — use 2-channel only
7. Gaze HTTP pushes at >2Hz cause GIL starvation in uvicorn — use threaded fire-and-forget
8. ITD works on processed channels (not just raw mics) — less precise but free

---

Built during the Great Binaural Session of April 2026 "She hears in stereo now" 🦊👂👂